{"data":{"schema_version":4,"snapshot_id":"snap_2026_10_06_001","published_at":"2026-10-06T06:09:18.557Z","data_as_of":"2026-10-06T06:09:18.557Z","data_run_id":"run_3c78271b52884660b9e3d7f6524fef09","health":{"status":"complete","reasons":[],"warnings":["3 of the top-25 arena models have no intelligence index: gemini-4-argon, muse-spark-1.3-max, mimo-2.6-pro","7 top-25 arena names resolve to no model row (queued for census): claude-fable-5.1-max, claude-opus-5-max, muse-spark, kimi-k3-max, gemini-3-pro, gpt-6.1-sol-max, glm-5.3-max","2 benchmark columns hold evidence that now folds onto a WEIGHTED core benchmark and is not counted toward any composite: arena_text_elo → arena_elo, lmarena_text_arena_elo → arena_elo","24 benchmark columns no longer fold to their own id: agi_eval → agieval, apex_agent → apex_agents, arena_hard_v2 → arena_hard, arenahard → arena_hard, automation_bench → automationbench, big_bench_hard → bbh, bigbench_hard → bbh, bigbenchhard → bbh, deepsearch_qa → deepsearchqa, gsm_8k → gsm8k, hmmt_2026_feb → hmmt, hmmt_feb25 → hmmt, hmmt_feb_2025 → hmmt, hmmt_feb_2026 → hmmt, humanevalplus → humaneval_plus, if_bench → ifbench, imo_answerbench → imoanswerbench, math500 → math_500, mathvision → math_vision, multiple → multipl_e, omnimath → omni_math, swe_multilingual → swe_bench_multilingual, videomme → video_mme, workspacebench → workspace_bench","enrichment limited to new-model identification this run (72h gate closed): 3 planned, 304 due models deferred to a later run","25 new models (entered the index within 21 days of release) await research; 19 have waited 7 days or more since they became researchable: fugu-1-max, ling-3.0-flash-vision, glm-5.3-flashx, aion-3.5, aion-3.5-mini, ember-1, glm-5.3-prime, gpt-6-luna +11","3 current models carry active < total params but is_moe = 0 (mistral-3-large-vision, nemotron-2-nano-12b-vision, olmo-2-0325-32b-instruct): active count hidden and frontier fit reads them as dense","154 score rows carry trust above current policy (arxiv.org 3→2 ×58, vellum.ai 2→1 ×36, computeprices.com 2→1 ×14, huggingface.co 3→1 ×13, designforonline.com 2→1 ×9, 11 more host groups); run POST /v1/admin/scores/retrust","53 composite cells hold cross-source readings more than 15 normalized points apart (trust ≥2, same tools lane): gpt-5.4-nano/mmlu_pro 77.17 (vals.ai) vs 35.61 (layerlens.ai); command-a-03-2025/gpqa_diamond 76.06 (artificialanalysis.ai) vs 48.48 (vals.ai); gpt-5.4-nano/aime 88.75 (vals.ai) vs 26.67 (layerlens.ai); gpt-oss-120-b-120b/swe_bench_verified 62.4 (arxiv.org) vs 33.6 (vals.ai); deepseek-r1-distill-llama-8b-reasoning/gpqa_diamond 49 (arxiv.org) vs 30.2 (artificialanalysis.ai)","30 ranked models read ≥5 pts off full-coverage peers on their own benchmark subset (largest: phi-4-14b-reasoning +16.01, hunyuan-1-a13b-instruct +15.46, reka-3-flash +15.18, …); 57 outside the peer range, no estimate"],"coverage_gaps":[{"model_id":"gemini-4-argon","arena_rank":1,"reason":"no_index"},{"model_id":"muse-spark-1.3-max","arena_rank":7,"reason":"no_index"},{"model_id":"mimo-2.6-pro","arena_rank":19,"reason":"no_index"}],"arena_guarantee":{"top_n":25,"ranked":25,"unresolved_names":["claude-fable-5.1-max","claude-opus-5-max","muse-spark","kimi-k3-max","gemini-3-pro","gpt-6.1-sol-max","glm-5.3-max"],"unresolved_names_source":"harvested"},"capacity":{"body_bytes":4525835,"body_models":421,"payload_rows":421,"payload_bytes_max":35031,"payload_bytes_total":4339624,"kv_value_limit_bytes":26214400,"kv_body_alarm_bytes":12582912}},"body_policy":{"inactive_cap":250,"inactive_included":104,"inactive_omitted":0,"note":"superseded/retired models beyond the cap are omitted from this body only — they remain in /v1/models?status=all and /v1/models/:id"},"counts":{"models_total":421,"current":317,"superseded":94,"retired":10,"disputed":0},"diff":{"added":[],"updated":["aion-2.0","aion-3.0","aion-3.0-mini","aion-3.5","aion-3.5-mini","apodex-1.1-mini-35b","claude-fable-5","claude-fable-5.1","claude-haiku-4.5","claude-opus-4.1","claude-opus-4.5","claude-opus-4.6","claude-opus-4.7","claude-opus-4.8","claude-opus-5","claude-opus-5.5","claude-sonnet-4","claude-sonnet-4.5","claude-sonnet-4.6","claude-sonnet-5","claude-sonnet-5.5","codestral-2508","command-a-03-2025","command-a-05-plus-2026","command-r-08-2024","command-r-08-plus-2024","command-r-7-b-12-2024-7b","deepseek-3-0324-instruct","deepseek-3.1-instruct","deepseek-r1-0528","deepseek-r1-reasoning","deepseek-v3","deepseek-v3.1-terminus","deepseek-v3.2","deepseek-v4-flash","deepseek-v4-flash-0731","deepseek-v4-flash-vision","deepseek-v4-pro","deepseek-v4-pro-0813","deepseek-v4.1-flash","devstral-1-2512","dots-3-note","ember-1","ernie-4.5-424b-a47b-vision","fugu-1-max","fugu-1-ultra","fugu-2-ultra","gemini-2.5-flash","gemini-2.5-flash-image","gemini-2.5-flash-lite","gemini-2.5-pro","gemini-2.5-pro-06-05","gemini-3-flash","gemini-3-pro-image","gemini-3.1-flash-image","gemini-3.1-flash-lite","gemini-3.1-flash-lite-image","gemini-3.1-pro","gemini-3.1-pro-customtools","gemini-3.5-flash","gemini-3.5-flash-lite","gemini-3.6-flash","gemini-3.7-flash","gemini-3.8-flash","gemma-2-27b-instruct","gemma-3-12b-instruct","gemma-3-27b-instruct","gemma-3-4b-instruct","gemma-4-26b-a4b-instruct","gemma-4-31b-instruct","glm-4.5-355b-a32b","glm-4.5-air-106b-a12b","glm-4.5-v","glm-4.6","glm-4.6-v-106b","glm-4.7","glm-4.7-flash","glm-5","glm-5-turbo","glm-5-v-turbo","glm-5.1","glm-5.2","glm-5.3","glm-5.3-flash","glm-5.3-flashx","glm-5.3-prime","gpt-3.5-turbo","gpt-3.5-turbo-0613","gpt-3.5-turbo-16k","gpt-3.5-turbo-instruct","gpt-4","gpt-4-o","gpt-4-o-mini","gpt-4-turbo","gpt-4.1","gpt-4.1-mini","gpt-4.1-nano","gpt-5","gpt-5-image","gpt-5-image-mini","gpt-5-mini","gpt-5-nano","gpt-5-pro","gpt-5.1","gpt-5.1-codex","gpt-5.1-codex-max","gpt-5.1-codex-mini","gpt-5.2","gpt-5.2-codex","gpt-5.2-instruct","gpt-5.2-pro","gpt-5.3-codex","gpt-5.4","gpt-5.4-image-2","gpt-5.4-mini","gpt-5.4-nano","gpt-5.4-pro","gpt-5.5","gpt-5.5-pro","gpt-5.6-luna","gpt-5.6-luna-pro","gpt-5.6-sol","gpt-5.6-sol-pro","gpt-5.6-terra","gpt-5.6-terra-pro","gpt-6-astra","gpt-6-astra-pro","gpt-6-luna","gpt-6-luna-pro","gpt-6-sol","gpt-6-sol-pro","gpt-6.1-sol","gpt-6.1-sol-pro","gpt-o1","gpt-o1-pro","gpt-o3","gpt-o3-mini","gpt-o3-mini-high","gpt-o3-pro","gpt-o4-mini","gpt-o4-mini-high","gpt-oss-1-safeguard-20b","gpt-oss-120-b-120b","gpt-oss-20-b-20b","granite-4.0-h-micro","granite-4.2-8b","grok-0.1-build","grok-4.20","grok-4.20-multi-agent","grok-4.3","grok-4.5","grok-4.6","grok-4.7","hermes-3-llama-3.1-405b","hermes-3-llama-3.1-70b","hermes-4-405b","hunyuan-1-a13b-instruct","hy-3","hy-4","inkling-1","inkling-1-small","kat-coder-2.5-pro","kimi-k2","kimi-k2-0905","kimi-k2-reasoning","kimi-k2.5-0127","kimi-k2.6","kimi-k2.7-code","kimi-k3","l-3-lunaris-8b","l-3.1-euryale-70b","l-3.3-euryale-70b-v2.3","laguna-s-2.1","laguna-xs-2.1","lfm-2.5-2.6b","ling-3.0-flash","ling-3.0-flash-fin","ling-3.0-flash-sante","ling-3.0-flash-vision","ling-3.1-flash","llama-3.1-70b-instruct","llama-3.1-8b-instruct","llama-3.2-1b-instruct","llama-3.2-3b-instruct","llama-3.3-70b-instruct","llama-4-guard-12b","llama-4-maverick-17b-128e-instruct","llama-4-scout-17b-16e-instruct","longcat-2.0","mercury-2","mercury-2.5","mimo-2.5","mimo-2.5-pro","mimo-2.6-flash","mimo-2.6-pro","mimo-2.6-pro-ultraspeed","minimax-01","minimax-m1-reasoning","minimax-m2","minimax-m2-her","minimax-m2.1","minimax-m2.5","minimax-m2.7","minimax-m3-reasoning","mistral-1-large","mistral-1-nemo","mistral-2407-large","mistral-2501-small-24b-instruct","mistral-2502-saba","mistral-2512-large","mistral-2603-small","mistral-3-14b-2512-instruct","mistral-3-8b-2512-instruct","mistral-3-b-2512-3b","mistral-3-medium","mistral-3.1-medium","mistral-3.1-small-24b-2503-instruct","mistral-3.2-small-24b-2506-instruct","mistral-3.5-medium-vision","mixtral-8-x22b-instruct","morph-3-fast","morph-3-large","muse-glimmer-1-30b","muse-spark-1.1","muse-spark-1.2","muse-spark-1.2-contributor","muse-spark-1.3","muse-spark-1.3-contributor","namazu-1","nemotron-3-nano-30b-a3b","nemotron-3-nano-omni-30b-a3b-reasoning","nemotron-3-super-120b-a12b","nemotron-3-ultra-550b-a55b","nemotron-3.5-content-safety","nemotron-3.5-lightning","nex-n2.5-mini","nex-n2.5-pro","nova-1-lite","nova-1-micro","nova-1-premier","nova-1-pro","nova-2-lite-v1","palmyra-x5","pareto-26.10","perceptron-mk1","perceptron-mk1.5","phi-4-14b-instruct","qwen-2.5-32b-instruct-code","qwen-2.5-72b-instruct","qwen-2.5-72b-instruct-vision","qwen-2.5-7b-instruct","qwen-2025-plus","qwen-3-14b-instruct","qwen-3-235b-a22b","qwen-3-235b-a22b-07-25","qwen-3-235b-a22b-2507-reasoning","qwen-3-235b-a22b-instruct-vision","qwen-3-235b-a22b-reasoning-vision","qwen-3-30b-a3b-2507-instruct","qwen-3-30b-a3b-2507-reasoning","qwen-3-30b-a3b-instruct","qwen-3-30b-a3b-instruct-code","qwen-3-30b-a3b-instruct-vision","qwen-3-30b-a3b-reasoning-vision","qwen-3-32b-instruct","qwen-3-32b-instruct-vision","qwen-3-480b-a35b-07-25-code","qwen-3-8b-instruct","qwen-3-8b-instruct-vision","qwen-3-8b-reasoning-vision","qwen-3-flash-code","qwen-3-max","qwen-3-max-reasoning","qwen-3-next-80b-a3b-2509-instruct","qwen-3-next-80b-a3b-2509-reasoning","qwen-3-next-code","qwen-3-plus-code","qwen-3.5-122b-a10b","qwen-3.5-27b","qwen-3.5-35b-a3b","qwen-3.5-397b-a17b","qwen-3.5-9b","qwen-3.5-flash","qwen-3.5-plus","qwen-3.6-27b","qwen-3.6-35b-a3b","qwen-3.6-flash","qwen-3.6-max","qwen-3.6-plus-04-02","qwen-3.7-flash","qwen-3.7-max","qwen-3.7-plus","qwen-3.8-2.4t-a95b","qwen-3.8-27b","qwen-3.8-flash","qwen-3.8-max-0902-2.4t","qwen-3.8-max-prime","qwen-3.8-omni-flash","reka-1-edge-2603","reka-3-flash","relace-apply-3","relace-search-1","seed-1.6","seed-1.6-flash","seed-2-1-turbo","seed-2.0-code","seed-2.0-lite","seed-2.0-mini","solar-mini-4","solar-pro-3","solar-pro-4","sonar-1","sonar-1-pro","sonar-1-pro-reasoning","sonar-deep-research-1","sonar-pro-search-1","step-3.5-flash","step-3.7-flash","trinity-1-large-reasoning","wizardlm-2-8x22b"],"superseded":[],"disputed":[],"removed":[],"changed":["claude-fable-5","claude-fable-5.1","claude-haiku-4.5","claude-opus-4.1","claude-opus-4.5","claude-opus-4.6","claude-opus-4.7","claude-opus-4.8","claude-opus-5","claude-opus-5.5","claude-sonnet-4","claude-sonnet-4.5","claude-sonnet-4.6","claude-sonnet-5","claude-sonnet-5.5","codestral-2508","deepseek-3.1-instruct","deepseek-r1-0528","deepseek-v3","deepseek-v3.1-terminus","deepseek-v3.2","deepseek-v4-flash","deepseek-v4-flash-0731","deepseek-v4-flash-vision","deepseek-v4-pro","deepseek-v4-pro-0813","deepseek-v4.1-flash","gemini-2.5-flash","gemini-2.5-flash-lite","gemini-2.5-pro","gemini-3-flash","gemini-3.1-flash-lite","gemini-3.1-pro","gemini-3.5-flash","gemini-3.6-flash","gemini-3.7-flash","gemini-3.8-flash","gemma-3-4b-instruct","gemma-4-26b-a4b-instruct","glm-4.5-355b-a32b","glm-4.5-air-106b-a12b","glm-4.5-v","glm-4.6","glm-4.6-v-106b","glm-4.7","glm-4.7-flash","glm-5","glm-5-turbo","glm-5-v-turbo","glm-5.1","glm-5.2","glm-5.3","glm-5.3-flash","gpt-3.5-turbo","gpt-4","gpt-4-o","gpt-4-o-mini","gpt-4-turbo","gpt-4.1","gpt-4.1-mini","gpt-4.1-nano","gpt-5","gpt-5-mini","gpt-5-nano","gpt-5.1","gpt-5.1-codex","gpt-5.1-codex-mini","gpt-5.2","gpt-5.2-codex","gpt-5.3-codex","gpt-5.4","gpt-5.5","gpt-o1","gpt-o1-pro","gpt-o3","gpt-o3-mini","gpt-o3-pro","gpt-o4-mini","gpt-oss-120-b-120b","gpt-oss-20-b-20b","granite-4.0-h-micro","grok-0.1-build","grok-4.20","grok-4.3","grok-4.5","grok-4.6","grok-4.7","hermes-3-llama-3.1-405b","hermes-3-llama-3.1-70b","hermes-4-405b","hy-3","inkling-1","kimi-k2","kimi-k2-0905","kimi-k2-reasoning","kimi-k2.5-0127","kimi-k2.6","kimi-k2.7-code","kimi-k3","lfm-2.5-2.6b","llama-3.1-70b-instruct","llama-3.1-8b-instruct","llama-3.2-1b-instruct","llama-3.2-3b-instruct","llama-3.3-70b-instruct","llama-4-maverick-17b-128e-instruct","llama-4-scout-17b-16e-instruct","mercury-2","mimo-2.5","mimo-2.5-pro","mimo-2.6-pro","minimax-m1-reasoning","minimax-m2","minimax-m2.1","minimax-m2.5","minimax-m2.7","minimax-m3-reasoning","mistral-1-large","mistral-2407-large","mistral-2501-small-24b-instruct","mistral-2502-saba","mistral-2512-large","mistral-3-14b-2512-instruct","mistral-3-8b-2512-instruct","mistral-3-b-2512-3b","mistral-3-medium","mistral-3.1-medium","mistral-3.2-small-24b-2506-instruct","mixtral-8-x22b-instruct","muse-glimmer-1-30b","muse-spark-1.1","muse-spark-1.2","muse-spark-1.3","nemotron-3-nano-omni-30b-a3b-reasoning","nemotron-3-ultra-550b-a55b","nova-1-lite","nova-1-micro","nova-1-premier","nova-1-pro","nova-2-lite-v1","phi-4-14b-instruct","qwen-2.5-32b-instruct-code","qwen-2.5-72b-instruct","qwen-3-14b-instruct","qwen-3-235b-a22b","qwen-3-235b-a22b-07-25","qwen-3-235b-a22b-2507-reasoning","qwen-3-235b-a22b-instruct-vision","qwen-3-235b-a22b-reasoning-vision","qwen-3-30b-a3b-2507-instruct","qwen-3-30b-a3b-instruct","qwen-3-30b-a3b-instruct-code","qwen-3-30b-a3b-instruct-vision","qwen-3-30b-a3b-reasoning-vision","qwen-3-32b-instruct","qwen-3-32b-instruct-vision","qwen-3-480b-a35b-07-25-code","qwen-3-8b-instruct","qwen-3-8b-instruct-vision","qwen-3-8b-reasoning-vision","qwen-3-max","qwen-3-max-reasoning","qwen-3-next-80b-a3b-2509-instruct","qwen-3-next-80b-a3b-2509-reasoning","qwen-3.5-122b-a10b","qwen-3.5-27b","qwen-3.5-35b-a3b","qwen-3.5-397b-a17b","qwen-3.5-9b","qwen-3.5-plus","qwen-3.6-27b","qwen-3.6-max","qwen-3.6-plus-04-02","qwen-3.7-max","qwen-3.7-plus","reka-3-flash","solar-pro-4","sonar-1","sonar-1-pro","sonar-1-pro-reasoning","step-3.5-flash","step-3.7-flash","trinity-1-large-reasoning"],"renamed":[],"merged":[]},"hardware":[{"id":"intel_b70","display_name":"Intel Arc Pro B70","vendor":"intel","kind":"gpu","memory_gb":32,"usable_memory_gb":null,"memory_bw_gbps":608,"confirm_sku":false,"reference_stack":"vLLM (XPU backend), OpenVINO"},{"id":"nvidia_dgx_spark","display_name":"NVIDIA DGX Spark (GB10 Grace Blackwell)","vendor":"nvidia","kind":"system","memory_gb":128,"usable_memory_gb":null,"memory_bw_gbps":273,"confirm_sku":false,"reference_stack":"llama.cpp (MXFP4), SGLang, vLLM (Grace-Blackwell build)"},{"id":"nvidia_h100_sxm_80g","display_name":"NVIDIA H100 SXM5 80GB","vendor":"nvidia","kind":"gpu","memory_gb":80,"usable_memory_gb":null,"memory_bw_gbps":3350,"confirm_sku":false,"reference_stack":"vLLM, TensorRT-LLM, SGLang"}],"benchmarks":[{"id":"aa_agentic_index","display_name":"Artificial Analysis Agentic Index","category":"composite","unit":"index","scale_min":0,"scale_max":100,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":122},{"id":"aa_briefcase","display_name":"AA-Briefcase","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_553b3da274594398a3713bcb7b3afa2d","models_covered":5},{"id":"aa_briefcase_analytical_quality_elo","display_name":"AA-Briefcase Analytical Quality Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"aa_briefcase_elo","display_name":"AA-Briefcase Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"aa_briefcase_presentation_elo","display_name":"AA-Briefcase Presentation Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"aa_briefcase_rubric_pass_rate","display_name":"AA-Briefcase rubric pass rate","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"aa_coding_agent_index","display_name":"AA Coding Agent Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5a7b2bf97fed4a3cb01efdb8d503d9b4","models_covered":6},{"id":"aa_coding_index","display_name":"Artificial Analysis Coding Index","category":"composite","unit":"index","scale_min":0,"scale_max":100,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":136},{"id":"aa_intelligence_index","display_name":"Artificial Analysis Intelligence Index","category":"composite","unit":"index","scale_min":0,"scale_max":100,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":245},{"id":"aa_lcr","display_name":"AA-LCR","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f7be5cdfc6ed41c99f63660105f869ce","models_covered":195},{"id":"aa_omniscience","display_name":"AA-Omniscience","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a60ebb3a51fe482db14bf1479d4f70ec","models_covered":3},{"id":"aa_omniscience_accuracy","display_name":"AA-Omniscience Accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2e5a562a40d64392a3a803438b1da5c3","models_covered":7},{"id":"aa_omniscience_hallucination_rate","display_name":"AA-Omniscience Hallucination Rate","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_553b3da274594398a3713bcb7b3afa2d","models_covered":1},{"id":"aa_omniscience_index","display_name":"aa_omniscience_index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_94b6620d00e9430381a455dc60e828f0","models_covered":46},{"id":"aa_omniscience_net_score","display_name":"AA-Omniscience Net Score","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"aa_omniscience_non_hallucination","display_name":"AA-Omniscience Non-Hallucination","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2e5a562a40d64392a3a803438b1da5c3","models_covered":1},{"id":"aa_omniscience_non_hallucination_rate","display_name":"AA-Omniscience Non-Hallucination Rate","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":6},{"id":"aav_capsid_structure","display_name":"AAV capsid structure","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"aav_capsid_tropism","display_name":"AAV capsid tropism","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"aav_capsid_viability_ood_auroc","display_name":"AAV capsid viability (OOD AUROC)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"aav_generative_design","display_name":"AAV generative design","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"aegis_2_0","display_name":"Aegis 2.0","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"afumed_drug","display_name":"AFUMED-Drug","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"afumed_drug_internal","display_name":"AFUMED-Drug (internal)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"afusafe_medsce","display_name":"AFUSAFE-MedSCE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"afusafe_medsce_internal","display_name":"AFUSAFE-MedSCE (internal)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"agent_startup_bench","display_name":"Agent Startup Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"agentic_if_index","display_name":"Agentic IF Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_09f703cd057f467692c632c989f68b58","models_covered":1},{"id":"agents_last_exam","display_name":"Agents' Last Exam","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_acf3136ad9694471b137d965b747a7f6","models_covered":10},{"id":"agi_eval","display_name":"AGI EVAL","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"agieval","display_name":"AGIEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":6},{"id":"ai2_reasoning_challenge","display_name":"AI2 Reasoning Challenge","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_118237c52f154f449b89d1999110b526","models_covered":1},{"id":"ai2d","display_name":"AI2D","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_eb1342ae95b443ed98b6f405b52f2d37","models_covered":3},{"id":"aider","display_name":"Aider","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_42f387694fac489b8a51914a4e11c794","models_covered":1},{"id":"aider_polyglot","display_name":"Aider-Polyglot","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97c943d2c5ee4107a9a57d2adea4a5ad","models_covered":3},{"id":"aime","display_name":"AIME (annual)","category":"math","unit":"percent","scale_min":20,"scale_max":100,"higher_better":true,"weight_in_composite":0.0735,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":175},{"id":"aime_2026_no_tools","display_name":"AIME 2026 no tools","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":0},{"id":"ale_bench","display_name":"ALE-Bench","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":0},{"id":"alpacaeval_2_0","display_name":"AlpacaEval 2.0","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97c943d2c5ee4107a9a57d2adea4a5ad","models_covered":3},{"id":"amc23","display_name":"AMC23","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aab24e5371704d9db079b045e5b11a05","models_covered":1},{"id":"analyst_agent","display_name":"analyst_agent","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c1f9150dcede44909ee388a3dc05d2ae","models_covered":26},{"id":"androidbench","display_name":"AndroidBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"androidworld","display_name":"AndroidWorld","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4b491fc5dc374246822a3d46b2e00025","models_covered":1},{"id":"anthropic_eci","display_name":"Anthropic ECI","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"apex","display_name":"Apex","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_81a9cdaaecff4d788a0686b15f5261b8","models_covered":1},{"id":"apex_agent","display_name":"APEX-Agent","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_e468d8ceb73f4797b3bd0b95d08879f8","models_covered":1},{"id":"apex_agents","display_name":"APEX-Agents","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":35},{"id":"apex_swe","display_name":"APEX-SWE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c666bf86747f4194a293ae4cb54a1df1","models_covered":1},{"id":"apexbench","display_name":"ApexBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fc6b593f74ec40498ea09b9410e24246","models_covered":1},{"id":"arc","display_name":"ARC","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_572e3239dfa848069a47e0a6d9284200","models_covered":3},{"id":"arc_agi_1","display_name":"ARC-AGI-1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d5e2006f4e3e4803b6aa364787492ff0","models_covered":60},{"id":"arc_agi_2","display_name":"ARC-AGI-2","category":"reasoning","unit":"percent","scale_min":5,"scale_max":95,"higher_better":true,"weight_in_composite":0.0735,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":68},{"id":"arc_agi_3","display_name":"ARC-AGI-3","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_553b3da274594398a3713bcb7b3afa2d","models_covered":13},{"id":"arc_agi_3_standard_harness","display_name":"ARC-AGI-3 (Standard harness)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_599a92d424b54865bf01fd5d7cbeb036","models_covered":1},{"id":"arc_c","display_name":"ARC-c","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f469e4e959734d1cbf42fcdfd1a6f177","models_covered":5},{"id":"arc_challenge","display_name":"ARC-Challenge","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":4},{"id":"arc_e","display_name":"ARC-E","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2e6f4a879195426b9e174acc0bc1346a","models_covered":1},{"id":"arena_elo","display_name":"LMArena / Arena Elo","category":"composite","unit":"elo","scale_min":1279.63,"scale_max":1503.99,"higher_better":true,"weight_in_composite":0.0735,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":167},{"id":"arena_elo_document","display_name":"Arena Elo document","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":30},{"id":"arena_elo_image_to_webdev","display_name":"Arena Elo image_to_webdev","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":29},{"id":"arena_elo_search","display_name":"Arena Elo search","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":8},{"id":"arena_elo_vision","display_name":"Arena Elo vision","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":79},{"id":"arena_elo_webdev","display_name":"Arena Elo webdev","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":82},{"id":"arena_hard","display_name":"Arena-Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_129dee01585c4c0b9ef45f159395a156","models_covered":18},{"id":"arena_hard_v2","display_name":"Arena-Hard v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c157934da2b1494e95383b8c151c338e","models_covered":7},{"id":"arena_text_elo","display_name":"Arena Text Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":2},{"id":"arenahard","display_name":"ArenaHard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97c943d2c5ee4107a9a57d2adea4a5ad","models_covered":3},{"id":"artifactsbench","display_name":"ArtifactsBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8a69fb40f715478dbe3668b58472c516","models_covered":1},{"id":"artificial_analysis_coding_agent_index","display_name":"Artificial Analysis Coding Agent Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_acf3136ad9694471b137d965b747a7f6","models_covered":0},{"id":"artificial_analysis_finance_accounting_index","display_name":"Artificial Analysis Finance & Accounting Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"arxivmath","display_name":"ArXivMath","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5cf46f49f2b545f2a95b2b42f6f2991d","models_covered":4},{"id":"arxivmath_june_2026_49_problems","display_name":"ArxivMath (June 2026, 49 problems)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"attaq","display_name":"AttaQ","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"autobench","display_name":"AutoBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a79d8f61dd56498990b7266ec18cfb4b","models_covered":1},{"id":"automation_bench","display_name":"Automation-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":2},{"id":"automation_bench_partial_score","display_name":"automation_bench_partial_score","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c1f9150dcede44909ee388a3dc05d2ae","models_covered":35},{"id":"automationbench","display_name":"AutomationBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fa44f3cefc824f43ba6c89894e5d3dd1","models_covered":13},{"id":"automationbench_aa","display_name":"AutomationBench-AA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":3},{"id":"babyvision","display_name":"BabyVision","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":2},{"id":"bbh","display_name":"BBH","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":13},{"id":"bbq","display_name":"BBQ","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"bbq_ambig","display_name":"BBQ Ambig","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_496cde7b49d248459d3e7e1c562450da","models_covered":1},{"id":"bbq_ambiguous","display_name":"BBQ ambiguous","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"bbq_ambiguous_accuracy","display_name":"BBQ ambiguous accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"bbq_disambiguated","display_name":"BBQ disambiguated","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"bbq_disambiguated_accuracy","display_name":"BBQ disambiguated accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"benchcad","display_name":"BenchCAD","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"beyondaime","display_name":"BeyondAIME","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_28687023448c4de59b92d6744c6737e6","models_covered":2},{"id":"bfcl","display_name":"BFCL","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":7},{"id":"bfcl_v2","display_name":"BFCL v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_282bf64514de478b9fde1d211282288c","models_covered":1},{"id":"bfcl_v2_live","display_name":"BFCL V2 Live","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_93aaa1834ee7431193e8e19629a0b424","models_covered":2},{"id":"bfcl_v3","display_name":"BFCL v3","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":12},{"id":"bfcl_v3_tool_calling","display_name":"BFCL v3 (Tool Calling)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"bfcl_v4","display_name":"BFCL (v4)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"big_bench_hard","display_name":"BIG-Bench Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f469e4e959734d1cbf42fcdfd1a6f177","models_covered":1},{"id":"bigbench_extra_hard","display_name":"BigBench Extra Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":5},{"id":"bigbench_hard","display_name":"BigBench Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8396420d65094d77a85eb04704426e9d","models_covered":2},{"id":"bigbenchhard","display_name":"BigBenchHard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6014f4daf7fb44ec934b2ae474f34e8e","models_covered":1},{"id":"bigcodebench","display_name":"BigCodeBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":3},{"id":"biomysterybench","display_name":"Biomysterybench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":2},{"id":"biomysterybench_human_difficult","display_name":"BioMysteryBench (Human-difficult)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":2},{"id":"biomysterybench_human_solvable","display_name":"BioMysteryBench human-solvable","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"biosecbench_function","display_name":"Biosecbench Function","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"biosecbench_surveillance","display_name":"Biosecbench Surveillance","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"bird_sql","display_name":"Bird-SQL","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0e010c3d05ae48ea8826c4174c27aed9","models_covered":1},{"id":"birdbench","display_name":"BirdBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"bixbench","display_name":"BixBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_131a8a1728b44785bf474e46e254b8be","models_covered":2},{"id":"blink","display_name":"BLINK","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"blueprint_bench_2","display_name":"Blueprint-Bench 2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a60ebb3a51fe482db14bf1479d4f70ec","models_covered":1},{"id":"boolq","display_name":"BoolQ","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2e6f4a879195426b9e174acc0bc1346a","models_covered":1},{"id":"brokenarxiv","display_name":"Brokenarxiv","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":2},{"id":"browsecomp","display_name":"BrowseComp","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":27},{"id":"browsecomp_multi_agent","display_name":"BrowseComp, multi-agent","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"browsecomp_single_agent","display_name":"BrowseComp, single agent","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"browsecomp_zh","display_name":"BrowseComp-ZH","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":1},{"id":"business_knowledge_accuracy","display_name":"business knowledge accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"business_knowledge_hallucination","display_name":"business knowledge hallucination","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"c3_bench","display_name":"C3-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8a69fb40f715478dbe3668b58472c516","models_covered":1},{"id":"c_eval","display_name":"C-Eval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_047f4152277541e28f06797a7ed31fdc","models_covered":2},{"id":"cadgenbench","display_name":"Cadgenbench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"case_law","display_name":"Case Law","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0209f765bc4548f78ed2985f52273135","models_covered":1},{"id":"cathedralbench","display_name":"Cathedralbench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"character_editing_elo","display_name":"Character Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"chartography","display_name":"Chartography","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fc6b593f74ec40498ea09b9410e24246","models_covered":4},{"id":"chartqa","display_name":"ChartQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4eaa68da41964b1f84786f0dd048a38a","models_covered":6},{"id":"charxiv","display_name":"CharXiv","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d96417a9afb74afd8322737c2b9940c9","models_covered":1},{"id":"charxiv_reasoning","display_name":"CharXiv Reasoning","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":5},{"id":"charxiv_rq","display_name":"CharXiv RQ","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_025dafb136bb47369cd6e8cc4e9d7543","models_covered":2},{"id":"chess_puzzles","display_name":"Chess Puzzles","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"claw_eval","display_name":"Claw-Eval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_e468d8ceb73f4797b3bd0b95d08879f8","models_covered":1},{"id":"claweval_mm","display_name":"ClawEval-MM","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"code_arena","display_name":"Code Arena","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":1},{"id":"code_migration","display_name":"Code Migration","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":6},{"id":"codeforces_elo","display_name":"Codeforces Elo (LLM, human scale)","category":"code","unit":"elo","scale_min":800,"scale_max":2500,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":10},{"id":"codeforces_rating","display_name":"CodeForces rating","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_31f47c29c79a4a90bd4872872876f9e8","models_covered":1},{"id":"coding_index","display_name":"Coding Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a469d4cc82b24e61965b5a930a9ba5ff","models_covered":1},{"id":"collegemath","display_name":"CollegeMath","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aab24e5371704d9db079b045e5b11a05","models_covered":1},{"id":"commonsenseqa","display_name":"CommonsenseQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_572e3239dfa848069a47e0a6d9284200","models_covered":4},{"id":"complexfuncbench","display_name":"ComplexFuncBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8a69fb40f715478dbe3668b58472c516","models_covered":1},{"id":"corp_fin","display_name":"Corp Fin","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0209f765bc4548f78ed2985f52273135","models_covered":1},{"id":"cosmos_reason1","display_name":"Cosmos-Reason1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"countbench","display_name":"CountBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"covost","display_name":"CoVoST","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":1},{"id":"coworkbench","display_name":"CoWorkBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":3},{"id":"creativework","display_name":"CreativeWork","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"critpt","display_name":"CritPt","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":175},{"id":"cruxeval_o","display_name":"CRUXEval-O","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"cursorbench","display_name":"CursorBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a927f856add64bd994092110b766a8f2","models_covered":4},{"id":"cursorbench_v3_2","display_name":"CursorBench v3.2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c666bf86747f4194a293ae4cb54a1df1","models_covered":2},{"id":"cve_bench","display_name":"Cve Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"cwe_bench","display_name":"CWE-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7ac666dcfe3949c691f7cb9da4f69ca1","models_covered":1},{"id":"cwe_bench_v1","display_name":"CWE-bench v1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"cyberbench_v1_1","display_name":"CyberBench v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":4},{"id":"cybergym","display_name":"CyberGym","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":7},{"id":"deepmind_math","display_name":"DeepMind Math","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"deepsearch_qa","display_name":"DeepSearch QA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4ce546e34701489ca7c890d1a3a0096c","models_covered":1},{"id":"deepsearchqa","display_name":"DeepSearchQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":5},{"id":"deepswe","display_name":"DeepSWE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":29},{"id":"deepswe_1_0","display_name":"DeepSWE 1.0","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_474270a493514b6589ef766381031e54","models_covered":0},{"id":"deepswe_1_1","display_name":"DeepSWE 1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":0},{"id":"deepswe_v1_1","display_name":"DeepSWE v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":0},{"id":"design_arena_3d","display_name":"design_arena_3d","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":104},{"id":"design_arena_agenticgamedev","display_name":"design_arena_agenticgamedev","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":25},{"id":"design_arena_agentichtmlslides","display_name":"design_arena_agentichtmlslides","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":10},{"id":"design_arena_agenticslides","display_name":"design_arena_agenticslides","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":10},{"id":"design_arena_androidnative","display_name":"design_arena_androidnative","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":37},{"id":"design_arena_asciiart","display_name":"design_arena_asciiart","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":65},{"id":"design_arena_codecategories","display_name":"design_arena_codecategories","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":112},{"id":"design_arena_dataviz","display_name":"design_arena_dataviz","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":111},{"id":"design_arena_fullstack","display_name":"design_arena_fullstack","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":43},{"id":"design_arena_gamedev","display_name":"design_arena_gamedev","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":112},{"id":"design_arena_godotgamedev","display_name":"design_arena_godotgamedev","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":35},{"id":"design_arena_graphicdesign","display_name":"design_arena_graphicdesign","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":5},{"id":"design_arena_htmlslides","display_name":"design_arena_htmlslides","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":24},{"id":"design_arena_image","display_name":"design_arena_image","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":5},{"id":"design_arena_imageediting","display_name":"design_arena_imageediting","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":1},{"id":"design_arena_logo","display_name":"design_arena_logo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":5},{"id":"design_arena_mobileapps","display_name":"design_arena_mobileapps","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":43},{"id":"design_arena_pptxslides","display_name":"design_arena_pptxslides","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":9},{"id":"design_arena_python_pptxslides","display_name":"design_arena_python-pptxslides","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":27},{"id":"design_arena_svg","display_name":"design_arena_svg","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":81},{"id":"design_arena_uicomponent","display_name":"design_arena_uicomponent","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":110},{"id":"design_arena_webapps","display_name":"design_arena_webapps","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":44},{"id":"design_arena_website","display_name":"design_arena_website","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68998df3bf3f44729cc6558681a176bf","models_covered":119},{"id":"diagnosisarena_mcq","display_name":"DiagnosisArena-MCQ","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"docvqa","display_name":"DocVQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4eaa68da41964b1f84786f0dd048a38a","models_covered":6},{"id":"doodle_editing_elo","display_name":"Doodle Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"drop","display_name":"DROP","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b117e42664ba451c8a56cc67a74071b4","models_covered":9},{"id":"dsbench_hard","display_name":"DSBench-Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fc6b593f74ec40498ea09b9410e24246","models_covered":1},{"id":"ebr_bench","display_name":"Ebr Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":1},{"id":"eci","display_name":"ECI","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"eebench","display_name":"Eebench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"egoschema","display_name":"EgoSchema","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"emb","display_name":"Emb","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":5},{"id":"embspatialbench","display_name":"EmbSpatialBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"enterprise_ops_gym","display_name":"enterprise_ops_gym","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c1f9150dcede44909ee388a3dc05d2ae","models_covered":30},{"id":"erqa","display_name":"ERQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4b491fc5dc374246822a3d46b2e00025","models_covered":4},{"id":"eval_avg","display_name":"Eval+ Avg","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"expert_swe","display_name":"Expert-SWE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_131a8a1728b44785bf474e46e254b8be","models_covered":1},{"id":"exploitbench","display_name":"ExploitBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":3},{"id":"exploitbench_v8_bench_autonudge_capability","display_name":"ExploitBench v8-bench AutoNudge capability","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"exploitgym","display_name":"ExploitGym","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"extractbench","display_name":"ExtractBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bb0a1075db914805b47a6006e33f169e","models_covered":1},{"id":"facts_grounding","display_name":"FACTS Grounding","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0e010c3d05ae48ea8826c4174c27aed9","models_covered":1},{"id":"fairpoliticsqa","display_name":"FairPoliticsQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_127804d15e9d458886264b7c2a8554ae","models_covered":1},{"id":"finance_accounting_index","display_name":"Finance & Accounting Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"finance_agent","display_name":"Finance Agent","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4c932a77cb624e88b1eb83c143534b88","models_covered":3},{"id":"finance_agent_v1_1","display_name":"Finance Agent v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"finance_agent_v2","display_name":"Finance Agent v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6d1eabd36ecb488885533ee2704804da","models_covered":6},{"id":"flenqa","display_name":"FlenQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_33e67aa483c747659f28bbf4ae6d3217","models_covered":1},{"id":"fleurs","display_name":"FLEURS","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":1},{"id":"frames","display_name":"FRAMES","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6923465be3b7424f8e7a207e809d0dc0","models_covered":1},{"id":"frontier_bench_v0_1","display_name":"Frontier-Bench v0.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_553b3da274594398a3713bcb7b3afa2d","models_covered":1},{"id":"frontiercode_1_1_extended_score","display_name":"FrontierCode 1.1 Extended (score)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"frontiercode_1_1_main","display_name":"FrontierCode 1.1 Main","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":2},{"id":"frontiercode_1_1_main_score","display_name":"FrontierCode 1.1 Main (score)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"frontiercode_v1_1_extended","display_name":"FrontierCode v1.1 (Extended)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c666bf86747f4194a293ae4cb54a1df1","models_covered":2},{"id":"frontierfinance","display_name":"FrontierFinance","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":2},{"id":"frontiermath","display_name":"FrontierMath","category":"math","unit":"percent","scale_min":0,"scale_max":55,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":2},{"id":"frontiermath_tier_4","display_name":"FrontierMath Tier 4 (v2)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":2},{"id":"frontiermath_tiers_1_3","display_name":"FrontierMath Tiers 1-3","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_131a8a1728b44785bf474e46e254b8be","models_covered":1},{"id":"frontiermath_tiers_1_3_v2","display_name":"FrontierMath Tiers 1-3 (v2)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"frontierscience_research","display_name":"FrontierScience-Research","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":2},{"id":"frontierswe","display_name":"FrontierSWE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":5},{"id":"frontierswe_v2","display_name":"FrontierSWE v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"fullstackbench","display_name":"FullStackBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8a69fb40f715478dbe3668b58472c516","models_covered":1},{"id":"furniture_assembly","display_name":"Furniture Assembly","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":2},{"id":"gaia2","display_name":"Gaia2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4ce546e34701489ca7c890d1a3a0096c","models_covered":1},{"id":"gameworld","display_name":"GameWorld","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"gaokao2023en","display_name":"GaoKao2023en","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aab24e5371704d9db079b045e5b11a05","models_covered":1},{"id":"gdm_mrcr_v2","display_name":"GDM-MRCR v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6c3d8d40b1334e50b419a6d5f5788334","models_covered":1},{"id":"gdm_mrcr_v2_8_needle","display_name":"GDM-MRCR v2 (8-needle)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":1},{"id":"gdp_pdf","display_name":"Gdp Pdf","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":3},{"id":"gdpval","display_name":"GDPval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_f7be5cdfc6ed41c99f63660105f869ce","models_covered":1},{"id":"gdpval_aa","display_name":"GDPval-AA","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":12},{"id":"gdpval_aa_normalized","display_name":"gdpval_aa_normalized","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_94b6620d00e9430381a455dc60e828f0","models_covered":100},{"id":"gdpval_aa_v2","display_name":"GDPval-AA v2","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_553b3da274594398a3713bcb7b3afa2d","models_covered":14},{"id":"gdpval_aa_v2_elo","display_name":"GDPval-AA v2 Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"genebench_pro","display_name":"GeneBench Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_acf3136ad9694471b137d965b747a7f6","models_covered":1},{"id":"general_editing_elo","display_name":"General Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"general_t2i_elo","display_name":"General T2I Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"global_mmlu","display_name":"Global MMLU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_49645f65fcba45d39e5f52ab080c7386","models_covered":2},{"id":"global_mmlu_lite","display_name":"Global MMLU-Lite","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_33d010562a42487e9578cf616dfeec53","models_covered":8},{"id":"gpqa_diamond","display_name":"GPQA Diamond","category":"reasoning","unit":"percent","scale_min":25,"scale_max":96,"higher_better":true,"weight_in_composite":0.1471,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":288},{"id":"gpqa_main","display_name":"GPQA Main","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_93559fd129504808ae725d13d08b529b","models_covered":3},{"id":"graphwalks","display_name":"GraphWalks","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":3},{"id":"gray_swan_ipi_benchmark","display_name":"Gray Swan IPI Benchmark","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6db0fbcfe8ff4848b50db480a1be5711","models_covered":1},{"id":"gsm8k","display_name":"GSM8K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":18},{"id":"gsm_8k","display_name":"GSM-8K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_118237c52f154f449b89d1999110b526","models_covered":1},{"id":"gsm_plus","display_name":"GSM-Plus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_49645f65fcba45d39e5f52ab080c7386","models_covered":1},{"id":"gsm_symbolic","display_name":"GSM Symbolic","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"harvey_lab_criteria_pass","display_name":"harvey_lab_criteria_pass","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c1f9150dcede44909ee388a3dc05d2ae","models_covered":36},{"id":"harvey_lab_vals","display_name":"Harvey LAB (Vals)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c666bf86747f4194a293ae4cb54a1df1","models_covered":3},{"id":"harveys_legal_agent_benchmark","display_name":"Harvey's Legal Agent Benchmark","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":4},{"id":"healthbench","display_name":"HealthBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":4},{"id":"healthbench_consensus","display_name":"HealthBench Consensus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a8319c22a50e4818a64634d19511f43d","models_covered":1},{"id":"healthbench_hard","display_name":"HealthBench Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a8319c22a50e4818a64634d19511f43d","models_covered":2},{"id":"healthbench_professional","display_name":"HealthBench Professional","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a8319c22a50e4818a64634d19511f43d","models_covered":5},{"id":"hellaswag","display_name":"HellaSwag","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":11},{"id":"hiddenmath","display_name":"HiddenMath","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0e010c3d05ae48ea8826c4174c27aed9","models_covered":1},{"id":"hle","display_name":"Humanity's Last Exam","category":"reasoning","unit":"percent","scale_min":2,"scale_max":70,"higher_better":true,"weight_in_composite":0.1029,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":252},{"id":"hle_diamond","display_name":"HLE Diamond","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":3},{"id":"hle_no_tools","display_name":"HLE no tools","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":0},{"id":"hle_verified","display_name":"HLE-Verified","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":1},{"id":"hle_with_search","display_name":"HLE with search","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":0},{"id":"hle_with_tools","display_name":"HLE with tools","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_b81290bb2ab64e88bf29b71d8ba48b7b","models_covered":0},{"id":"hmmt","display_name":"HMMT","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":7},{"id":"hmmt_2026_feb","display_name":"HMMT 2026 Feb","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_652c7c2de97b4b489991fb719fa85b4a","models_covered":3},{"id":"hmmt_feb25","display_name":"HMMT Feb25","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"hmmt_feb_2025","display_name":"HMMT Feb 2025","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_195871e76b544cee9e330f942e1e8051","models_covered":1},{"id":"hmmt_feb_2026","display_name":"HMMT Feb 2026","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":2},{"id":"hmmt_nov_2025","display_name":"HMMT Nov 2025","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":1},{"id":"hotpotqa","display_name":"HotpotQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0d2165b50b56406db56047fe17ac199c","models_covered":1},{"id":"humaneval","display_name":"HumanEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":19},{"id":"humaneval_plus","display_name":"HumanEval Plus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_eb1342ae95b443ed98b6f405b52f2d37","models_covered":1},{"id":"humanevalplus","display_name":"HumanEvalPlus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_33e67aa483c747659f28bbf4ae6d3217","models_covered":1},{"id":"humanitys_last_exam_with_tools","display_name":"Humanity's Last Exam (with tools)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_652c7c2de97b4b489991fb719fa85b4a","models_covered":0},{"id":"if_bench","display_name":"IF-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_1785de774a2b440ea167d1ddd5e1b591","models_covered":1},{"id":"ifbench","display_name":"IFBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":175},{"id":"ifbench_prompt","display_name":"IFBench (prompt)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"ifeval","display_name":"IFEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":42},{"id":"ifeval_avg","display_name":"IFEval Avg","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"ifeval_instruction_level_accuracy","display_name":"IFEval (instruction-level accuracy)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"ifeval_prompt_level_accuracy","display_name":"IFEval (prompt-level accuracy)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"imo","display_name":"IMO 2025","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"imo_answerbench","display_name":"IMO-AnswerBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_652c7c2de97b4b489991fb719fa85b4a","models_covered":1},{"id":"imo_proofbench_advanced","display_name":"IMO-ProofBench Advanced","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"imo_proofbench_basic","display_name":"IMO-ProofBench Basic","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"imoanswerbench","display_name":"IMOAnswerBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":3},{"id":"include","display_name":"INCLUDE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"infographicvqa","display_name":"InfographicVQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":2},{"id":"ioi","display_name":"Ioi","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":5},{"id":"it_bench_sre","display_name":"it_bench_sre","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5c9f9bb0d3484940bdb86d6e7e014bcf","models_covered":31},{"id":"jobbench","display_name":"JobBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":8},{"id":"judgebench","display_name":"JudgeBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_93aaa1834ee7431193e8e19629a0b424","models_covered":1},{"id":"judgebench_overall_score","display_name":"JudgeBench Overall Score","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a469d4cc82b24e61965b5a930a9ba5ff","models_covered":1},{"id":"kat_claw_bench","display_name":"KAT Claw Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8daae255e67c4e4c9364b5b22c3ec58e","models_covered":1},{"id":"kat_code_bench","display_name":"KAT Code Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8daae255e67c4e4c9364b5b22c3ec58e","models_covered":1},{"id":"kernelbench_cuda","display_name":"Kernelbench CUDA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_599a92d424b54865bf01fd5d7cbeb036","models_covered":1},{"id":"kernelbench_cuda_canonical_board_rtx_pro_6000","display_name":"KernelBench CUDA (canonical board, RTX PRO 6000)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"kernelbench_cuda_glm_5_2_fused_moe_rtx_pro_6000","display_name":"KernelBench CUDA GLM-5.2 Fused MoE (RTX PRO 6000)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"kernelbench_cuda_grid_mingru_sps_rtx_pro_6000","display_name":"KernelBench CUDA Grid + MinGRU SPS (RTX PRO 6000)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"kernelbench_cuda_megaqwen_decode_rtx_pro_6000","display_name":"KernelBench CUDA MegaQwen Decode (RTX PRO 6000)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"kernelbench_grid_mingru_sps","display_name":"Kernelbench Grid + MinGRU SPS","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_599a92d424b54865bf01fd5d7cbeb036","models_covered":1},{"id":"kernelbench_hard","display_name":"KernelBench Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f8e2332c4c034edba09fea1b7882b8b5","models_covered":1},{"id":"kernelbench_mega","display_name":"Kernelbench Mega","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_599a92d424b54865bf01fd5d7cbeb036","models_covered":1},{"id":"kernelbench_mega_kimi_linear_decode_rtx_pro_6000","display_name":"KernelBench Mega (Kimi-Linear Decode, RTX PRO 6000)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"kina","display_name":"KINA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fd2f13c110594b94ae58f678edcca06a","models_covered":1},{"id":"lab_bench_practical","display_name":"Lab Bench Practical","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"labbench_2","display_name":"LABBench 2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"latchbio_capabilities","display_name":"Latchbio Capabilities","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"latchbio_singlecellbench","display_name":"LatchBio SingleCellBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"latchbio_spatialbench_verified","display_name":"LatchBio SpatialBench Verified","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_366215880a6a48db9efc6198ca848c0e","models_covered":1},{"id":"lcr","display_name":"LCR","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"legal_bench","display_name":"Legal Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":4},{"id":"legal_research_bench","display_name":"Legal Research Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":5},{"id":"ling_3_0_flash_vl_agentic_index","display_name":"Ling-3.0-flash-VL Agentic Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"ling_3_0_flash_vl_coding_index","display_name":"Ling-3.0-flash-VL Coding Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"ling_3_0_flash_vl_intelligence_index","display_name":"Ling-3.0-flash-VL Intelligence Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"linguasafe","display_name":"LinguaSafe","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"livebench","display_name":"LiveBench","category":"composite","unit":"index","scale_min":40,"scale_max":85,"higher_better":true,"weight_in_composite":0.1103,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":81},{"id":"livecodebench","display_name":"LiveCodeBench v6","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_15dad7768a114aca88498cbfff0fd7a5","models_covered":149},{"id":"lmarena_text_arena_elo","display_name":"LMArena Text Arena Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":3},{"id":"loft_text_retrieval","display_name":"LOFT Text Retrieval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":2},{"id":"logicvista","display_name":"LogicVista","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b9316cd9218541edba4a54b3f349acd3","models_covered":1},{"id":"long_context_reasoning","display_name":"Long Context Reasoning","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_3a3bd5282f83458e9783b742cd434d55","models_covered":3},{"id":"long_horizon_terminal_bench","display_name":"Long-Horizon-Terminal-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d7b4ee6078684e3a9a7eb5ff12f0c81d","models_covered":1},{"id":"longbench_v2","display_name":"LongBench v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":2},{"id":"longcodebench","display_name":"LongCodeBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_1785de774a2b440ea167d1ddd5e1b591","models_covered":1},{"id":"longcodebench_1m","display_name":"LongCodeBench 1M","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d8b288e8a4314a718d62098d4463f8f4","models_covered":1},{"id":"lvbench","display_name":"LVBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":4},{"id":"math","display_name":"MATH","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":24},{"id":"math500","display_name":"MATH500","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_93aaa1834ee7431193e8e19629a0b424","models_covered":3},{"id":"math_500","display_name":"MATH-500","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_15dad7768a114aca88498cbfff0fd7a5","models_covered":21},{"id":"math_hard","display_name":"MATH hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_abc78461c41b4ac8b5d1f69d3ee42e4b","models_covered":2},{"id":"math_index","display_name":"Math Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a469d4cc82b24e61965b5a930a9ba5ff","models_covered":1},{"id":"math_lvl_5","display_name":"MATH Lvl 5","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":3},{"id":"math_vision","display_name":"MATH-Vision","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":6},{"id":"matharena_apex_2025","display_name":"MathArena Apex 2025","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5cf46f49f2b545f2a95b2b42f6f2991d","models_covered":1},{"id":"mathvision","display_name":"MathVision","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b9316cd9218541edba4a54b3f349acd3","models_covered":3},{"id":"mathvista","display_name":"MathVista","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4eaa68da41964b1f84786f0dd048a38a","models_covered":6},{"id":"mbpp","display_name":"MBPP","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":10},{"id":"mbpp_evalplus","display_name":"MBPP EvalPlus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_282bf64514de478b9fde1d211282288c","models_covered":1},{"id":"mbpp_plus","display_name":"MBPP Plus","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_eb1342ae95b443ed98b6f405b52f2d37","models_covered":1},{"id":"mceval","display_name":"McEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_42f387694fac489b8a51914a4e11c794","models_covered":1},{"id":"mcp_atlas","display_name":"MCP Atlas","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":12},{"id":"mdeval","display_name":"MdEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_42f387694fac489b8a51914a4e11c794","models_covered":1},{"id":"medcode","display_name":"Medcode","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":5},{"id":"medethicalign","display_name":"MedEthicAlign","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"medmcqa","display_name":"MedMCQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"medqa_usmle","display_name":"MedQA-USMLE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"medscribe","display_name":"Medscribe","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":5},{"id":"medxpertqa_mm","display_name":"MedXPertQA MM","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":4},{"id":"medxpertqa_text","display_name":"MedXpertQA-Text","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"mgsm","display_name":"MGSM","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":7},{"id":"milu","display_name":"Milu","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":1},{"id":"mimo_code_bench","display_name":"MiMo Code Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"mimo_cyber_bench","display_name":"MiMo Cyber Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"mimo_v2_6_pro_aa_lcr","display_name":"MiMo-V2.6-Pro AA-LCR","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_aa_omniscience_accuracy","display_name":"MiMo-V2.6-Pro AA-Omniscience Accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_aa_omniscience_non_hallucination_rate","display_name":"MiMo-V2.6-Pro AA-Omniscience Non-Hallucination Rate","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_critpt","display_name":"MiMo-V2.6-Pro CritPt","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_gdpval_aa","display_name":"MiMo-V2.6-Pro GDPval-AA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_hle","display_name":"MiMo-V2.6-Pro HLE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_intelligence_index","display_name":"MiMo-V2.6-Pro Intelligence Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_v2_6_pro_scicode","display_name":"MiMo-V2.6-Pro SciCode","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"mimo_visualcoding","display_name":"MiMo VisualCoding","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"minerva","display_name":"Minerva","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"minerva_math","display_name":"Minerva Math","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":3},{"id":"mixeval_hard","display_name":"MixEval Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_49645f65fcba45d39e5f52ab080c7386","models_covered":1},{"id":"mlcr_overall","display_name":"mlcr_overall","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c1f9150dcede44909ee388a3dc05d2ae","models_covered":41},{"id":"mle_bench","display_name":"MLE-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6c3d8d40b1334e50b419a6d5f5788334","models_covered":1},{"id":"mle_bench_lite","display_name":"MLE-Bench Lite","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7531f4d597aa43ddb248859c44f27611","models_covered":1},{"id":"mls_bench_lite","display_name":"MLS-Bench-Lite","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":2},{"id":"mlvu","display_name":"MLVU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"mm_claw","display_name":"MM Claw","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7531f4d597aa43ddb248859c44f27611","models_covered":1},{"id":"mm_mtbench","display_name":"MM MTBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_23d9f6edc5c14b038816d74071a4dea8","models_covered":1},{"id":"mm_safetybench","display_name":"MM-SafetyBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"mmau","display_name":"MMAU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_025dafb136bb47369cd6e8cc4e9d7543","models_covered":1},{"id":"mmlongbench_128k","display_name":"MMLongBench-128K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fd2f13c110594b94ae58f678edcca06a","models_covered":1},{"id":"mmlu","display_name":"MMLU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":29},{"id":"mmlu_pro","display_name":"MMLU-Pro","category":"knowledge","unit":"percent","scale_min":40,"scale_max":100,"higher_better":true,"weight_in_composite":0.1103,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":173},{"id":"mmlu_prox","display_name":"MMLU-ProX","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_eec8957c31744a3c81cf76968cce1c36","models_covered":1},{"id":"mmlu_prox_lite_ibm","display_name":"MMLU-ProX lite (IBM)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"mmlu_redux","display_name":"MMLU-Redux","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97c943d2c5ee4107a9a57d2adea4a5ad","models_covered":15},{"id":"mmmlu","display_name":"MMMLU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":9},{"id":"mmmu","display_name":"MMMU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4eaa68da41964b1f84786f0dd048a38a","models_covered":7},{"id":"mmmu_pro","display_name":"MMMU-Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":106},{"id":"mmstar","display_name":"MMStar","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4492b4b1a6fe4d84b6d3770f93bb7b2f","models_covered":1},{"id":"mmvu","display_name":"MMVU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"mobile_actions","display_name":"Mobile Actions","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"mobileworld","display_name":"MobileWorld","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"mortgagetax","display_name":"MortgageTax","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0209f765bc4548f78ed2985f52273135","models_covered":1},{"id":"mrcr","display_name":"MRCR (multi-needle retrieval)","category":"other","unit":"percent","scale_min":20,"scale_max":97,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":6},{"id":"mrcr_1m","display_name":"MRCR 1M","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c59159c0343e49c096da469a4161a2e2","models_covered":1},{"id":"mrcr_256k_512k","display_name":"MRCR 256K-512K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_09f703cd057f467692c632c989f68b58","models_covered":1},{"id":"mrcr_v2_8_needle","display_name":"MRCR v2 (8-needle)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":2},{"id":"mrcr_v2_8_needle_128k","display_name":"MRCR v2 8 needle 128k","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_21ea043aa0d647e0b594a78b711644f0","models_covered":1},{"id":"mrcr_v2_8_needle_128k_average","display_name":"MRCR v2 8 needle 128k (average)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":3},{"id":"mtbench_avg","display_name":"MTBench Avg","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"mtob_eng_kgv","display_name":"MTOB (eng->kgv)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":2},{"id":"mtob_kgv_eng","display_name":"MTOB (kgv->eng)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":1},{"id":"multi_character_editing_elo","display_name":"Multi Character Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"multi_input_editing_elo","display_name":"Multi Input Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"multi_policy_accuracy","display_name":"Multi-policy accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"multi_product_editing_elo","display_name":"Multi Product Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"multi_swe_bench","display_name":"Multi-SWE-bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7531f4d597aa43ddb248859c44f27611","models_covered":1},{"id":"multichallenge","display_name":"MultiChallenge","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d8b288e8a4314a718d62098d4463f8f4","models_covered":1},{"id":"multiif","display_name":"MultiIF","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68ba010710a0403caedc8b612420d5e3","models_covered":4},{"id":"multilingual_mmlu","display_name":"Multilingual MMLU","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4eaa68da41964b1f84786f0dd048a38a","models_covered":3},{"id":"multipl_e","display_name":"MultiPL-E","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9cb4e004675749079014377f68f26d29","models_covered":3},{"id":"multipl_e_humaneval","display_name":"MultiPL-E HumanEval","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_9c1fc86de31b467badcaeb6b68cb1153","models_covered":1},{"id":"multiple","display_name":"MULTIPLE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"musr","display_name":"MuSR","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":6},{"id":"mvbench","display_name":"MVBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4492b4b1a6fe4d84b6d3770f93bb7b2f","models_covered":2},{"id":"mysterymechanism","display_name":"Mysterymechanism","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":3},{"id":"naturalquestions","display_name":"NaturalQuestions","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_438a8c7ec05149b9904338625ef1bf48","models_covered":1},{"id":"next_qa","display_name":"NExT-QA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"nl2repo","display_name":"NL2Repo","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":4},{"id":"nl2repo_bench","display_name":"NL2Repo-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":5},{"id":"object_environment_editing_elo","display_name":"Object Environment Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"ocr_reasoning","display_name":"OCR-Reasoning","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a13211de6c0a4398bf54b5b63e097761","models_covered":1},{"id":"ocrbench","display_name":"OCRBench","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4492b4b1a6fe4d84b6d3770f93bb7b2f","models_covered":2},{"id":"ocrbench_v2","display_name":"OCRBench v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_a13211de6c0a4398bf54b5b63e097761","models_covered":1},{"id":"officeqa","display_name":"Officeqa","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":1},{"id":"officeqa_pro","display_name":"Officeqa Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":2},{"id":"official_a","display_name":"Official A","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_599a92d424b54865bf01fd5d7cbeb036","models_covered":1},{"id":"ojbench","display_name":"OJBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68ba010710a0403caedc8b612420d5e3","models_covered":2},{"id":"olympiadbench","display_name":"OlympiadBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8a69fb40f715478dbe3668b58472c516","models_covered":2},{"id":"omni_math","display_name":"Omni-MATH","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_76c09e2bfa10439f8a5c27910779ba35","models_covered":1},{"id":"omnidoc","display_name":"OmniDoc","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":2},{"id":"omnidocbench","display_name":"OmniDocBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4ce546e34701489ca7c890d1a3a0096c","models_covered":1},{"id":"omnidocbench_1_5","display_name":"OmniDocBench 1.5","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":3},{"id":"omnimath","display_name":"OmniMath","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_33e67aa483c747659f28bbf4ae6d3217","models_covered":1},{"id":"online_mind2web","display_name":"Online-Mind2Web","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6d1eabd36ecb488885533ee2704804da","models_covered":1},{"id":"openai_internal_multi_policy_moderation_accuracy","display_name":"OpenAI internal multi-policy moderation accuracy","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fabbc8b6d4b74f5095713f8300d109a8","models_covered":1},{"id":"openai_moderation_2022","display_name":"OpenAI Moderation 2022","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"openai_moderation_2022_f1","display_name":"OpenAI Moderation (2022) F1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fabbc8b6d4b74f5095713f8300d109a8","models_covered":1},{"id":"openbookqa","display_name":"OpenBookQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_572e3239dfa848069a47e0a6d9284200","models_covered":5},{"id":"osworld","display_name":"OSWorld","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5a7b2bf97fed4a3cb01efdb8d503d9b4","models_covered":25},{"id":"osworld_2_0","display_name":"OSWorld-2.0","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_5f55387d72cd481ebb4a7dc5d6f940e5","models_covered":0},{"id":"osworld_2_0_v2026_08_08_offline_set_partial_score","display_name":"OSWorld 2.0 (v2026.08.08, offline set, partial score)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"osworld_g","display_name":"OSWorld-G","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":2},{"id":"osworld_verified","display_name":"OSWorld-Verified","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_f8e2332c4c034edba09fea1b7882b8b5","models_covered":0},{"id":"otis_mock_aime_2024_2025","display_name":"OTIS Mock AIME 2024-2025","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"ovobench","display_name":"OVOBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"paperbench","display_name":"PaperBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":2},{"id":"perception_test","display_name":"Perception Test","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"personqa","display_name":"PersonQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"phibench","display_name":"PhiBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b117e42664ba451c8a56cc67a74071b4","models_covered":1},{"id":"phybench","display_name":"PhyBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fa88a610ad23483087d24af0b402fafa","models_covered":2},{"id":"pinchbench","display_name":"PinchBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_407580b701a043a38dfb38f539a26607","models_covered":2},{"id":"piqa","display_name":"PIQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f469e4e959734d1cbf42fcdfd1a6f177","models_covered":5},{"id":"plawbench","display_name":"PLawBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"polyguard","display_name":"PolyGuard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"polymath","display_name":"PolyMATH","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68ba010710a0403caedc8b612420d5e3","models_covered":3},{"id":"posttrainbench","display_name":"PostTrainBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":2},{"id":"prbench_finance","display_name":"PRBench-Finance","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"prbench_legal","display_name":"PRBench-Legal","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"profbench","display_name":"ProfBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"programbench","display_name":"ProgramBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":8},{"id":"programbench_almost_solved","display_name":"ProgramBench (Almost Solved)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"proofbench","display_name":"ProofBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_45cf67d420f54d08b826c6a784ae397c","models_covered":4},{"id":"proofbench_v1_1","display_name":"ProofBench v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":3},{"id":"proteingym_hard","display_name":"ProteinGym Hard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"protocolqa_open_ended","display_name":"Protocolqa Open Ended","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"public_benefits_bench","display_name":"Public Benefits Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":2},{"id":"public_benefits_bench_v1_1","display_name":"Public Benefits Bench v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"qwenqoderbench","display_name":"QwenQoderBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"qwenreactbench","display_name":"QwenReactBench","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"qwensvgbench","display_name":"QwenSVGBench","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"qwenswebench","display_name":"QwenSWEBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"qwenswebench_v2","display_name":"QwenSWEBench V2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"real_world_vulnerability_discovery_20_languages","display_name":"Real world vulnerability discovery (20 languages)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6db0fbcfe8ff4848b50db480a1be5711","models_covered":1},{"id":"realworldqa","display_name":"RealWorldQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4b491fc5dc374246822a3d46b2e00025","models_covered":2},{"id":"redisqa","display_name":"ReDisQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_8b6e066584eb4d1d93a29fd4f9f67362","models_covered":1},{"id":"refcoco","display_name":"RefCOCO","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"refcoco_a","display_name":"RefCOCO-A","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"refcoco_b","display_name":"RefCOCO-B","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"refspatialbench","display_name":"RefSpatialBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"riemannbench","display_name":"RiemannBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"robospatial_pointing","display_name":"RoboSpatial Pointing","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"robospatial_vqa","display_name":"RoboSpatial VQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"ruler","display_name":"RULER","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f7be5cdfc6ed41c99f63660105f869ce","models_covered":3},{"id":"ruler_128k","display_name":"RULER 128K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"ruler_64k","display_name":"RULER 64K","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"sage","display_name":"SAGE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4c932a77cb624e88b1eb83c143534b88","models_covered":5},{"id":"salad_bench","display_name":"SALAD-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":2},{"id":"scicode","display_name":"SciCode","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":195},{"id":"screenspot","display_name":"ScreenSpot","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4492b4b1a6fe4d84b6d3770f93bb7b2f","models_covered":1},{"id":"screenspot_pro","display_name":"ScreenSpot-Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2febc58fd4a64b58b24e49b649573bde","models_covered":4},{"id":"sec_bench_pro","display_name":"SEC Bench Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":2},{"id":"simpleqa","display_name":"SimpleQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d96417a9afb74afd8322737c2b9940c9","models_covered":17},{"id":"skillsbench","display_name":"SkillsBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":4},{"id":"social_media_trends_editing_elo","display_name":"Social Media Trends Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"social_media_trends_t2i_elo","display_name":"Social Media Trends T2I Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"spreadsheetbench_2","display_name":"SpreadsheetBench 2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"spreadsheetbench_v1","display_name":"SpreadSheetBench-v1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"sre_bench","display_name":"SRE Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":3},{"id":"stylization_elo","display_name":"Stylization Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"superchem","display_name":"SUPERChem","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_5cf46f49f2b545f2a95b2b42f6f2991d","models_covered":1},{"id":"supergpqa","display_name":"SuperGPQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_c157934da2b1494e95383b8c151c338e","models_covered":8},{"id":"swe_atlas","display_name":"SWE Atlas","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_801af9927a3a46269acc9c832171c20e","models_covered":3},{"id":"swe_atlas_qna","display_name":"SWE Atlas Qna","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":3},{"id":"swe_bench","display_name":"SWE-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4f83241a7eda4f5c9c28b9e12028ed54","models_covered":1},{"id":"swe_bench_multilingual","display_name":"SWE-bench Multilingual","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_15dad7768a114aca88498cbfff0fd7a5","models_covered":17},{"id":"swe_bench_multimodal","display_name":"SWE Bench Multimodal","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":1},{"id":"swe_bench_pro","display_name":"SWE-bench Pro","category":"code","unit":"percent","scale_min":5,"scale_max":65,"higher_better":true,"weight_in_composite":0,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":58},{"id":"swe_bench_verified","display_name":"SWE-bench Verified","category":"code","unit":"percent","scale_min":10,"scale_max":100,"higher_better":true,"weight_in_composite":0.1471,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":110},{"id":"swe_fficiency","display_name":"SWE-fficiency","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f8e2332c4c034edba09fea1b7882b8b5","models_covered":1},{"id":"swe_lancer_ic_swe","display_name":"SWE-Lancer IC SWE","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_777e09aae04a48f3bc63eda13c64f254","models_covered":1},{"id":"swe_marathon","display_name":"SWE-Marathon","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":6},{"id":"swe_mm","display_name":"SWE-MM","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":2},{"id":"swe_multilingual","display_name":"SWE Multilingual","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_28687023448c4de59b92d6744c6737e6","models_covered":1},{"id":"swefish","display_name":"SWEFish","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"sysbench","display_name":"SysBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_76c09e2bfa10439f8a5c27910779ba35","models_covered":2},{"id":"tau2_airline","display_name":"Tau2 airline","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":0},{"id":"tau2_average_over_3","display_name":"Tau2 (average over 3)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_8af334977ff34304807c6f587f5abffa","models_covered":0},{"id":"tau2_bench_retail","display_name":"tau2-bench (Retail)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":0},{"id":"tau2_bench_telecom","display_name":"tau2-bench (Telecom)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_25639cafaa534a87b7332e203f141975","models_covered":0},{"id":"tau2_retail","display_name":"Tau2 retail","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":0},{"id":"tau2_telecom","display_name":"Tau2 telecom","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_57858c93b69d4f50be7228ce5954a748","models_covered":0},{"id":"tau3_banking","display_name":"tau3-Banking","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_3a3bd5282f83458e9783b742cd434d55","models_covered":1},{"id":"tau3_bench","display_name":"tau3-bench","category":"agentic","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"admin","first_seen_run_id":null,"models_covered":1},{"id":"tau3_bench_avg","display_name":"τ³-bench (AVG)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cf71e14d26914696a09b7b6f0b29b27e","models_covered":1},{"id":"tau3_bench_banking","display_name":"tau3-bench (Banking)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_407580b701a043a38dfb38f539a26607","models_covered":9},{"id":"tau3_bench_telecom","display_name":"tau3-bench (Telecom)","category":"agentic","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"admin","first_seen_run_id":null,"models_covered":2},{"id":"tau3_telecom","display_name":"tau3-Telecom","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4c932a77cb624e88b1eb83c143534b88","models_covered":1},{"id":"tau_bench","display_name":"tau-bench / tau2-bench","category":"agentic","unit":"percent","scale_min":20,"scale_max":95,"higher_better":true,"weight_in_composite":0.0442,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":192},{"id":"tau_bench_banking","display_name":"tau_bench_banking","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_94b6620d00e9430381a455dc60e828f0","models_covered":96},{"id":"tau_bench_retail","display_name":"tau-bench Retail","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_6c738b9f3a2d4a97b22b79e5db77ddfa","models_covered":0},{"id":"tau_bench_v3_average","display_name":"tau-bench V3 Average","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f7be5cdfc6ed41c99f63660105f869ce","models_covered":1},{"id":"tax_agent_bench","display_name":"Tax Agent Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":5},{"id":"tempcompass","display_name":"TempCompass","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"terminal_bench","display_name":"Terminal-Bench (Hard)","category":"agentic","unit":"percent","scale_min":5,"scale_max":90,"higher_better":true,"weight_in_composite":0.0441,"status":"core","origin":"seed","first_seen_run_id":null,"models_covered":232},{"id":"terminal_bench_2_1_vals","display_name":"Terminal-Bench 2.1 (Vals)","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"terminal_bench_4_0","display_name":"Terminal-Bench 4.0","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"deprecated","origin":"auto","first_seen_run_id":"run_b743334c36e74035a3b3c99d2fbb86b8","models_covered":0},{"id":"terminal_bench_science","display_name":"Terminal-Bench-Science","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fa44f3cefc824f43ba6c89894e5d3dd1","models_covered":7},{"id":"terminal_bench_science_0_1","display_name":"Terminal-Bench-Science 0.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b743334c36e74035a3b3c99d2fbb86b8","models_covered":3},{"id":"text_editing_elo","display_name":"Text Editing Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"tomato","display_name":"TOMATO","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":2},{"id":"tool_decathlon","display_name":"Tool-Decathlon","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_bf713be07f1c4f6781df4a9b64ce09c5","models_covered":2},{"id":"toolathlon","display_name":"Toolathlon","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d5e2006f4e3e4803b6aa364787492ff0","models_covered":7},{"id":"toolathlon_verified","display_name":"Toolathlon Verified","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":11},{"id":"toxicchat","display_name":"ToxicChat","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7a3458f60e154a0ebff41b4ff6a24883","models_covered":1},{"id":"toxicchat_f1","display_name":"ToxicChat F1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fabbc8b6d4b74f5095713f8300d109a8","models_covered":1},{"id":"toxigen","display_name":"Toxigen","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_496cde7b49d248459d3e7e1c562450da","models_covered":1},{"id":"triviaqa","display_name":"TriviaQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_572e3239dfa848069a47e0a6d9284200","models_covered":6},{"id":"truthfulqa","display_name":"TruthfulQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":6},{"id":"tulu3_safety_eval_avg","display_name":"Tulu3 Safety Eval Avg","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_199699ef9abb492e825cd5e67ed4501d","models_covered":1},{"id":"usamo_2026","display_name":"USAMO 2026","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_ef9b4984839b4ab5bb4343f87af67480","models_covered":1},{"id":"vals_finance_agent_v2","display_name":"Vals Finance Agent v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":1},{"id":"vals_index","display_name":"Vals Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_2e5a562a40d64392a3a803438b1da5c3","models_covered":8},{"id":"vals_multimodal_index","display_name":"Vals Multimodal Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_0209f765bc4548f78ed2985f52273135","models_covered":1},{"id":"vals_rsi_index","display_name":"Vals Rsi Index","category":"other","unit":"index","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":2},{"id":"vct","display_name":"Vct","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"vibe_code_bench","display_name":"Vibe Code Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_b4dff29e984149d2a2058f09ba55b63e","models_covered":3},{"id":"vibe_code_bench_1_100","display_name":"Vibe Code Bench 1 100","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":1},{"id":"vibe_code_bench_v1_1","display_name":"Vibe Code Bench v1.1","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f075fa67af7b4ec7873cceb3d715f171","models_covered":3},{"id":"vibe_pro","display_name":"VIBE-Pro","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7531f4d597aa43ddb248859c44f27611","models_covered":1},{"id":"video_mme","display_name":"Video-MME","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_f8e2332c4c034edba09fea1b7882b8b5","models_covered":4},{"id":"videohallucer","display_name":"VideoHallucer","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"videomme","display_name":"VideoMME","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fd2f13c110594b94ae58f678edcca06a","models_covered":1},{"id":"videosimpleqa","display_name":"VideoSimpleQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"vision2web","display_name":"Vision2Web","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_4b491fc5dc374246822a3d46b2e00025","models_covered":3},{"id":"visual_design_t2i_elo","display_name":"Visual Design T2I Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_074aeec360a54414826536d5c35dc3cc","models_covered":1},{"id":"vlguard","display_name":"VLGuard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"voicebench","display_name":"VoiceBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_025dafb136bb47369cd6e8cc4e9d7543","models_covered":1},{"id":"vqa_v2","display_name":"VQA-v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_519e7de29aa54015827d548919d40783","models_covered":1},{"id":"vsi_bench","display_name":"VSI-Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_908db40ad08d44228fdcb01f4ee2ad7e","models_covered":1},{"id":"wandr","display_name":"Wandr","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_90e29a1a531f4ac58cd684e94d71b97b","models_covered":1},{"id":"webarena_verified","display_name":"WebArena-Verified","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":2},{"id":"webtest","display_name":"WebTest","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":2},{"id":"widesearch","display_name":"WideSearch","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":4},{"id":"wildbench","display_name":"WildBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_94e056bd14a14c68998308fa2fa0c463","models_covered":5},{"id":"wildbench_v2","display_name":"Wildbench v2","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_eb1342ae95b443ed98b6f405b52f2d37","models_covered":1},{"id":"wildclawbench","display_name":"WildClawBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_aff8de7dba644d48940dfb9eb3d32d96","models_covered":6},{"id":"wildguard","display_name":"WildGuard","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"winogrande","display_name":"Winogrande","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_d705283605f5404eb26540a36bb3d460","models_covered":8},{"id":"wmdp_bio","display_name":"Wmdp Bio","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"wmdp_chem","display_name":"Wmdp Chem","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"wmdp_cyber","display_name":"Wmdp Cyber","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_97d9de8e45c846f28b6d5d8b27d512b8","models_covered":1},{"id":"workarena_elo","display_name":"WorkArena Elo","category":"other","unit":"elo","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7fc23edf575148a083cb15bfba2b88d7","models_covered":1},{"id":"workspace_bench","display_name":"Workspace Bench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fd2f13c110594b94ae58f678edcca06a","models_covered":1},{"id":"workspacebench","display_name":"WorkSpaceBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6028ac39c0fb4d9a90025f0eba1e265f","models_covered":1},{"id":"worldvqa","display_name":"WorldVQA","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"writingbench","display_name":"WritingBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_68ba010710a0403caedc8b612420d5e3","models_covered":3},{"id":"xdailybench","display_name":"xDailyBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_7d9a7c8edc82492584a1771d1e102ec4","models_covered":1},{"id":"xstest","display_name":"XSTest","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_6045fccb592d48c0b7f7c3b41cdd25e5","models_covered":1},{"id":"zebralogic","display_name":"ZebraLogic","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_cc28075b1de84a93a408d60c29c5416f","models_covered":2},{"id":"zerobench","display_name":"ZeroBench","category":"other","unit":"percent","scale_min":null,"scale_max":null,"higher_better":true,"weight_in_composite":0,"status":"provisional","origin":"auto","first_seen_run_id":"run_fc6b593f74ec40498ea09b9410e24246","models_covered":2}],"vram_frontier_fit":{"method":"frontier-residual-v1","n":132,"r2":0.646,"resid_sd":1.211,"beta0":3.671,"beta1":0.737,"beta2_moe":1.489,"fitted_over":"current released/preview open-weight ranked models with an INT8 estimate","caveat":"Efficiency relative to a frontier FIT ON THIS INDEX’S OWN FLEET (n and r2 stated in vram_frontier_fit): expected INT8 GB for the model’s intelligence and architecture class, divided by its actual INT8 GB. Above 1 = leaner than the frontier predicts. Refit every render; values move as the fleet moves. Sources: documentation/research/R11 (2026-08-19)."},"index_subset_bias_fit":{"method":"subset-knn-v1","peer_min_coverage":0.8,"k":10,"min_peers":8,"peers":60,"benchmark_offsets":{"aime":{"mean":18.87,"n":39},"arc_agi_2":{"mean":-27.73,"n":37},"arena_elo":{"mean":1.62,"n":57},"gpqa_diamond":{"mean":15.84,"n":60},"hle":{"mean":-26.63,"n":60},"livebench":{"mean":-0.72,"n":30},"mmlu_pro":{"mean":7.53,"n":59},"swe_bench_verified":{"mean":0.98,"n":58},"tau_bench":{"mean":9.47,"n":55},"terminal_bench":{"mean":-14.66,"n":60}},"caveat":"Mean (subset index − own index) over the K full-coverage peers nearest on this model's own benchmark subset. Positive = the subset flatters the index. Diagnostic only; ranking is unchanged. Blind to row-level mismatches (edition/config) inside a benchmark."},"models":[{"id":"aion-2.0","display_name":"Aion-2.0","family":{"id":"aion","vendor":"aion-labs"},"generation":"2.0","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"aion-labs/aion-2.0","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-02-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.8,"completion_usd_per_m":1.6,"cache_read_usd_per_m":0.2,"context_length":131072,"source":"openrouter","source_model_id":"aion-labs/aion-2.0","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#aion-labs/aion-2.0","publisher":"OpenRouter (listed 2026-02-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"aion-3.0","display_name":"Aion-3.0","family":{"id":"aion","vendor":"aion-labs"},"generation":"3.0","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"aion-labs/aion-3.0","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-07-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":6,"cache_read_usd_per_m":0.75,"context_length":131072,"source":"openrouter","source_model_id":"aion-labs/aion-3.0","as_of":"2026-10-06","blended_usd_per_m":3.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#aion-labs/aion-3.0","publisher":"OpenRouter (listed 2026-07-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"aion-3.0-mini","display_name":"Aion-3.0-Mini","family":{"id":"aion","vendor":"aion-labs"},"generation":"3.0","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"aion-labs/aion-3.0-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-07-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.7,"completion_usd_per_m":1.4,"cache_read_usd_per_m":0.18,"context_length":131072,"source":"openrouter","source_model_id":"aion-labs/aion-3.0-mini","as_of":"2026-10-06","blended_usd_per_m":0.8749999999999999,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#aion-labs/aion-3.0-mini","publisher":"OpenRouter (listed 2026-07-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"aion-3.5","display_name":"Aion 3.5","family":{"id":"aion","vendor":"aion-labs"},"generation":"3.5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"aion-labs/aion-3.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-09-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":6,"cache_read_usd_per_m":0.75,"context_length":262144,"source":"openrouter","source_model_id":"aion-labs/aion-3.5","as_of":"2026-10-06","blended_usd_per_m":3.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#aion-labs/aion-3.5","publisher":"OpenRouter (listed 2026-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"aion-3.5-mini","display_name":"Aion 3.5 Mini","family":{"id":"aion","vendor":"aion-labs"},"generation":"3.5","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"aion-labs/aion-3.5-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-09-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.7,"completion_usd_per_m":1.4,"cache_read_usd_per_m":0.18,"context_length":262144,"source":"openrouter","source_model_id":"aion-labs/aion-3.5-mini","as_of":"2026-10-06","blended_usd_per_m":0.8749999999999999,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#aion-labs/aion-3.5-mini","publisher":"OpenRouter (listed 2026-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"apodex-1.1","display_name":"Apodex 1.1","family":{"id":"apodex","vendor":"Apodex"},"generation":"1.1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":null,"release_date":"2026-08-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aav_capsid_structure","score_raw":0.649,"score_norm":null,"config":"60-mer shell, apodex-1.1 solver, averaged over surface loops, ligand complex, traces","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.11341","measured_date":"2026-08","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"aav_capsid_tropism","score_raw":0.635,"score_norm":null,"config":"apodex-1.1 solver with domain-specific environment, traces","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.11341","measured_date":"2026-08","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"aav_capsid_viability_ood_auroc","score_raw":0.904,"score_norm":null,"config":"apodex-1.1 solver with domain-specific environment, traces","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.11341","measured_date":"2026-08","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"aav_generative_design","score_raw":0.18,"score_norm":null,"config":"apodex-1.1 solver, capped experimental budget, traces","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.11341","measured_date":"2026-08","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":34.4,"score_norm":null,"config":"react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":38.5,"score_norm":null,"config":"agent team","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"biomysterybench_human_difficult","score_raw":23.5,"score_norm":null,"config":"human-difficult subset, react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"biomysterybench_human_difficult","score_raw":35.3,"score_norm":null,"config":"agent team, human-difficult subset","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":88.2,"score_norm":null,"config":"f1, react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":92.4,"score_norm":null,"config":"agent team, f1","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"frontierfinance","score_raw":48.7,"score_norm":null,"config":"official grader, public data boundary, react, rubric qualification rate macro-averaged","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"frontierfinance","score_raw":54.3,"score_norm":null,"config":"agent team, official grader, public data boundary, rubric qualification rate macro-averaged","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"frontierscience_research","score_raw":55,"score_norm":null,"config":"react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"frontierscience_research","score_raw":63.3,"score_norm":null,"config":"agent team","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":53.2,"score_norm":75.29,"config":"react, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":56.1,"score_norm":79.56,"config":"agent team, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":"tools"},{"benchmark":"imo","score_raw":18.5,"score_norm":null,"config":"2026, matharena protocol, react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo","score_raw":30.5,"score_norm":null,"config":"2026, agent team, matharena protocol","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo","score_raw":24.3,"score_norm":null,"config":"2025, matharena protocol, react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo","score_raw":36.5,"score_norm":null,"config":"2025, agent team, matharena protocol","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo_proofbench_advanced","score_raw":46.4,"score_norm":null,"config":"react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo_proofbench_advanced","score_raw":63.3,"score_norm":null,"config":"agent team","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo_proofbench_basic","score_raw":80,"score_norm":null,"config":"react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"imo_proofbench_basic","score_raw":96.7,"score_norm":null,"config":"agent team","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.7,"score_norm":75.22,"config":"coding-agent, highest reported setting, one result per model","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.8,"score_norm":77.41,"config":"2.1, coding-agent, highest reported setting, one result per model","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":true,"excluded":null},{"benchmark":"usamo_2026","score_raw":16,"score_norm":null,"config":"matharena protocol, react","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"usamo_2026","score_raw":26.5,"score_norm":null,"config":"agent team, matharena protocol","trust":1,"source_type":"vendor_claim","source_url":"https://www.apodex.com/pdf/20260824","measured_date":"2026-08-24","observed_at":"2026-08-25","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.apodex.com/blog/apodex-1.1-scaling-agentic-intelligence-for-complex-work","publisher":"Apodex"},{"url":"https://arxiv.org/abs/2608.23283","publisher":"arXiv"}],"last_verified":"2026-08-25","last_enriched_at":"2026-08-25T06:13:16.543Z"},{"id":"apodex-1.1-mini-35b","display_name":"Apodex 1.1 Mini","family":{"id":"apodex","vendor":"Apodex"},"generation":"1.1","size_class":"35b","params_total_b":35,"params_active_b":35,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":70,"int8_gb":37.1,"int4_gb":21,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"apodex/apodex-1.1-mini:free","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-08-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["apodex-1.1-mini"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"apodex/apodex-1.1-mini:free","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":70,"kv_cache_gb_per_8k":null,"min_serve_gb":74.9,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":37.1,"kv_cache_gb_per_8k":null,"min_serve_gb":40.355,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":21,"kv_cache_gb_per_8k":null,"min_serve_gb":23.45,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":37.1,"kv_cache_gb_per_8k":null,"min_serve_gb":40.355,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":20.42,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":35,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":34.74,"bw_limited_tok_s":20.85,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.66,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":35,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":7.8,"bw_limited_tok_s":4.68,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":54.31,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":35,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":95.71,"bw_limited_tok_s":57.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":74.7,"score_norm":null,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":24.2,"score_norm":null,"config":"35b apodex 1.1 mini, minimal react scaffold, react, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":27.7,"score_norm":null,"config":"35b apodex 1.1 mini, agent team, frontieragent harness, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":27.7,"score_norm":null,"config":"agent team workflow","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/apodex/Apodex-1.1-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":4.6,"score_norm":null,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"frontierfinance","score_raw":40,"score_norm":null,"config":"35b apodex 1.1 mini, minimal react scaffold, react, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontierfinance","score_raw":50.2,"score_norm":null,"config":"35b apodex 1.1 mini, agent team, frontieragent harness, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontierfinance","score_raw":50.2,"score_norm":null,"config":"agent team workflow","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/apodex/Apodex-1.1-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"frontierscience_research","score_raw":45,"score_norm":null,"config":"35b apodex 1.1 mini, minimal react scaffold, react, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontierscience_research","score_raw":51.7,"score_norm":null,"config":"35b apodex 1.1 mini, agent team, frontieragent harness, tools on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2608.23283","measured_date":"2026-08-25","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontierscience_research","score_raw":51.7,"score_norm":null,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.4,"score_norm":86.48,"config":"artificial analysis harness (aa-gpqa diamond)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":34.1,"score_norm":47.21,"config":"artificial analysis harness (aa-hle)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":79.2,"score_norm":null,"config":"artificial analysis harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.9,"score_norm":null,"config":"artificial analysis harness (aa-scicode)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/apodex-1-1-mini","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://arxiv.org/abs/2608.23283","publisher":"arXiv"},{"url":"https://huggingface.co/collections/apodex/apodex-11","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-02T06:06:55.267Z"},{"id":"claude-5.1-mythos","display_name":"Claude Mythos 5.1","family":{"id":"claude","vendor":"Anthropic"},"generation":"5.1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-01","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["claude-mythos-5.1"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_omniscience_net_score","score_raw":0.57,"score_norm":null,"config":"0.02 abstained), 0.20 incorrect, net score (0.77 correct","trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"anthropic_eci","score_raw":161.98,"score_norm":null,"config":"169.00], 95% ci [158.20, n=46","trust":1,"source_type":"vendor_claim","source_url":"https://systemcards.org/docs/anthropic-claude-fable-5-1-system-card","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":93.88,"score_norm":null,"config":"49 problems, averaged over four runs per problem, june 2026 release, max effort, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/document/claude-fable-5-1-mythos-5-1-system-card?ref=apidog.com","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":91.33,"score_norm":null,"config":"49 problems, averaged over four runs per problem, june 2026 release, max effort, without tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/document/claude-fable-5-1-mythos-5-1-system-card?ref=apidog.com","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arxivmath_june_2026_49_problems","score_raw":93.88,"score_norm":null,"config":"averaged over four runs per problem, max effort, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/document/claude-fable-5-1-mythos-5-1-system-card","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arxivmath_june_2026_49_problems","score_raw":91.33,"score_norm":null,"config":"averaged over four runs per problem, max effort, without tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/document/claude-fable-5-1-mythos-5-1-system-card","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"bbq","score_raw":89.9,"score_norm":null,"config":"bias -0.9% / 0.0%), disambiguated accuracy (ambiguous accuracy 100.0%","trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"bbq_ambiguous_accuracy","score_raw":100,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"bbq_disambiguated_accuracy","score_raw":89.9,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"biomysterybench_human_difficult","score_raw":44.1,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"biomysterybench_human_solvable","score_raw":90.3,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"exploitbench_v8_bench_autonudge_capability","score_raw":83,"score_norm":null,"config":"222/410 full aces across both arms, autonudge, mean 12.61 flags","trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"latchbio_singlecellbench","score_raw":61.9,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"latchbio_spatialbench_verified","score_raw":77.6,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"proteingym_hard","score_raw":49.3,"score_norm":null,"config":"rank correlation","trust":1,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/anthropic-claude-mythos-5.1/","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.9,"score_norm":65.76,"config":"10 trials per task (660 trials), 4.0, claude code --bare mode, maximum thinking effort, standard error ±1.6–2 points","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.9,"score_norm":65.76,"config":"claude mythos 5.1 column, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","publisher":"Anthropic"},{"url":"https://www.anthropic.com/claude/mythos","publisher":"Anthropic"},{"url":"https://platform.claude.com/docs/en/models/fable-5-1/overview.md","publisher":"Anthropic"}],"last_verified":"2026-09-02","last_enriched_at":"2026-10-03T06:04:05.831Z"},{"id":"claude-fable-5.1","display_name":"Claude Fable 5.1","family":{"id":"claude","vendor":"Anthropic"},"generation":"5.1","size_class":"frontier-fable","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-fable-5.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["claude-fable-5"],"superseded_by":null,"aliases":[],"intelligence":{"index":91.72,"inputs":[{"benchmark":"arena_elo","score_raw":1504,"score_norm":100,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"2906 votes, claude-fable-5.1-max, rank 3, text arena overall"},{"benchmark":"gpqa_diamond","score_raw":94,"score_norm":97.18,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, max effort with fallback"},{"benchmark":"hle","score_raw":60.9,"score_norm":86.62,"trust":1,"source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","config":"no tools"},{"benchmark":"livebench","score_raw":83.4,"score_norm":96.44,"trust":3,"source_url":"https://livebench.ai/","config":"global average (overall), livebench-2026-06-25 release, max effort"},{"benchmark":"terminal_bench","score_raw":55.8,"score_norm":59.76,"trust":1,"source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","config":"4.0, agentic coding, anthropic launch table (vendor-reported), production safeguards enabled"}],"coverage":0.5158,"subset_bias":null},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":50,"cache_read_usd_per_m":0.25,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-fable-5.1","as_of":"2026-10-06","blended_usd_per_m":20,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":57.9,"score_norm":57.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":81.6,"score_norm":81.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":53.4,"score_norm":53.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":90,"score_norm":94.44,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5-1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1504,"score_norm":100,"config":"2906 votes, claude-fable-5.1-max, rank 3, text arena overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"automation_bench","score_raw":31.4,"score_norm":null,"config":"anthropic launch table (vendor-reported), business workflows","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":31.4,"score_norm":null,"config":"business workflows","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":73.4,"score_norm":null,"config":"agentic coding, v3.2.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"cursorbench_v3_2","score_raw":73.4,"score_norm":null,"config":"3.2.0, agentic coding, anthropic launch table (vendor-reported), max effort","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1408,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1333,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1342,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1352,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1325,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1394,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1327,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1265,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1328,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1302,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1345,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1310,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1320,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1853,"score_norm":null,"config":"anthropic launch table (vendor-reported), knowledge work","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94,"score_norm":97.18,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, max effort with fallback","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.4,"score_norm":96.34,"config":"probe","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5.1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.4,"score_norm":96.34,"config":"vals ai harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5-1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.6,"score_norm":95.21,"config":"not present in anthropic's own launch-table, reported as an anthropic figure by data science dojo; reclassified 2026-09-05: attributed to anthropic but absent from anthropic's launch page","trust":0,"source_type":"community","source_url":"https://datasciencedojo.com/blog/claude-fable-5-1-performance-and-safety/","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":65,"score_norm":92.65,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":60.9,"score_norm":86.62,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":65,"score_norm":92.65,"config":"anthropic launch table (vendor-reported), with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":60.9,"score_norm":86.62,"config":"anthropic launch table (vendor-reported), no tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":83.4,"score_norm":96.44,"config":"global average (overall), livebench-2026-06-25 release, max effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":"2026-06-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":90.5,"score_norm":null,"config":"vals ai harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5-1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":90.52,"score_norm":null,"config":"reported as an anthropic figure by data science dojo; reclassified 2026-09-05: attributed to anthropic but absent from anthropic's launch page","trust":0,"source_type":"community","source_url":"https://datasciencedojo.com/blog/claude-fable-5-1-performance-and-safety/","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":92.4,"score_norm":87.33,"config":"vals ai harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5-1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":41.7,"score_norm":null,"config":"aug 2026 task release, osworld 2.0, strict pass","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":41.7,"score_norm":null,"config":"2.0, anthropic launch table (vendor-reported), august 2026 task release, computer use, strict","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":77.9,"score_norm":null,"config":"2.0, anthropic launch table (vendor-reported), august 2026 task release, computer use, partial credit","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"proofbench","score_raw":100,"score_norm":null,"config":"reported as an anthropic figure by data science dojo, v1.1; reclassified 2026-09-05: attributed to anthropic but absent from anthropic's launch page","trust":0,"source_type":"community","source_url":"https://datasciencedojo.com/blog/claude-fable-5-1-performance-and-safety/","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":81.2,"score_norm":100,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/claude-fable-5-1","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":80,"score_norm":100,"config":"reported as an anthropic figure by data science dojo; reclassified 2026-09-05: attributed to anthropic but absent from anthropic's launch page","trust":0,"source_type":"community","source_url":"https://datasciencedojo.com/blog/claude-fable-5-1-performance-and-safety/","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95,"score_norm":94.44,"config":"not present in anthropic's own launch-table, reported as an anthropic figure by data science dojo; reclassified 2026-09-05: attributed to anthropic but absent from anthropic's launch page","trust":0,"source_type":"community","source_url":"https://datasciencedojo.com/blog/claude-fable-5-1-performance-and-safety/","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.8,"score_norm":59.76,"config":"production safeguards active, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.8,"score_norm":59.76,"config":"4.0, agentic coding, anthropic launch table (vendor-reported), production safeguards enabled","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":52.6,"score_norm":null,"config":"v0.1","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science_0_1","score_raw":52.6,"score_norm":null,"config":"agentic scientific research, anthropic launch table (vendor-reported), production safeguards enabled, standard error 3.5-4.5 pts","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-fable-and-mythos-5-1","measured_date":"2026-09-01","observed_at":"2026-09-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5.1","publisher":"OpenRouter (listed 2026-09-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-haiku-4.5","display_name":"Claude Haiku 4.5","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.5","size_class":"frontier-haiku","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-haiku-4.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-10-15","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["claude-haiku-3"],"superseded_by":null,"aliases":["claude-4.5-haiku"],"intelligence":{"index":51.06,"inputs":[{"benchmark":"aime","score_raw":82.708,"score_norm":78.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"60 questions), claude-haiku-4-5-20251001-thinking, no tools, vals aime (2024+2025"},{"benchmark":"arc_agi_2","score_raw":4.03,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.3766, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-haiku-4-5-20251001, org=anthropic, run: thinking 32k, system_type=cot"},{"benchmark":"arena_elo","score_raw":1413,"score_norm":59.44,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"arena.ai text leaderboard, claude-haiku-4-5-20251001 (+/-3)"},{"benchmark":"gpqa_diamond","score_raw":67.17,"score_norm":59.39,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.671717171717172 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":10.38,"score_norm":12.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","config":"claude 4.5 haiku (reasoning), no tools"},{"benchmark":"mmlu_pro","score_raw":78.715,"score_norm":64.53,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"claude-haiku-4-5-20251001-thinking (extended thinking), max output 64k, no tools, temp 1"},{"benchmark":"swe_bench_verified","score_raw":73.3,"score_norm":70.33,"trust":1,"source_url":"https://www.anthropic.com/news/claude-haiku-4-5","config":"averaged over 50 trials, no test-time compute, 128k thinking budget, bash+file-edit scaffold"},{"benchmark":"tau_bench","score_raw":54.68,"score_norm":46.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","config":"artificial analysis tau-squared bench (telecom), claude 4.5 haiku (reasoning), tau2"},{"benchmark":"terminal_bench","score_raw":44.19,"score_norm":46.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.441947565543071 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":5,"cache_read_usd_per_m":0.1,"context_length":200000,"source":"openrouter","source_model_id":"anthropic/claude-haiku-4.5","as_of":"2026-10-06","blended_usd_per_m":2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":8,"score_norm":8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":16.47,"score_norm":16.47,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":43.9,"score_norm":43.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":16.9,"score_norm":16.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.89,"score_norm":29.89,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.736666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":82.708,"score_norm":78.38,"config":"60 questions), claude-haiku-4-5-20251001-thinking, no tools, vals aime (2024+2025","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":83.67,"score_norm":79.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.836666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":15,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.15 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":47.67,"score_norm":null,"config":"cost_per_task_usd=0.2558, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-haiku-4-5-20251001, org=anthropic, run: thinking 32k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.03,"score_norm":0,"config":"cost_per_task_usd=0.3766, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-haiku-4-5-20251001, org=anthropic, run: thinking 32k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.03,"score_norm":0,"config":"arc prize leaderboard, claude-haiku-4-5-20251001-thinking-32k, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/media/data/evaluations.json","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1413,"score_norm":59.44,"config":"arena.ai text leaderboard, claude-haiku-4-5-20251001 (+/-3)","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":9.55,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.09553946275772039 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1087,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1121,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1133,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1110,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1025,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1107,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1129,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-haiku-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":29.39,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.29394210683378097 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":20.67,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.20672500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73,"score_norm":67.61,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://airank.dev/models/claude-haiku-4.5","measured_date":"2025-10-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.17,"score_norm":59.39,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.671717171717172 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":61.05,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.610508033000434 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":10.38,"score_norm":12.32,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.103799814643188 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.38,"score_norm":12.32,"config":"claude 4.5 haiku (reasoning), no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":54.29,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.542857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":27.31,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.273069679849341 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":61.48,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.614814814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":4.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0444444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":78.715,"score_norm":64.53,"config":"claude-haiku-4-5-20251001-thinking (extended thinking), max output 64k, no tools, temp 1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":58.55,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.585549132947977 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.29,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.43287037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.3,"score_norm":70.33,"config":"averaged over 50 trials, no test-time compute, 128k thinking budget, bash+file-edit scaffold","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/news/claude-haiku-4-5","measured_date":"2025-10-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":54.68,"score_norm":46.24,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.546783625730994 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":54.68,"score_norm":46.24,"config":"artificial analysis tau-squared bench (telecom), claude 4.5 haiku (reasoning), tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.28,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0927835051546392 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.19,"score_norm":46.11,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.441947565543071 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":27.27,"score_norm":26.2,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.272727272727273 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":41.75,"score_norm":43.24,"config":"32k thinking budget, avg of 5 runs","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/news/claude-haiku-4-5","measured_date":"2025-10-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":40.21,"score_norm":41.42,"config":"no thinking, avg of 6 runs","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/news/claude-haiku-4-5","measured_date":"2025-10-15","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.claude.com/docs/en/about-claude/models/overview","publisher":"Anthropic"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.7-fast","display_name":"claude-4.7-opus-fast","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.7","size_class":"frontier-opus-fast","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.7-fast","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-05-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["claude-4.7-opus-fast"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7-fast","publisher":"OpenRouter (listed 2026-05-12)"}],"last_verified":"2026-09-01","last_enriched_at":null},{"id":"claude-opus-4.8-fast","display_name":"claude-4.8-opus-fast","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.8","size_class":"frontier-opus-fast","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.8-fast","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-05-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["claude-4.8-opus-fast"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8-fast","publisher":"OpenRouter (listed 2026-05-27)"}],"last_verified":"2026-09-01","last_enriched_at":null},{"id":"claude-opus-5-fast","display_name":"claude-opus-5-fast","family":{"id":"claude","vendor":"Anthropic"},"generation":"5","size_class":"frontier-opus-fast","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-5-fast","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-07-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5-fast","publisher":"OpenRouter (listed 2026-07-24)"}],"last_verified":"2026-09-01","last_enriched_at":null},{"id":"claude-opus-5.5","display_name":"Claude Opus 5.5","family":{"id":"claude","vendor":"Anthropic"},"generation":"5.5","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-5.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["claude-opus-4","claude-opus-4.1","claude-opus-4.5","claude-opus-4.6","claude-opus-4.7","claude-opus-4.8","claude-opus-5"],"superseded_by":null,"aliases":[],"intelligence":{"index":92.48,"inputs":[{"benchmark":"arc_agi_2","score_raw":93.33,"score_norm":98.14,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"effort=high, run: high, semi-private"},{"benchmark":"arena_elo","score_raw":1503.7,"score_norm":99.87,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":90.24,"score_norm":91.89,"trust":2,"source_url":"https://openrouter.ai/anthropic/claude-opus-5.5","config":"openrouter autoexacto, provider=anthropic"},{"benchmark":"hle","score_raw":61.35,"score_norm":87.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning, tools=off"},{"benchmark":"livebench","score_raw":83.22,"score_norm":96.04,"trust":3,"source_url":"https://livebench.ai/","config":"effort=max, livebench-2026-06-25 release, run: max effort"},{"benchmark":"tau_bench","score_raw":73.4,"score_norm":71.2,"trust":2,"source_url":"https://openrouter.ai/anthropic/claude-opus-5.5","config":"domain=airline (verified), openrouter autoexacto, provider=anthropic"},{"benchmark":"terminal_bench","score_raw":87.64,"score_norm":97.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"effort=high, v2.1"}],"coverage":0.6428,"subset_bias":null},"pricing":{"prompt_usd_per_m":4,"completion_usd_per_m":20,"cache_read_usd_per_m":0.2,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-opus-5.5","as_of":"2026-10-06","blended_usd_per_m":8,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_briefcase","score_raw":1821.9,"score_norm":null,"config":"default fallback, effort=max, elo, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_briefcase","score_raw":1822,"score_norm":null,"config":"effort=max, elo, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":65.99,"score_norm":null,"config":"agent=claude code, effort=max, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":57.62,"score_norm":57.62,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning, v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":57.6,"score_norm":57.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":84.67,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":66.22,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":46.42,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":41.4,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":98.5,"score_norm":null,"config":"effort=high, run: high, semi-private","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-09-24","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":93.33,"score_norm":98.14,"config":"effort=high, run: high, semi-private","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-09-24","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1503.7,"score_norm":99.87,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1508.6,"score_norm":100,"config":"effort=high, run: high, style control, text/overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-25","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1826.7,"score_norm":null,"config":"effort=max, raw, run: max, webdev/overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-09-25","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":83.65,"score_norm":null,"config":"effort=high, overall, run: high, tools=on","trust":3,"source_type":"leaderboard","source_url":"https://matharena.ai/models/anthropic_opus_55_high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":96.9,"score_norm":null,"config":"2026-08 release, effort=max, tools=on, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":91.2,"score_norm":null,"config":"2026-08 release, effort=max, tools=off","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":69.54,"score_norm":null,"config":"default fallback, effort=max, max effort, partial score, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":40,"score_norm":null,"config":"effort=max, no fallback","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":42.47,"score_norm":null,"config":"default fallbacks, effort=max, max, run: default fallbacks, v1.0.6","trust":1,"source_type":"leaderboard","source_url":"https://zapier.com/benchmarks","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"biomysterybench","score_raw":79.259,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"brokenarxiv","score_raw":90.79,"score_norm":null,"config":"effort=high, overall, run: high, tools=on","trust":3,"source_type":"leaderboard","source_url":"https://matharena.ai/models/anthropic_opus_55_high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"chartography","score_raw":64.4,"score_norm":null,"config":"effort=max, tools=off","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"chartography","score_raw":89,"score_norm":null,"config":"effort=max, tools=on","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":66.646,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":31.71,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":57.8,"score_norm":null,"config":"effort=max, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":57.8,"score_norm":null,"config":"effort=max, run: max, v4.0","trust":1,"source_type":"leaderboard","source_url":"https://cursor.com/cursorbench","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":68.44,"score_norm":null,"config":"agent=claude code, effort=max, run: max, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":74.2,"score_norm":null,"config":"effort=max, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1491,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1406,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1395,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1480,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1328,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1334,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1402,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1341,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"ebr_bench","score_raw":71.43,"score_norm":null,"config":"effort=max, run: max","trust":3,"source_type":"leaderboard","source_url":"https://epoch.ai/models/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"emb","score_raw":75.938,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"finance_agent","score_raw":58.587,"score_norm":null,"config":"effort=max, v2","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontiercode_1_1_main","score_raw":54.6,"score_norm":null,"config":"effort=medium","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontiercode_1_1_main","score_raw":54.4,"score_norm":null,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontiercode_v1_1_extended","score_raw":65.3,"score_norm":null,"config":"effort=medium","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":62.3,"score_norm":null,"config":"effort=max, v2","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"furniture_assembly","score_raw":83.33,"score_norm":null,"config":"effort=max, run: max","trust":3,"source_type":"leaderboard","source_url":"https://epoch.ai/models/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdp_pdf","score_raw":28.8,"score_norm":null,"config":"all-pass, default fallback, effort=high, high effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":67.31,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1846.2,"score_norm":null,"config":"default fallback, effort=max, elo, max effort, reasoning=on, run: adaptive reasoning, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1846,"score_norm":null,"config":"effort=max, elo, v2.1","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"global_mmlu","score_raw":94.3,"score_norm":null,"config":"42 languages, effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.24,"score_norm":91.89,"config":"openrouter autoexacto, provider=anthropic","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":91.24,"score_norm":null,"config":"default fallback, effort=xhigh, held-out set, reasoning=on, run: adaptive reasoning, xhigh effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5-xhigh","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":91.2,"score_norm":null,"config":"effort=max, held-out set","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"harvey_lab_vals","score_raw":3.75,"score_norm":null,"config":"all-pass, effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"healthbench","score_raw":60.6,"score_norm":null,"config":"effort=max, length-adjusted","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":65.6,"score_norm":null,"config":"effort=max, length-adjusted","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":61.35,"score_norm":87.28,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning, tools=off","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":64.4,"score_norm":91.76,"config":"effort=max, tools=off","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":67.7,"score_norm":96.62,"config":"effort=max, tools=on","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle_diamond","score_raw":55,"score_norm":null,"config":null,"trust":3,"source_type":"leaderboard","source_url":"https://labs.scale.com/leaderboard/hle-diamond","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"ioi","score_raw":95.056,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":38.23,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"legal_research_bench","score_raw":50.481,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":83.22,"score_norm":96.04,"config":"effort=max, livebench-2026-06-25 release, run: max effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"medcode","score_raw":49.797,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"medscribe","score_raw":91.43,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"milu","score_raw":93.1,"score_norm":null,"config":"11 languages, effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":66.67,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":87.69,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"mysterymechanism","score_raw":49.55,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"officeqa","score_raw":78.9,"score_norm":null,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"officeqa_pro","score_raw":67.7,"score_norm":null,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":48.7,"score_norm":null,"config":"effort=max, strict pass, v2.0","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":81.8,"score_norm":null,"config":"effort=max, partial credit, v2.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":18.5,"score_norm":null,"config":"effort=max, fully resolved","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":91.2,"score_norm":null,"config":"166 tasks, hidden test pass rate","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"proofbench","score_raw":100,"score_norm":null,"config":"effort=max, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"public_benefits_bench","score_raw":70.636,"score_norm":null,"config":"effort=max, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"sage","score_raw":45.831,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":66.9,"score_norm":null,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"sre_bench","score_raw":33.588,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_atlas_qna","score_raw":66.4,"score_norm":null,"config":"agent=claude code, effort=max, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":93.9,"score_norm":null,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_bench_multimodal","score_raw":61.4,"score_norm":null,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":89.9,"score_norm":100,"config":"effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":73.4,"score_norm":71.2,"config":"domain=airline (verified), openrouter autoexacto, provider=anthropic","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/anthropic/claude-opus-5.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tax_agent_bench","score_raw":70.499,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.64,"score_norm":97.22,"config":"effort=high, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.616,"score_norm":66.61,"config":"effort=max, v4.0","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-4","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.6,"score_norm":64.24,"config":"default fallback, effort=max, max effort, reasoning=on, run: adaptive reasoning, v4.0","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":64.8,"score_norm":70.35,"config":"effort=max, safeguards on, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":66.4,"score_norm":72.24,"config":"effort=xhigh, safeguards on, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":48.571,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-science","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":61.9,"score_norm":null,"config":"default fallback, effort=xhigh, reasoning=on, run: adaptive reasoning, xhigh effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-5-xhigh","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":58.7,"score_norm":null,"config":"effort=max, v0.1","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"toolathlon","score_raw":77.8,"score_norm":null,"config":"effort=max, mean of 3 trials, verified","trust":1,"source_type":"vendor_claim","source_url":"https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":69.689,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vals_rsi_index","score_raw":37.13,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench","score_raw":90.294,"score_norm":null,"config":"effort=max, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench_1_100","score_raw":30.361,"score_norm":null,"config":"effort=max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"wandr","score_raw":72.3,"score_norm":null,"config":"980k budget, code exec, effort=max, run: max, run: offline web tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-opus-5-5","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5.5","publisher":"OpenRouter (listed 2026-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-sonnet-5","display_name":"Claude Sonnet 5","family":{"id":"claude","vendor":"Anthropic"},"generation":"5","size_class":"frontier-sonnet","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-sonnet-5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-06-30","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["claude-sonnet-4","claude-sonnet-4.5","claude-sonnet-4.6"],"superseded_by":null,"aliases":[],"intelligence":{"index":75.34,"inputs":[{"benchmark":"arena_elo","score_raw":1461.8,"score_norm":81.2,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":88.89,"score_norm":89.99,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"compute_effort=max, edition=gpqa diamond v1, runner=platform, stderr=2.224, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":41.29,"score_norm":57.78,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.412882298424467 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":76.04,"score_norm":80.09,"trust":3,"source_url":"https://livebench.ai/table_2026_06_25.csv","config":"2026-06-25 question release, claude-sonnet-5-xhigh-effort, global average of 7 category averages"},{"benchmark":"mmlu_pro","score_raw":87.55,"score_norm":79.25,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.368, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"compute_effort=max, edition=swe-bench verified v1, runner=external, stderr=1.804, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":37.32,"score_norm":23.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","config":"adaptive reasoning, max effort, tau^3-banking (artificial analysis edition)"},{"benchmark":"terminal_bench","score_raw":74.53,"score_norm":81.8,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"compute_effort=max, edition=terminal-bench 2.1, runner=external, stderr=2.085, task=overall, updated=2026-08-19"}],"coverage":0.8413,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-sonnet-5","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":43.6,"score_norm":43.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":49.72,"score_norm":49.72,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.5,"score_norm":71.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.2,"score_norm":38.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":55.26,"score_norm":55.26,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":77,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.77 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":16.45,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":46.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.4625 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1461.8,"score_norm":81.2,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1461.2,"score_norm":80.93,"config":"27366 votes, arena entry claude-sonnet-5-text, leaderboard text-overall-style_control, rank 45, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1462.4,"score_norm":81.46,"config":"23256 votes, claude-sonnet-5-high, rank 45, text arena","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1466.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1470.2,"score_norm":null,"config":"3601 votes, arena entry claude-sonnet-5-document, document/overall, leaderboard document-overall-raw, rank 14, raw (no style control), run: high, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1542.5,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1264.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1270.7,"score_norm":null,"config":"7862 votes, arena entry claude-sonnet-5-vision, leaderboard vision-overall-style_control, rank 27, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1539.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1538.6,"score_norm":null,"config":"7267 votes, arena entry claude-sonnet-5-webdev, code/overall, leaderboard webdev-overall-raw, rank 22, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":39.19,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.39190097119704015 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":16.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.168571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":57,"score_norm":null,"config":"third-party, reported by cursor","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1277,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1226,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1237,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1221,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1283,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1256,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1248,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1296,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1225,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1217,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1211,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1216,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1192,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1286,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1237,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1282,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":44.67,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.4467323187108326 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":54.77,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: adaptive reasoning, source value 0.547745 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1618,"score_norm":null,"config":"elo scale, knowledge work","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.89,"score_norm":89.99,"config":"compute_effort=max, edition=gpqa diamond v1, runner=platform, stderr=2.224, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.889,"score_norm":89.98,"config":"compute effort: max, no tools","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.11,"score_norm":93.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.911111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.11,"score_norm":93.11,"config":"adaptive reasoning, max effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":90.07,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.900709219858156 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":57.4,"score_norm":81.47,"config":"with tools","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":41.29,"score_norm":57.78,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.412882298424467 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":76.04,"score_norm":80.09,"config":"2026-06-25 question release, claude-sonnet-5-xhigh-effort, global average of 7 category averages","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2026_06_25.csv","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":76.04,"score_norm":80.09,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":55,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.55 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.55,"score_norm":79.25,"config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.368, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.545,"score_norm":79.24,"config":"compute effort: max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":77.28,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, mmmu-pro, reasoning=true, run: adaptive reasoning, source value 0.772832369942197 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":81.2,"score_norm":null,"config":"computer use, verified","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.59,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, scicode, source value 0.53587962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":63.2,"score_norm":97,"config":"anthropic claude sonnet 5 system card, agentic coding","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"config":"compute_effort=max, edition=swe-bench verified v1, runner=external, stderr=1.804, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"config":"500 tasks, compute effort: max, max_output_tokens 128000","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":37.32,"score_norm":23.09,"config":"adaptive reasoning, max effort, tau^3-banking (artificial analysis edition)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":37.32,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.37319587628866 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.53,"score_norm":81.8,"config":"compute_effort=max, edition=terminal-bench 2.1, runner=external, stderr=2.085, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.532,"score_norm":81.8,"config":"2.1, compute effort: max","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.4,"score_norm":88.71,"config":"anthropic claude sonnet 5 system card","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-sonnet-5-benchmarks-explained","measured_date":"2026-06-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.52,"score_norm":88.85,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.805243445692884 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.claude.com/docs/en/about-claude/models/overview","publisher":"Anthropic"},{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5","publisher":"OpenRouter (listed 2026-06-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-sonnet-5.5","display_name":"Claude Sonnet 5.5","family":{"id":"claude","vendor":"Anthropic"},"generation":"5.5","size_class":"frontier-sonnet","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-sonnet-5.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-sonnet-5.5","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":56,"score_norm":56,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5.5","measured_date":null,"observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1471,"score_norm":85.3,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1268.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1715.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1402,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-5.5","publisher":"OpenRouter (listed 2026-09-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"codestral-25.01-22b","display_name":"Codestral 25.01","family":{"id":"codestral","vendor":"mistralai"},"generation":"25.01","size_class":"frontier","params_total_b":22,"params_active_b":22,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":44,"int8_gb":23.3,"int4_gb":13.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":256000,"release_date":"2025-01-13","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["codestral-25.01"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/codestral-2501/","publisher":"Mistral AI"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"codestral-25.08","display_name":"Codestral 25.08","family":{"id":"codestral","vendor":"mistralai"},"generation":"25.08","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2025-07-30","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.mistral.ai/models/codestral-25-08","publisher":"Mistral AI"},{"url":"https://mistral.ai/news/codestral-25-08/","publisher":"Mistral AI"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"codestral-2508","display_name":"Codestral 2508","family":{"id":"codestral","vendor":"mistralai"},"generation":"2508","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/codestral-2508","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":256000,"release_date":"2025-08-01","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["codestral-1-2508"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":0.9,"cache_read_usd_per_m":0.03,"context_length":256000,"source":"openrouter","source_model_id":"mistralai/codestral-2508","as_of":"2026-10-06","blended_usd_per_m":0.44999999999999996,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_3d","score_raw":1035,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1013,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1026,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":980,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1024,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1019,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/codestral-2508","publisher":"OpenRouter (listed 2025-08-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"command-a-03-2025","display_name":"Command A","family":{"id":"command","vendor":"Cohere"},"generation":"03","size_class":"frontier-a","params_total_b":111.06,"params_active_b":111.06,"params_exact":111057580032,"params_source":"hf_safetensors","vram_est":{"bf16_gb":222,"int8_gb":118,"int4_gb":66.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.15,"expected_int8_gb":17.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"CohereForAI/c4ai-command-a-03-2025","openrouter_id":"cohere/command-a","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"cc-by-nc-4.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-03","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["c4ai-command-a-03-2025","command-a","command-a-1-03-2025"],"intelligence":{"index":25.28,"inputs":[{"benchmark":"aime","score_raw":13.33,"score_norm":0,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.945, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1353.7,"score_norm":33.01,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":48.48,"score_norm":33.07,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.276, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":11.96,"score_norm":14.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/command-a-plus","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: command a+, source value 0.119555143651529 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":69.17,"score_norm":48.62,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.463, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":7.8,"score_norm":0,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.201, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":80.7,"score_norm":80.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/command-a-plus","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: command a+, source value 0.807017543859649 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/command-a-plus","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: command a+, source value 0.25 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":10,"cache_read_usd_per_m":null,"context_length":256000,"source":"openrouter","source_model_id":"cohere/command-a","as_of":"2026-10-06","blended_usd_per_m":4.375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":222.12,"kv_cache_gb_per_8k":null,"min_serve_gb":237.226,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":222.115,"kv_cache_gb_per_8k":null,"min_serve_gb":237.221,"method":"vendor","source_url":"https://huggingface.co/CohereForAI/c4ai-command-a-03-2025/tree/main"},{"quant":"fp8","weights_gb":117.724,"kv_cache_gb_per_8k":null,"min_serve_gb":127.61,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":66.636,"kv_cache_gb_per_8k":null,"min_serve_gb":73.968,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":117.724,"kv_cache_gb_per_8k":null,"min_serve_gb":127.61,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":1.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":111.06,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2.46,"bw_limited_tok_s":1.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":34.93,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":111.06,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":60.33,"bw_limited_tok_s":36.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#cohere/command-a","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":9.23,"score_norm":9.23,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":27.8,"score_norm":27.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#cohere/command-a","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.1,"score_norm":13.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#cohere/command-a","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.77,"score_norm":22.77,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: command a+","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":48.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: command a+, source value 0.486666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":13.33,"score_norm":0,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.945, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":13,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.13 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":23.3,"score_norm":4.13,"config":"2024 aime competition set, command a row, no tools, non-reasoning model. year mismatch vs preferred aime 2025/2026 - model predates those sets. [raw: aime (2024)], table 15 'aime (2024)' column","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2504.00698","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1353.7,"score_norm":33.01,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1353.8,"score_norm":33.06,"config":"56404 votes, arena entry command-a-03-2025, leaderboard text-overall-style_control, rank 193, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: command a+, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":10.84,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.10844 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":48.48,"score_norm":33.07,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.276, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.06,"score_norm":71.92,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: command a+, source value 0.760606060606061 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":50.8,"score_norm":36.34,"config":"answer parsing) per sec 4.1, command a row, no tools. table 3's 'gpqa' column reports the same 50.8 for c [raw: gpqa diamond], prompting, simple-evals implementation (data, table 15 'gpqa (diamond)' column, task settings","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2504.00698","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.96,"score_norm":14.65,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: command a+, source value 0.119555143651529 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.95,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: command a+, source value 0.739455782312925 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":28.68,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.286772486772487 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.17,"score_norm":48.62,"config":"edition=mmlu pro v1, runner=platform, stderr=0.463, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.6,"score_norm":49.33,"config":"command a row, mmlu-pro column, no tools. [raw: mmlu-pro], simple-evals implementation, table 3 (standard academic benchmarks)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2504.00698","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":63.24,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.632369942196532 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":37.85,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: command a+, scicode, source value 0.378472222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":7.8,"score_norm":0,"config":"edition=swe-bench verified v1, runner=external, stderr=1.201, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":26.8,"score_norm":18.67,"config":"command a row. note: non-agentic 'swe-bench verified patch generation task in python' (sec 4.4), internal reproduction with cohere's own prompt - not the standard a [raw: swe-bench verified], table 13 (code editing benchmarks)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2504.00698","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":80.7,"score_norm":80.93,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: command a+, source value 0.807017543859649 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":51.7,"score_norm":42.27,"config":"[raw: taubench (tau-bench)], aggregated over 10 runs, command a column - published single aggregate number (corresponds to retail+airline pass@1 average, table 1 'taubench' row, table 6 details retail p@1 60.0 / airline p@1 45.3","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2504.00698","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.98,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0597938144329897 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":22.85,"score_norm":21,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.228464419475655 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: command a+, source value 0.25 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/command-a-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.cohere.com/docs/models","publisher":"Cohere"},{"url":"https://openrouter.ai/api/v1/models#cohere/command-a","publisher":"OpenRouter (listed 2025-03-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"command-a-05-plus-2026","display_name":"Command A+","family":{"id":"command","vendor":"Cohere"},"generation":"05","size_class":"frontier-a-plus","params_total_b":218.75,"params_active_b":null,"params_exact":218750546160,"params_source":"hf_safetensors","vram_est":{"bf16_gb":438,"int8_gb":232,"int4_gb":131,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.74,"expected_int8_gb":172,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"CohereLabs/command-a-plus-05-2026-w4a4","openrouter_id":"cohere/command-a-plus","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":192000,"release_date":"2026-05-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["command-a-05-plus-2026-vision","command-a-1-plus-05-2026","command-a-plus","command-a-plus-05-2026","command-a-plus-05-2026-w4a4"],"intelligence":{"index":49.65,"inputs":[{"benchmark":"gpqa_diamond","score_raw":76,"score_norm":71.83,"trust":2,"source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","config":"artificial analysis independent eval; reported rounded (~76%)"},{"benchmark":"hle","score_raw":11,"score_norm":13.24,"trust":2,"source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","config":"artificial analysis independent eval; reported rounded (~11%)"},{"benchmark":"tau_bench","score_raw":85,"score_norm":86.67,"trust":1,"source_url":"https://cohere.com/blog/command-a-plus","config":"standard settings + user simulation per barres et al. 2025, tau2/telecom"},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"trust":1,"source_url":"https://cohere.com/blog/command-a-plus","config":"terminus-2 harness, artificial analysis methodology"}],"coverage":0.3651,"subset_bias":{"offset":-6.12,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.5,"cache_read_usd_per_m":0.15,"context_length":192000,"source":"openrouter","source_model_id":"cohere/command-a-plus","as_of":"2026-10-06","blended_usd_per_m":0.6,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":437.5,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":460.046,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":231.875,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":244.14,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":131.25,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":138.484,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":131.894,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":139.16,"method":"vendor","source_url":"https://huggingface.co/CohereLabs/command-a-plus-05-2026-w4a4/tree/main"},{"quant":"int8","weights_gb":231.875,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":244.14,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":37,"score_norm":37,"config":"artificial analysis intelligence index","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":9,"score_norm":null,"config":"artificial analysis aa-omniscience accuracy","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination","score_raw":86,"score_norm":null,"config":"artificial analysis aa-omniscience non-hallucination rate","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"charxiv_reasoning","score_raw":52.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76,"score_norm":71.83,"config":"artificial analysis independent eval; reported rounded (~76%)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11,"score_norm":13.24,"config":"artificial analysis independent eval; reported rounded (~11%)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mathvista","score_raw":80.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu","score_raw":75.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":63,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":38,"score_norm":null,"config":"artificial analysis independent eval; reported rounded (~38%)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/cohere-launches-open-weights-model-command-a-more-than-a-year-since-the-command-a-release","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":85,"score_norm":86.67,"config":"standard settings + user simulation per barres et al. 2025, tau2/telecom","trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"config":"terminus-2 harness, artificial analysis methodology","trust":1,"source_type":"vendor_claim","source_url":"https://cohere.com/blog/command-a-plus","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"vals_index","score_raw":24.69,"score_norm":null,"config":"vals ai index accuracy, +/-0.76","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/cohere_command-a-plus-05-2026","measured_date":"2026-05-20","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.cohere.com/docs/models","publisher":"Cohere"},{"url":"https://cohere.com/blog/command-a-plus","publisher":"Cohere"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"command-r-08-2024","display_name":"Command R (08-2024)","family":{"id":"command","vendor":"Cohere"},"generation":"08","size_class":"frontier-r","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"cohere/command-r-08-2024","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-08-30","release_status":"released","knowledge_cutoff":"2024-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["command-r-1-08-2024"],"intelligence":{"index":3.55,"inputs":[{"benchmark":"aime","score_raw":0.3,"score_norm":12.5,"trust":2,"source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run"},{"benchmark":"arena_elo","score_raw":1250.4,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":28.9,"score_norm":5.49,"trust":2,"source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run"},{"benchmark":"hle","score_raw":5.1,"score_norm":4.56,"trust":2,"source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run, no tools"},{"benchmark":"livebench","score_raw":31.4,"score_norm":0,"trust":3,"source_url":"https://livebench.ai/table_2025_04_25.csv","config":"global average of the 6 category averages, leaderboard row command-r-08-2024, livebench 2025-04-25 release"},{"benchmark":"mmlu_pro","score_raw":33.7,"score_norm":0,"trust":2,"source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","config":"artificial analysis independent run, read from the 2025-11-13 wayback capture of aa's now-deprecated command-r (aug '24) page"}],"coverage":0.6666,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.6,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"cohere/command-r-08-2024","as_of":"2026-10-06","blended_usd_per_m":0.26249999999999996,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":0.3,"score_norm":12.5,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1250.4,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1250.1,"score_norm":0,"config":"10140 votes, arena entry command-r-08-2024, leaderboard text-overall-style_control, rank 300, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":28.9,"score_norm":5.49,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":26.77,"score_norm":2.49,"config":"evaleval run, surfaced in the hugging face model-page evaluation results panel (unverified)","trust":2,"source_type":"aggregator","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r-08-2024","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.1,"score_norm":4.56,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":31.4,"score_norm":0,"config":"global average of the 6 category averages, leaderboard row command-r-08-2024, livebench 2025-04-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2025_04_25.csv","measured_date":"2025-04-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":4.4,"score_norm":null,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":14.9,"score_norm":null,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":33.7,"score_norm":0,"config":"artificial analysis independent run, read from the 2025-11-13 wayback capture of aa's now-deprecated command-r (aug '24) page","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":8.7,"score_norm":null,"config":"2025-11-13 wayback capture of aa's deprecated command-r (aug '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20251113030850/https://artificialanalysis.ai/models/command-r","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#cohere/command-r-08-2024","publisher":"OpenRouter (listed 2024-08-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"command-r-08-plus-2024","display_name":"Command R+ (08-2024)","family":{"id":"command","vendor":"Cohere"},"generation":"08","size_class":"frontier-r-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"cohere/command-r-plus-08-2024","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-08-30","release_status":"released","knowledge_cutoff":"2024-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["command-r-1-plus-08-2024","command-r-plus-08-2024"],"intelligence":{"index":5.85,"inputs":[{"benchmark":"arena_elo","score_raw":1276.3,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":34.34,"score_norm":13.15,"trust":1,"source_url":"https://huggingface.co/CohereLabs/c4ai-command-r-plus-08-2024","config":"as listed in the model card's evaluation entry"},{"benchmark":"livebench","score_raw":34.9,"score_norm":0,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2025-04-25 release"}],"coverage":0.3572,"subset_bias":null},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":10,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"cohere/command-r-plus-08-2024","as_of":"2026-10-06","blended_usd_per_m":4.375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_elo","score_raw":1276.3,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1276.1,"score_norm":0,"config":"9866 votes, arena entry command-r-plus-08-2024, leaderboard text-overall-style_control, rank 284, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":34.34,"score_norm":13.15,"config":"as listed in the model card's evaluation entry","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r-plus-08-2024","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":34.9,"score_norm":0,"config":"livebench-2025-04-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2025-04-25","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#cohere/command-r-plus-08-2024","publisher":"OpenRouter (listed 2024-08-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"command-r-7-b-12-2024-7b","display_name":"Command R7B (12-2024)","family":{"id":"command","vendor":"Cohere"},"generation":"7","size_class":"frontier-r","params_total_b":7,"params_active_b":7,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":14,"int8_gb":7.4,"int4_gb":4.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"cohere/command-r7b-12-2024","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-12","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["command-r-1-7b-12-2024","command-r-12-7b-12-2024","command-r-7-b-12-2024","command-r7b-12-2024"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.0375,"completion_usd_per_m":0.15,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"cohere/command-r7b-12-2024","as_of":"2026-10-06","blended_usd_per_m":0.06562499999999999,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"bbh","score_raw":36.1,"score_norm":null,"config":"huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":26.77,"score_norm":2.49,"config":"idavidrein/gpqa community evaluation on the model card","trust":0,"source_type":"community","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":77.9,"score_norm":null,"config":"huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_hard","score_raw":26.4,"score_norm":null,"config":"huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":35.72,"score_norm":0,"config":"5-shot raw accuracy (the leaderboard-normalised value for the same run is 28.58), huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":28.5,"score_norm":0,"config":"huggingface open llm leaderboard v2 (normalized)","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":11.6,"score_norm":null,"config":"huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/CohereLabs/c4ai-command-r7b-12-2024","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.cohere.com/docs/models","publisher":"Cohere"},{"url":"https://openrouter.ai/api/v1/models#cohere/command-r7b-12-2024","publisher":"OpenRouter (listed 2024-12-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-2-instruct-236b-code","display_name":"DeepSeek-Coder-V2-Instruct","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"2","size_class":"236b","params_total_b":235.74,"params_active_b":21,"params_exact":235741434880,"params_source":"hf_safetensors","vram_est":{"bf16_gb":471,"int8_gb":250,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-Coder-V2-Instruct","openrouter_id":null,"is_moe":true,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2024-06","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-coder-v2-instruct"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":471.48,"kv_cache_gb_per_8k":0.5662,"min_serve_gb":495.62,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":471.487,"kv_cache_gb_per_8k":0.5662,"min_serve_gb":495.628,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct/tree/main"},{"quant":"fp8","weights_gb":249.884,"kv_cache_gb_per_8k":0.5662,"min_serve_gb":262.944,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.444,"kv_cache_gb_per_8k":0.5662,"min_serve_gb":149.082,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.884,"kv_cache_gb_per_8k":0.5662,"min_serve_gb":262.944,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Base","publisher":"Hugging Face"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"deepseek-2-math-685b","display_name":"DeepSeek-Math-V2","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"2","size_class":"685b","params_total_b":685,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":1370,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-11-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-math-v2"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":1370,"kv_cache_gb_per_8k":null,"min_serve_gb":1442.5,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":726.1,"kv_cache_gb_per_8k":null,"min_serve_gb":766.405,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":411,"kv_cache_gb_per_8k":null,"min_serve_gb":435.55,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":726.1,"kv_cache_gb_per_8k":null,"min_serve_gb":766.405,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://thursdai.news/companies/deepseek","publisher":"ThursdAI"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"deepseek-3.1-instruct","display_name":"DeepSeek V3.1","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3.1","size_class":"frontier","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.3,"expected_int8_gb":220,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V3.1","openrouter_id":"deepseek/deepseek-chat-v3.1","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-08-21","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":["deepseek-3-0324-instruct"],"superseded_by":null,"aliases":["deepseek-chat-v3.1","deepseek-v3.1","deepseek-v3.1-671b"],"intelligence":{"index":57.85,"inputs":[{"benchmark":"aime","score_raw":89.67,"score_norm":87.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.896666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1417.1,"score_norm":61.27,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.88,"score_norm":74.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.778787878787879 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":14.27,"score_norm":18.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.142724745134384 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":83.7,"score_norm":72.83,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","config":"deepseek-v3.1 nonthinking mode, exact-match — official model-card eval table [raw: mmlu-pro (em)]"},{"benchmark":"swe_bench_verified","score_raw":66,"score_norm":62.22,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","config":"agent mode using deepseek's internal code-agent framework (not a third-party harness) [raw: swe verified (agent mode)], deepseek-v3.1 nonthinking mode, swe-bench verified"},{"benchmark":"tau_bench","score_raw":37.43,"score_norm":23.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.374269005847953 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.25 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":0.95,"cache_read_usd_per_m":0.13,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-chat-v3.1","as_of":"2026-10-06","blended_usd_per_m":0.425,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1369.06,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.089,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.587,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.592,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1/tree/main"},{"quant":"int4","weights_gb":410.718,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":13.7,"score_norm":13.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.37,"score_norm":21.37,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":56.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.566666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":89.67,"score_norm":87.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.896666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":49.8,"score_norm":37.25,"config":"aime 2025, deepseek-v3.1 nonthinking mode, no tools — official model-card eval table [raw: aime 2025 (pass@1)], pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417.1,"score_norm":61.27,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417.1,"score_norm":61.27,"config":"15002 votes, arena entry deepseek-v3.1, leaderboard text-overall-style_control, rank 117, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417.1,"score_norm":61.27,"config":"15002 votes, arena entry deepseek-v3.1, leaderboard text-overall-style_control, rank 117, run: deepseek-v3.1, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":2,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.02 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1092,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1118,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1104,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1097,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":966,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1092,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1129,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.88,"score_norm":74.48,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.778787878787879 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.9,"score_norm":70.28,"config":"deepseek-v3.1 nonthinking mode (default chat/instruct mode), no tools — official model-card eval table [raw: gpqa-diamond (pass@1)], pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.27,"score_norm":18.04,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.142724745134384 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":15.9,"score_norm":20.44,"config":"deepseek-v3.1 thinking mode (vendor reports no nonthinking hle number) [raw: humanity's last exam (pass@1)], hle text-only subset, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.5,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.414965986394558 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":78.41,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.784126984126984 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.7,"score_norm":72.83,"config":"deepseek-v3.1 nonthinking mode, exact-match — official model-card eval table [raw: mmlu-pro (em)]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":39.12,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.391203703703704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":66,"score_norm":62.22,"config":"agent mode using deepseek's internal code-agent framework (not a third-party harness) [raw: swe verified (agent mode)], deepseek-v3.1 nonthinking mode, swe-bench verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":37.43,"score_norm":23.24,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.374269005847953 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.25 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.3,"score_norm":30.94,"config":"deepseek-v3.1 nonthinking mode [raw: terminal-bench (terminus 1 framework)], terminal-bench with terminus 1 agent framework","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3.1","publisher":"OpenRouter (listed 2025-08-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-r1-0528","display_name":"DeepSeek R1 0528","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"frontier-r","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.33,"expected_int8_gb":237,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1-0528","openrouter_id":"deepseek/deepseek-r1-0528","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-05-28","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":60.31,"inputs":[{"benchmark":"aime","score_raw":76,"score_norm":70,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1","config":"2025, artificial analysis independent run"},{"benchmark":"arena_elo","score_raw":1421.6,"score_norm":63.28,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":81,"score_norm":78.87,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","config":"16 samples, max gen 64k, pass@1, temp 0.6, top-p 0.95"},{"benchmark":"hle","score_raw":15.85,"score_norm":20.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1","config":"artificial analysis independent run, no tools"},{"benchmark":"mmlu_pro","score_raw":85,"score_norm":75,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","config":"em, max gen 64k"},{"benchmark":"swe_bench_verified","score_raw":57.6,"score_norm":52.89,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","config":"agentless framework, resolved rate"},{"benchmark":"tau_bench","score_raw":63.9,"score_norm":58.53,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","config":"gpt-4.1 as user simulator, pass@1, tau-bench retail"}],"coverage":0.754,"subset_bias":{"offset":4.11,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":2.15,"cache_read_usd_per_m":0.35,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-r1-0528","as_of":"2026-10-06","blended_usd_per_m":0.9125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1369.06,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.089,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.587,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.592,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/tree/main"},{"quant":"int4","weights_gb":410.718,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":13.1,"score_norm":13.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":76,"score_norm":70,"config":"2025, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87.5,"score_norm":84.38,"config":"16 samples, 2025, pass@1, temp 0.6, top-p 0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1421.6,"score_norm":63.28,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1421.8,"score_norm":63.37,"config":"18492 votes, arena entry deepseek-r1-0528, leaderboard text-overall-style_control, rank 112, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1126,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1190,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1110,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1038,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1109,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1156,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81,"score_norm":78.87,"config":"16 samples, max gen 64k, pass@1, temp 0.6, top-p 0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":15.85,"score_norm":20.37,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.7,"score_norm":23.09,"config":"pass@1, text-only prompts","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85,"score_norm":75,"config":"em, max gen 64k","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":57.6,"score_norm":52.89,"config":"agentless framework, resolved rate","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":53.5,"score_norm":44.67,"config":"gpt-4.1 as user simulator, pass@1, tau-bench airline","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":63.9,"score_norm":58.53,"config":"gpt-4.1 as user simulator, pass@1, tau-bench retail","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-0528","measured_date":"2025-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-0528","publisher":"OpenRouter (listed 2025-05-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-r1-distill-llama-70b-reasoning","display_name":"DeepSeek-R1-Distill-Llama-70B","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.16,"expected_int8_gb":12,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1-Distill-Llama-70B","openrouter_id":"deepseek/deepseek-r1-distill-llama-70b","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2025-01-20","release_status":"released","knowledge_cutoff":"2024-07-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-r1-distill-llama-70b"],"intelligence":{"index":16.59,"inputs":[{"benchmark":"aime","score_raw":53.67,"score_norm":42.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.536666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":40.2,"score_norm":21.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.402020202020202 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.13,"score_norm":4.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0513 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":21.93,"score_norm":2.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.219298245614035 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.839,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.847,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B/tree/main"},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":47.131,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.87,"bw_limited_tok_s":2.32,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":53.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70.55,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":94.97,"bw_limited_tok_s":56.98,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.81,"score_norm":9.81,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":9.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.0933333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":53.67,"score_norm":42.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.536666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40.2,"score_norm":21.41,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.402020202020202 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.13,"score_norm":4.6,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0513 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":27.55,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.275510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":26.56,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.265608465608466 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":31.25,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.3125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":21.93,"score_norm":2.57,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.219298245614035 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B","publisher":"Hugging Face"},{"url":"https://github.com/deepseek-ai/deepseek-r1","publisher":"GitHub"},{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1-distill-llama-70b","publisher":"OpenRouter (listed 2025-01-23)"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"deepseek-r1-distill-llama-8b-reasoning","display_name":"DeepSeek-R1-Distill-Llama-8B","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"8b","params_total_b":8.03,"params_active_b":8.03,"params_exact":8030261248,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.1,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.15,"expected_int8_gb":9.8,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1-Distill-Llama-8B","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-01-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-r1-distill-llama-8b"],"intelligence":{"index":13.17,"inputs":[{"benchmark":"aime","score_raw":41.33,"score_norm":26.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":30.2,"score_norm":7.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","config":"artificial analysis harness"},{"benchmark":"livebench","score_raw":40.6,"score_norm":1.33,"trust":2,"source_url":"https://arxiv.org/html/2505.09388v1","config":"livebench 2024-11-25, qwen3 technical report table 19, thinking mode"},{"benchmark":"mmlu_pro","score_raw":54.3,"score_norm":23.83,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=deepseek-r1-distill-llama-8b","config":"artificial analysis harness, mmlu-pro"}],"coverage":0.4762,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16.06,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":18.134,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.061,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":18.135,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B/tree/main"},{"quant":"fp8","weights_gb":8.512,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":10.586,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.818,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.892,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.512,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":10.586,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.72,"bw_limited_tok_s":45.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.99,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34,"bw_limited_tok_s":20.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":417.19,"bw_limited_tok_s":250.31,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.15,"score_norm":6.15,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":41.33,"score_norm":26.66,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":41.33,"score_norm":26.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.413333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":50.4,"score_norm":38,"config":"aime 2024, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"c_eval","score_raw":50.4,"score_norm":null,"config":"qwen3 technical report table 19, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"codeforces_rating","score_raw":1205,"score_norm":null,"config":"elo rating","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49,"score_norm":33.8,"config":"qwen3 technical report table 19, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":30.2,"score_norm":7.32,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":30.2,"score_norm":7.32,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.302020202020202 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49,"score_norm":33.8,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":17.55,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":17.55,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.175510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":59,"score_norm":null,"config":"qwen3 technical report table 19, strict prompt, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":40.6,"score_norm":1.33,"config":"livebench 2024-11-25, qwen3 technical report table 19, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":23.28,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":23.28,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.232804232804233 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":39.6,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":89.1,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":54.3,"score_norm":23.83,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":66.4,"score_norm":null,"config":"qwen3 technical report table 19, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":11.86,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.118577075098814 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-llama-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/deepseek-ai/deepseek-r1","publisher":"GitHub"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"deepseek-r1-distill-qwen-14b-reasoning","display_name":"DeepSeek-R1-Distill-Qwen-14B","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"14b","params_total_b":14.77,"params_active_b":14.77,"params_exact":14770033664,"params_source":"hf_safetensors","vram_est":{"bf16_gb":29.5,"int8_gb":15.7,"int4_gb":8.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.17,"expected_int8_gb":18.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1-Distill-Qwen-14B","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-01-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-r1-distill-qwen-14b"],"intelligence":{"index":26.1,"inputs":[{"benchmark":"aime","score_raw":55.67,"score_norm":44.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.556666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":48.38,"score_norm":32.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.483838383838384 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.13,"score_norm":3.13,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0413 is a 0-1 fraction, x100 at harvest"}],"coverage":0.3492,"subset_bias":{"offset":-1.43,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":29.54,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":32.628,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":29.54,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":32.628,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B/tree/main"},{"quant":"fp8","weights_gb":15.656,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":18.267,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.862,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":11.473,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":15.656,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":18.267,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":24.1,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":41.16,"bw_limited_tok_s":24.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":10.97,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":18.48,"bw_limited_tok_s":11.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":119.79,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":226.81,"bw_limited_tok_s":136.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.71,"score_norm":9.71,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":7.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.0733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":55.67,"score_norm":44.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.556666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":48.38,"score_norm":32.93,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.483838383838384 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.13,"score_norm":3.13,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0413 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":22.11,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.22108843537415 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":37.57,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.375661375661376 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":23.87,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.238683127572016 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-14b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B","publisher":"Hugging Face"},{"url":"https://github.com/deepseek-ai/deepseek-r1","publisher":"GitHub"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"deepseek-r1-distill-qwen-32b-reasoning","display_name":"DeepSeek-R1-Distill-Qwen-32B","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"32b","params_total_b":32.76,"params_active_b":32.76,"params_exact":32763876352,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.5,"int8_gb":34.7,"int4_gb":19.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.76,"expected_int8_gb":26.4,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1-Distill-Qwen-32B","openrouter_id":"deepseek/deepseek-r1-distill-qwen-32b","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-01-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-r1-distill-qwen-32b"],"intelligence":{"index":36.81,"inputs":[{"benchmark":"aime","score_raw":63,"score_norm":53.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.63 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":61.52,"score_norm":51.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.615151515151515 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.58,"score_norm":3.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0458 is a 0-1 fraction, x100 at harvest"}],"coverage":0.3492,"subset_bias":{"offset":-1.62,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":65.52,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.943,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.528,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.952,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/tree/main"},{"quant":"fp8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.656,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":22.804,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.79,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32.76,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":37.12,"bw_limited_tok_s":22.27,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.98,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.33,"bw_limited_tok_s":5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":57.81,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":102.26,"bw_limited_tok_s":61.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":10.96,"score_norm":10.96,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":8.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.0833333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":63,"score_norm":53.75,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.63 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.52,"score_norm":51.44,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.615151515151515 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.58,"score_norm":3.79,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0458 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":22.93,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.229251700680272 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":26.98,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.26984126984127 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":37.57,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.375690607734807 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-distill-qwen-32b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/deepseek-ai/deepseek-r1","publisher":"GitHub"},{"url":"https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B","publisher":"Hugging Face"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"deepseek-r1-reasoning","display_name":"DeepSeek-R1","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"1","size_class":"frontier-r","params_total_b":684.49,"params_active_b":37,"params_exact":684489845504,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.21,"expected_int8_gb":150,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-R1","openrouter_id":"deepseek/deepseek-r1","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":64000,"release_date":"2025-01-22","release_status":"released","knowledge_cutoff":"2024-07-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-r1","deepseek-r1-671b"],"intelligence":{"index":44.87,"inputs":[{"benchmark":"aime","score_raw":73.96,"score_norm":67.45,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.343, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":1.3,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.08, eval=arc-agi-2 semi-private (v2_semi_private), org=deepseek, run: deepseek r1, system_type=cot"},{"benchmark":"arena_elo","score_raw":1398.1,"score_norm":52.8,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":71.5,"score_norm":65.49,"trust":3,"source_url":"https://arxiv.org/html/2501.12948v1","config":"pass@1"},{"benchmark":"hle","score_raw":15.85,"score_norm":20.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: may '25, source value 0.158480074142725 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":83.18,"score_norm":71.97,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.444, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":49.2,"score_norm":43.56,"trust":3,"source_url":"https://arxiv.org/html/2501.12948v1","config":"resolved %, agentless scaffold"},{"benchmark":"tau_bench","score_raw":36.55,"score_norm":22.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may '25, source value 0.365497076023392 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":19.1,"score_norm":16.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-r1-0120","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: jan '25, source value 0.191011235955056 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.7,"completion_usd_per_m":2.5,"cache_read_usd_per_m":null,"context_length":64000,"source":"openrouter","source_model_id":"deepseek/deepseek-r1","as_of":"2026-10-06","blended_usd_per_m":1.15,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1368.98,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.005,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.559,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.413,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.587,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.592,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main"},{"quant":"int4","weights_gb":410.694,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.804,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.559,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.413,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.1,"score_norm":1.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":3.09,"score_norm":3.09,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: jan '25","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-0120","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":24.6,"score_norm":24.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.4,"score_norm":11.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.37,"score_norm":20.37,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: may '25","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":56.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: may '25, source value 0.566666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aider_polyglot","score_raw":53.3,"score_norm":null,"config":"accuracy","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":73.96,"score_norm":67.45,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.343, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":79.8,"score_norm":74.75,"config":"pass@1 (64 samples averaged), no tools","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":76,"score_norm":70,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may '25, source value 0.76 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"alpacaeval_2_0","score_raw":87.6,"score_norm":null,"config":"lc-winrate","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":21.21,"score_norm":null,"config":"cost_per_task_usd=0.0464, eval=arc-agi-1 semi-private (v1_semi_private), org=deepseek, run: 05/28, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":1.3,"score_norm":0,"config":"cost_per_task_usd=0.08, eval=arc-agi-2 semi-private (v2_semi_private), org=deepseek, run: deepseek r1, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1398.1,"score_norm":52.8,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1398.1,"score_norm":52.8,"config":"18524 votes, arena entry deepseek-r1, leaderboard text-overall-style_control, rank 146, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: may '25, source value 0.0142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":1.48,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: jan '25, source value 0.014845000000000027 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-0120","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.5,"score_norm":65.49,"config":"pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.31,"score_norm":79.31,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: may '25, source value 0.813131313131313 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.5,"score_norm":65.49,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":15.85,"score_norm":20.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: may '25, source value 0.158480074142725 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":39.59,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: may '25, source value 0.395918367346939 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":65.9,"score_norm":null,"config":"pass@1-cot, 2024-08 to 2025-01","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":77.04,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: may '25, source value 0.77037037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":97.3,"score_norm":null,"config":"pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":90.8,"score_norm":null,"config":"pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.18,"score_norm":71.97,"config":"edition=mmlu pro v1, runner=platform, stderr=0.444, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84,"score_norm":73.33,"config":"em","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84,"score_norm":73.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-R1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":92.9,"score_norm":null,"config":"em","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.28,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may '25, scicode, source value 0.402777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":49.2,"score_norm":43.56,"config":"resolved %, agentless scaffold","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2501.12948v1","measured_date":"2025-01-22","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":36.55,"score_norm":22.07,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may '25, source value 0.365497076023392 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":6.39,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: jan '25, source value 0.0639175257731959 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-0120","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":19.1,"score_norm":16.59,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: jan '25, source value 0.191011235955056 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1-0120","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.91,"score_norm":12.84,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may '25, source value 0.159090909090909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-r1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/deepseek-ai/DeepSeek-R1","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-r1","publisher":"OpenRouter (listed 2025-01-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v3.2","display_name":"DeepSeek-V3.2","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3.2","size_class":"frontier-v","params_total_b":685.36,"params_active_b":37,"params_exact":685355329792,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1371,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.32,"expected_int8_gb":234,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V3.2","openrouter_id":"deepseek/deepseek-v3.2","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-12-01","release_status":"released","knowledge_cutoff":"2025-07-31","status":"current","supersedes":["deepseek-v3","deepseek-v3.1-terminus"],"superseded_by":null,"aliases":["deepseek-v3.2-671b","deepseek-v3.2-exp"],"intelligence":{"index":59.95,"inputs":[{"benchmark":"aime","score_raw":84.58,"score_norm":80.73,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=1.125, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":4.03,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.12, eval=arc-agi-2 semi-private (v2_semi_private), org=deepseek, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1422.1,"score_norm":63.5,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":80.3,"score_norm":77.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=1.997, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":25.1,"score_norm":33.97,"trust":3,"source_url":"https://arxiv.org/html/2512.02556v1","config":"text-only, pass@1, thinking mode, boxed-answer template (23.9 with official template)"},{"benchmark":"mmlu_pro","score_raw":84.92,"score_norm":74.87,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=0.353, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":67.6,"score_norm":64,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.095, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":80.3,"score_norm":80.4,"trust":3,"source_url":"https://arxiv.org/html/2512.02556v1","config":"pass@1 overall, thinking mode"},{"benchmark":"terminal_bench","score_raw":46.4,"score_norm":48.71,"trust":3,"source_url":"https://arxiv.org/html/2512.02556v1","config":"accuracy, thinking mode, claude code framework (39.3 non-thinking w/ terminus)"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.27,"completion_usd_per_m":0.41,"cache_read_usd_per_m":null,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-v3.2-exp","as_of":"2026-10-06","blended_usd_per_m":0.305,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1370.72,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1439.832,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":726.482,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":763.382,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":689.483,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":724.533,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.2/tree/main"},{"quant":"int4","weights_gb":411.216,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":432.352,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":726.482,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":763.382,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":44.2,"score_norm":44.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":16.6,"score_norm":16.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.5,"score_norm":21.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":32.8,"score_norm":32.8,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.706666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":84.58,"score_norm":80.73,"config":"archived=true, edition=aime v1, reasoning_effort=high, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=1.125, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":93.1,"score_norm":91.38,"config":"pass@1, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92,"score_norm":90,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.92 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":14.53,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.145280235988201 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":57,"score_norm":null,"config":"cost_per_task_usd=0.08, eval=arc-agi-1 semi-private (v1_semi_private), org=deepseek, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.03,"score_norm":0,"config":"cost_per_task_usd=0.12, eval=arc-agi-2 semi-private (v2_semi_private), org=deepseek, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1422.1,"score_norm":63.5,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1425.1,"score_norm":64.84,"config":"47295 votes, arena entry deepseek-v3.2, leaderboard text-overall-style_control, rank 104, run: deepseek-v3.2, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1271.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1360.3,"score_norm":null,"config":"6656 votes, arena entry deepseek-v3.2-thinking, code/overall, leaderboard webdev-overall-raw, rank 82, raw (no style control), run: deepseek-v3.2-thinking, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":51.4,"score_norm":null,"config":"pass@1, search agent, without context management (67.6 with)","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":2386,"score_norm":93.29,"config":"rating, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":2.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1163,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1148,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1093,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1167,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1161,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1157,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1031,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1031,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1173,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1156,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1185,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2-exp","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1181,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":18.19,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.18187 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.3,"score_norm":77.89,"config":"edition=gpqa diamond v1, reasoning_effort=high, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=1.997, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.4,"score_norm":80.85,"config":"pass@1, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.04,"score_norm":83.15,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.84040404040404 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.4,"score_norm":80.85,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":25.1,"score_norm":33.97,"config":"text-only, pass@1, thinking mode, boxed-answer template (23.9 with official template)","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":24.56,"score_norm":33.18,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.245597775718258 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":40.8,"score_norm":57.06,"config":"no tools, text-only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hmmt_feb_2025","score_raw":92.5,"score_norm":null,"config":"pass@1, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":60.68,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.606802721088435 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":83.3,"score_norm":null,"config":"pass@1-cot, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":86.24,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.862433862433862 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.92,"score_norm":74.87,"config":"edition=mmlu pro v1, run: fireworks/deepseek-v3p2-thinking, runner=platform, stderr=0.353, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85,"score_norm":75,"config":"em, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85,"score_norm":75,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":38.89,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.388888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":70.2,"score_norm":null,"config":"resolved %, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":15.56,"score_norm":17.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":67.6,"score_norm":64,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.095, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.1,"score_norm":70.11,"config":"resolved %, thinking mode, internal framework","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70,"score_norm":66.67,"config":"high reasoning","trust":2,"source_type":"aggregator","source_url":"https://www.swebench.com/","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":80.3,"score_norm":80.4,"config":"pass@1 overall, thinking mode","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":90.64,"score_norm":94.19,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.906432748538012 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.4,"score_norm":48.71,"config":"accuracy, thinking mode, claude code framework (39.3 non-thinking w/ terminus)","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2512.02556v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":39.6,"score_norm":40.71,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2026-02-10","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.82,"score_norm":49.2,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.468164794007491 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":35.61,"score_norm":36.01,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.356060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://api-docs.deepseek.com/news/news251201/","publisher":"DeepSeek"},{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.2","publisher":"OpenRouter (listed 2025-12-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4-flash","display_name":"DeepSeek-V4-Flash","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4","size_class":"frontier-v-flash","params_total_b":290.94,"params_active_b":13,"params_exact":290944616402,"params_source":"hf_safetensors","vram_est":{"bf16_gb":582,"int8_gb":308,"int4_gb":175,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.37,"expected_int8_gb":424,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V4-Flash","openrouter_id":"deepseek/deepseek-v4-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-07-31","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-v4-flash-284b","deepseek-v4-flash-reasoning"],"intelligence":{"index":76.97,"inputs":[{"benchmark":"arc_agi_2","score_raw":55.97,"score_norm":56.63,"trust":3,"source_url":"https://arcprize.org/results/deepseek-v4-flash-0731","config":"cost_per_task_usd=0.0453381305, eval=arc-agi-2 semi-private (v2_semi_private), model_group=deepseek-v4-flash-0731, org=deepseek, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1436,"score_norm":69.7,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.694, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":38.55,"score_norm":53.75,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, deepseek v4 flash 0731 (reasoning, max effort)"},{"benchmark":"livebench","score_raw":74.17,"score_norm":75.93,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run deepseek-v4-flash-0731"},{"benchmark":"mmlu_pro","score_raw":86.21,"score_norm":77.02,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.339, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":88.8,"score_norm":87.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.412, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420-high","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=true, run: reasoning, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":67.04,"score_norm":72.99,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning=false, runner=external, stderr=1.35, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.0063,"completion_usd_per_m":1.28,"cache_read_usd_per_m":0.0063,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4-flash","as_of":"2026-10-06","blended_usd_per_m":0.324725,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":581.88,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":611.695,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":308.396,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":324.537,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":174.564,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":184.014,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":159.617,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":168.319,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/tree/main"},{"quant":"int8","weights_gb":308.396,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":324.537,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":1262,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"vendor","trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash","derivation":null}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":26.3,"score_norm":26.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":48.41,"score_norm":48.41,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":52,"score_norm":52,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.4,"score_norm":24.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":51.77,"score_norm":51.77,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: reasoning, source value 0.743333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.25 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":87,"score_norm":null,"config":"cost_per_task_usd=0.021402901000000005, eval=arc-agi-1 semi-private (v1_semi_private), model_group=deepseek-v4-flash-0731, org=deepseek, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/deepseek-v4-flash-0731","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":55.97,"score_norm":56.63,"config":"cost_per_task_usd=0.0453381305, eval=arc-agi-2 semi-private (v2_semi_private), model_group=deepseek-v4-flash-0731, org=deepseek, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/deepseek-v4-flash-0731","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":61.39,"score_norm":62.66,"config":"run deepseek-v4-flash-0731-max, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1436,"score_norm":69.7,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1435.5,"score_norm":69.47,"config":"49001 votes, arena entry deepseek-v4-ch3-text, leaderboard text-overall-style_control, rank 86, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1435.4,"score_norm":69.43,"config":"arena entry deepseek-v4-flash, rank 85, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1581,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1579.1,"score_norm":null,"config":"3537 votes, arena entry deepseek-v4-flash-high-20260730-webdev, code/overall, leaderboard webdev-overall-raw, rank 13, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":16.58,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.16582172712070506 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":53.5,"score_norm":null,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":16.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: reasoning, source value 0.165714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":54.4,"score_norm":null,"config":"0731 checkpoint (via vllm recipe page)","trust":0,"source_type":"community","source_url":"https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1204,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1122,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1211,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1136,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1210,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1155,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1214,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":39.57,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.39570277529095793 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":52.94,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: reasoning, source value 0.529435 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.694, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.899,"score_norm":91.41,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.81,"score_norm":92.69,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: reasoning, source value 0.908080808080808 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.81,"score_norm":92.69,"config":"artificial analysis eval, deepseek v4 flash 0731 (reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.1,"score_norm":88.87,"config":"pass@1; reasoning effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.4,"score_norm":87.89,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":81.33,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.813287016934433 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":38.55,"score_norm":53.75,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: reasoning, source value 0.385542168674699 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":38.55,"score_norm":53.75,"config":"artificial analysis eval, deepseek v4 flash 0731 (reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":40.3,"score_norm":56.32,"config":"pass@1; with tools; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":34.8,"score_norm":48.24,"config":"pass@1; no tools; reasoning effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":29.4,"score_norm":40.29,"config":"pass@1; no tools; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hmmt_2026_feb","score_raw":91.9,"score_norm":null,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":79.18,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: reasoning, source value 0.791836734693877 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":31.52,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, max effort, reasoning=true, run: reasoning, source value 0.315160075329567 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":74.17,"score_norm":75.93,"config":"livebench-2026-06-25 release, run: deepseek v4 flash 0731","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":74.17,"score_norm":75.93,"config":"global average, livebench-2026-06-25 release, run deepseek-v4-flash-0731","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":91.6,"score_norm":null,"config":"pass@1; reasoning effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":88.4,"score_norm":null,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":67.4,"score_norm":null,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":15,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.15 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.21,"score_norm":77.02,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.339, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.206,"score_norm":77.01,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.4,"score_norm":77.33,"config":"em; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mrcr_1m","score_raw":76.9,"score_norm":null,"config":"mmr at 1m context; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":49.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, scicode, source value 0.498842592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":70.2,"score_norm":null,"config":"resolved; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":52.3,"score_norm":78.83,"config":"resolved; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":88.8,"score_norm":87.56,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.412, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":88.8,"score_norm":87.56,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79,"score_norm":76.67,"config":"resolved; reasoning effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.6,"score_norm":76.22,"config":"resolved; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=true, run: reasoning, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":39.38,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.393814432989691 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.04,"score_norm":72.99,"config":"edition=terminal-bench 2.1, reasoning=false, runner=external, stderr=1.35, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.041,"score_norm":72.99,"config":"overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38.64,"score_norm":39.58,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=true, run: reasoning, source value 0.386363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-0420-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.65,"score_norm":86.65,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.786516853932584 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.65,"score_norm":86.65,"config":"artificial analysis eval, deepseek v4 flash 0731 (reasoning, max effort), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.6,"score_norm":60.71,"config":"accuracy; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.9,"score_norm":61.06,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"0731 checkpoint, deepseek-reported harness (via vllm recipe page)","trust":0,"source_type":"community","source_url":"https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"toolathlon","score_raw":43.5,"score_norm":null,"config":"pass@1; reasoning effort=high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash","measured_date":"2026-07-31","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://api-docs.deepseek.com/quick_start/pricing/","publisher":"DeepSeek"},{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash","publisher":"OpenRouter (listed 2026-04-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4-flash-0731","display_name":"DeepSeek V4 Flash 0731","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4","size_class":"frontier-v-flash","params_total_b":304.18,"params_active_b":null,"params_exact":304180418494,"params_source":"hf_safetensors","vram_est":{"bf16_gb":608,"int8_gb":322,"int4_gb":183,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-V4-Flash-0731","openrouter_id":"deepseek/deepseek-v4-flash-0731","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-07-31","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.0118,"completion_usd_per_m":1.28,"cache_read_usd_per_m":0.0118,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4-flash-0731","as_of":"2026-10-06","blended_usd_per_m":0.32885000000000003,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":608.36,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":639.499,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":322.431,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":339.274,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":182.508,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":192.355,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":166.887,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":175.953,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/tree/main"},{"quant":"int8","weights_gb":322.431,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":339.274,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":41,"score_norm":41,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":69.1,"score_norm":69.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.3,"score_norm":34.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1224,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1100,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1236,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1190,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1221,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1186,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1245,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"2.1, deepseek harness (minimal mode), max reasoning effort, temperature=1.0, top_p=0.95.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731","measured_date":null,"observed_at":"2026-09-28","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-0731","publisher":"OpenRouter (listed 2026-07-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4-flash-vision","display_name":"DeepSeek V4 Flash Vision Exp","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4","size_class":"frontier-v-flash","params_total_b":304.65,"params_active_b":null,"params_exact":304646824126,"params_source":"hf_safetensors","vram_est":{"bf16_gb":609,"int8_gb":323,"int4_gb":183,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.33,"expected_int8_gb":429,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","openrouter_id":"deepseek/deepseek-v4-flash-vision-exp","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-21","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-v4-flash-vision-exp"],"intelligence":{"index":77.28,"inputs":[{"benchmark":"gpqa_diamond","score_raw":91.31,"score_norm":93.39,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":34.48,"score_norm":47.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","config":"artificial analysis independent evaluation"},{"benchmark":"livebench","score_raw":76.76,"score_norm":81.69,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"terminal_bench","score_raw":74.16,"score_norm":81.36,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","config":"2.1, artificial analysis independent evaluation"}],"coverage":0.4365,"subset_bias":{"offset":-1.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2156,"completion_usd_per_m":0.6468,"cache_read_usd_per_m":0.00686,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4-flash-vision-exp","as_of":"2026-10-06","blended_usd_per_m":0.3234,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":609.3,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":640.486,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":322.929,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":339.797,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":182.79,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":192.651,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":167.819,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":176.931,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp/tree/main"},{"quant":"int8","weights_gb":322.929,"kv_cache_gb_per_8k":0.7214,"min_serve_gb":339.797,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":47.5,"score_norm":47.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-vision-exp","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":65,"score_norm":65,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-vision-exp","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.8,"score_norm":34.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-vision-exp","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81.33,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"agents_last_exam","score_raw":27.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"apexbench","score_raw":36.5,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":25.7,"score_norm":null,"config":"public split","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"chartography","score_raw":64.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":10.86,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":75.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":59.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"dsbench_hard","score_raw":63.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.31,"score_norm":93.39,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":34.48,"score_norm":47.76,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":76.76,"score_norm":81.69,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.8,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"nl2repo","score_raw":57.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":49.65,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.16,"score_norm":81.36,"config":"2.1, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-flash-vision","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":83.9,"score_norm":92.82,"config":"terminal bench 2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":75.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"zerobench","score_raw":35,"score_norm":null,"config":"pass@5","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-flash-vision-exp","publisher":"OpenRouter (listed 2026-08-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4-pro","display_name":"DeepSeek-V4-Pro","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4","size_class":"frontier-v-pro","params_total_b":1598.84,"params_active_b":49,"params_exact":1598839674782,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3198,"int8_gb":1695,"int4_gb":959,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.28,"expected_int8_gb":471,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V4-Pro","openrouter_id":"deepseek/deepseek-v4-pro","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-13","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["deepseek-v4-pro-0813"],"superseded_by":null,"aliases":["deepseek-v4-pro-1.6t","deepseek-v4-pro-1.6t-a49b","deepseek-v4-pro-reasoning"],"intelligence":{"index":79.4,"inputs":[{"benchmark":"arc_agi_2","score_raw":59.72,"score_norm":60.8,"trust":3,"source_url":"https://arcprize.org/results/deepseek-v4-pro-0813","config":"cost_per_task_usd=0.5969432040000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=deepseek-v4-pro-0813, org=deepseek, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1457.8,"score_norm":79.41,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":92.42,"score_norm":94.96,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=platform, stderr=2.02, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":41.01,"score_norm":57.37,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, deepseek v4 pro 0813 (reasoning, max effort)"},{"benchmark":"livebench","score_raw":71.6,"score_norm":70.22,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench leaderboard row \"deepseek v4 pro\" (open)"},{"benchmark":"mmlu_pro","score_raw":87.25,"score_norm":78.75,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro, runner=platform, stderr=0.335, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":96.4,"score_norm":96,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=external, stderr=0.834, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":96.2,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.961988304093567 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":54.68,"score_norm":58.45,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=external, stderr=1.498, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2088,"completion_usd_per_m":0.4176,"cache_read_usd_per_m":0.0174,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4-pro","as_of":"2026-10-06","blended_usd_per_m":0.261,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":3197.68,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":3358.587,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1694.77,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1780.532,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":959.304,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1008.293,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":864.721,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":908.98,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main"},{"quant":"int8","weights_gb":1694.77,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1780.532,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":26.3,"score_norm":26.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":49.56,"score_norm":49.56,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":59.4,"score_norm":59.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":30.4,"score_norm":30.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":53.2,"score_norm":53.2,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: reasoning, source value 0.753333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.83,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":18.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.1875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":24.26,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.242625368731563 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":90.5,"score_norm":null,"config":"cost_per_task_usd=0.17823142919999996, eval=arc-agi-1 semi-private (v1_semi_private), model_group=deepseek-v4-pro-0813, org=deepseek, run: low, setting=evaluation conducted with 'low' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/deepseek-v4-pro-0813","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":59.72,"score_norm":60.8,"config":"cost_per_task_usd=0.5969432040000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=deepseek-v4-pro-0813, org=deepseek, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/deepseek-v4-pro-0813","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1457.8,"score_norm":79.41,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1459.4,"score_norm":80.13,"config":"3621 votes, arena entry deepseek-v4-pro-high-20260813-text, leaderboard text-overall-style_control, rank 50, run: deepseek-v4-pro-high-20260813, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1458.2,"score_norm":79.59,"config":"arena entry deepseek-v4-pro, rank 51, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1446,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1581.6,"score_norm":null,"config":"2869 votes, arena entry deepseek-v4-pro-high-20260813-webdev, code/overall, leaderboard webdev-overall-raw, rank 12, raw (no style control), run: deepseek-v4-pro-high-20260813, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":18.83,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.18827715939133852 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":83.4,"score_norm":null,"config":"v4-pro-max mode, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":3206,"score_norm":100,"config":"rating, v4-pro-max mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":18,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: reasoning, source value 0.18 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1269,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1160,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1246,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1212,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":948,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1059,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1228,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1000,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1241,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":40.44,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.40435690838555655 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1554,"score_norm":null,"config":"elo, v4-pro-max mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":54.52,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: reasoning, source value 0.5451900000000001 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.42,"score_norm":94.96,"config":"edition=gpqa diamond v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=platform, stderr=2.02, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.424,"score_norm":94.96,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.83,"score_norm":95.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: reasoning, source value 0.928282828282828 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.83,"score_norm":95.54,"config":"artificial analysis eval, deepseek v4 pro 0813 (reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.1,"score_norm":90.28,"config":"v4-pro high (reasoning) mode, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.1,"score_norm":91.69,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":84.4,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.843971631205674 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":41.01,"score_norm":57.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: reasoning, source value 0.410101946246525 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":41.01,"score_norm":57.37,"config":"artificial analysis eval, deepseek v4 pro 0813 (reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48.2,"score_norm":67.94,"config":"pass@1, tools enabled, v4-pro-max mode, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":34.5,"score_norm":47.79,"config":"v4-pro high mode, pass@1, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37.7,"score_norm":52.5,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hmmt_2026_feb","score_raw":94,"score_norm":null,"config":"v4-pro high mode, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":76.46,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: reasoning, source value 0.764625850340136 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":38.32,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, max effort, reasoning=true, run: reasoning, source value 0.383239171374765 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":71.6,"score_norm":70.22,"config":"global average, livebench leaderboard row \"deepseek v4 pro\" (open)","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":77.44,"score_norm":83.2,"config":"livebench-2026-06-25 release, run: deepseek v4 pro 0813","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":77.44,"score_norm":83.2,"config":"global average, livebench-2026-06-25 release, run deepseek-v4-pro-0813","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":89.8,"score_norm":null,"config":"v4-pro high mode, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":21.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.211111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.25,"score_norm":78.75,"config":"edition=mmlu pro v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro, runner=platform, stderr=0.335, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.972,"score_norm":78.29,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.1,"score_norm":78.5,"config":"v4-pro high (reasoning) mode, em","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.5,"score_norm":79.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":50,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, scicode, source value 0.5 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":57.9,"score_norm":null,"config":"v4-pro-max mode, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":76.2,"score_norm":null,"config":"v4-pro-max mode, % resolved","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":55.4,"score_norm":84,"config":"v4-pro-max mode, % resolved","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":96.4,"score_norm":96,"config":"edition=swe-bench verified v1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=external, stderr=0.834, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":96.4,"score_norm":96,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.4,"score_norm":77.11,"config":"v4-pro high mode, % resolved","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.6,"score_norm":78.44,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":96.2,"score_norm":100,"config":"max effort, reasoning, tau2-bench telecom, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://benchmarklist.com/benchmarks/tau2_bench_telecom/","measured_date":"2026-06-10","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":96.2,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.961988304093567 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":39.59,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.395876288659794 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":54.68,"score_norm":58.45,"config":"edition=terminal-bench 2.1, reasoning_effort=max, run: deepseek/deepseek-v4-pro-0813, runner=external, stderr=1.498, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":54.682,"score_norm":58.45,"config":"max reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.21,"score_norm":48.48,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.462121212121212 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro-0424","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.65,"score_norm":86.65,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: reasoning, source value 0.786516853932584 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.65,"score_norm":86.65,"config":"artificial analysis eval, deepseek v4 pro 0813 (reasoning, max effort), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.3,"score_norm":68.59,"config":"v4-pro high mode, accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.9,"score_norm":74,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://api-docs.deepseek.com/quick_start/pricing/","publisher":"DeepSeek"},{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro","publisher":"OpenRouter (listed 2026-04-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4.1-flash","display_name":"DeepSeek V4.1 Flash","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4.1","size_class":"frontier-v-flash","params_total_b":763.21,"params_active_b":16,"params_exact":763205315794,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1526,"int8_gb":809,"int4_gb":458,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.58,"expected_int8_gb":469,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"deepseek-ai/DeepSeek-V4.1-Flash","openrouter_id":"deepseek/deepseek-v4.1-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-09-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["deepseek-v4.1-flash-552b"],"intelligence":{"index":79.33,"inputs":[{"benchmark":"gpqa_diamond","score_raw":90.9,"score_norm":92.82,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","config":null},{"benchmark":"hle","score_raw":36.8,"score_norm":51.18,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","config":null},{"benchmark":"terminal_bench","score_raw":90.6,"score_norm":100,"trust":1,"source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","config":"1m-token context., 2.1, deepseek harness, minimal mode"}],"coverage":0.3174,"subset_bias":{"offset":-0.68,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.003,"completion_usd_per_m":2.4,"cache_read_usd_per_m":0.003,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4.1-flash","as_of":"2026-10-06","blended_usd_per_m":0.60225,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1526.42,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":1603.412,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":809.003,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":850.124,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":457.926,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":481.493,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":510.297,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":536.483,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/tree/main"},{"quant":"int8","weights_gb":809.003,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":850.124,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":39.5,"score_norm":39.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4.1-flash","measured_date":null,"observed_at":"2026-09-11","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.9,"score_norm":92.82,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":63.9,"score_norm":91.03,"config":"harness not specified in the model card., with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":36.8,"score_norm":51.18,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":90.6,"score_norm":100,"config":"1m-token context., 2.1, deepseek harness, minimal mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4.1-flash","publisher":"OpenRouter (listed 2026-09-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"devstral-1-2512","display_name":"Devstral 2 2512","family":{"id":"devstral","vendor":"Mistral AI"},"generation":"1","size_class":"frontier","params_total_b":125.03,"params_active_b":125.03,"params_exact":125025989840,"params_source":"hf_safetensors","vram_est":{"bf16_gb":250,"int8_gb":133,"int4_gb":75,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.23,"expected_int8_gb":29.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Devstral-2-123B-Instruct-2512","openrouter_id":"mistralai/devstral-2512","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-12-09","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["devstral-2-123b-instruct-2512","devstral-2-2512","devstral-2512"],"intelligence":{"index":40.81,"inputs":[{"benchmark":"aime","score_raw":36.67,"score_norm":20.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"2025, artificial analysis independent evaluation"},{"benchmark":"gpqa_diamond","score_raw":59.39,"score_norm":48.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":3.57,"score_norm":2.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":76.16,"score_norm":60.27,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"artificial analysis independent evaluation"},{"benchmark":"swe_bench_verified","score_raw":72.2,"score_norm":69.11,"trust":1,"source_url":"https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512","config":null},{"benchmark":"tau_bench","score_raw":24.85,"score_norm":6.47,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"artificial analysis independent evaluation, tau2/telecom"},{"benchmark":"terminal_bench","score_raw":30.34,"score_norm":29.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/devstral-2","config":"2.1, artificial analysis independent evaluation"}],"coverage":0.7223,"subset_bias":{"offset":1.04,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":2,"cache_read_usd_per_m":0.04,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/devstral-2512","as_of":"2026-10-06","blended_usd_per_m":0.8,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":250.06,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":265.516,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":132.532,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":142.111,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":128.25,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":137.615,"method":"vendor","source_url":"https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512/tree/main"},{"quant":"int4","weights_gb":75.018,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":81.722,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":132.532,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":142.111,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.61,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":125.03,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.37,"bw_limited_tok_s":2.62,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":2.3,"score_norm":2.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/devstral-2512","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":31.3,"score_norm":31.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/devstral-2512","measured_date":null,"observed_at":"2026-08-25","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.6,"score_norm":8.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/devstral-2512","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":32.33,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":36.67,"score_norm":20.84,"config":"2025, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":77.49,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":59.39,"score_norm":48.44,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.57,"score_norm":2.31,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":38.1,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":44.76,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":76.16,"score_norm":60.27,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":61.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.2,"score_norm":69.11,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":24.85,"score_norm":6.47,"config":"artificial analysis independent evaluation, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":30.34,"score_norm":29.81,"config":"2.1, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/devstral-2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":32.6,"score_norm":32.47,"config":"2","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/devstral-2512","publisher":"OpenRouter (listed 2025-12-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"devstral-2-small-24b","display_name":"Devstral Small 2","family":{"id":"devstral","vendor":"Mistral AI"},"generation":"2","size_class":"24b","params_total_b":24,"params_active_b":24,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":48,"int8_gb":25.4,"int4_gb":14.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-12-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["devstral-small-2"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":48,"kv_cache_gb_per_8k":null,"min_serve_gb":51.36,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":25.44,"kv_cache_gb_per_8k":null,"min_serve_gb":27.672,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":14.4,"kv_cache_gb_per_8k":null,"min_serve_gb":16.36,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":25.44,"kv_cache_gb_per_8k":null,"min_serve_gb":27.672,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":14.97,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":25.33,"bw_limited_tok_s":15.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":6.78,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":11.38,"bw_limited_tok_s":6.83,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":77.28,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":139.58,"bw_limited_tok_s":83.75,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://weel.co.jp/media/tech/devstral-2-devstral-small-2/","publisher":"WEEL"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"dots-3-note","display_name":"Dots3-Note Preview","family":{"id":"dots","vendor":"dots-studio"},"generation":"3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"dots-studio/dots-3-note-preview:free","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":512000,"release_date":"2026-08-14","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["dots-3-note-preview"],"intelligence":{"index":79.55,"inputs":[{"benchmark":"arc_agi_2","score_raw":81.4,"score_norm":84.89,"trust":1,"source_url":"https://huggingface.co/dots-studio/dots3-note-prev","config":"public evaluation set"},{"benchmark":"swe_bench_verified","score_raw":78.4,"score_norm":76,"trust":1,"source_url":"https://huggingface.co/dots-studio/dots3-note-prev","config":"live-swe-agent harness, with tools"},{"benchmark":"terminal_bench","score_raw":75.1,"score_norm":82.47,"trust":1,"source_url":"https://huggingface.co/dots-studio/dots3-note-prev","config":"terminal-bench 2.1, terminus-2 harness"}],"coverage":0.2857,"subset_bias":{"offset":-1.72,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":512000,"source":"openrouter","source_model_id":"dots-studio/dots-3-note-preview:free","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":"2026-12-31"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"apex_agent","score_raw":30.8,"score_norm":null,"config":"archipelago harness, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":81.4,"score_norm":84.89,"config":"public evaluation set","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":32.1,"score_norm":null,"config":"general harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":83.3,"score_norm":null,"config":"with context management, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"claw_eval","score_raw":73.4,"score_norm":null,"config":"general set, pass^3","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":92.1,"score_norm":null,"config":"internal react harness, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":52.6,"score_norm":74.41,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":80.4,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":93.9,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"imoanswerbench","score_raw":90.9,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":91.5,"score_norm":null,"config":"v6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"nl2repo","score_raw":49.8,"score_norm":null,"config":"official harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"skillsbench","score_raw":52.8,"score_norm":null,"config":"79 tasks, claude code","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":75.7,"score_norm":null,"config":"live-swe-agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":61,"score_norm":93.33,"config":"live-swe-agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.4,"score_norm":76,"config":"live-swe-agent harness, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":75.1,"score_norm":82.47,"config":"terminal-bench 2.1, terminus-2 harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon","score_raw":55.6,"score_norm":null,"config":"verified, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"widesearch","score_raw":78.9,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildclawbench","score_raw":61.7,"score_norm":null,"config":"openclaw 2026.6.1 agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/dots-studio/dots3-note-prev","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#dots-studio/dots-3-note-preview:free","publisher":"OpenRouter (listed 2026-08-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ember-1","display_name":"Ember-1","family":{"id":"ember","vendor":"fireworks"},"generation":"1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"fireworks/ember-1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":0.3,"context_length":1048576,"source":"openrouter","source_model_id":"fireworks/ember-1","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#fireworks/ember-1","publisher":"OpenRouter (listed 2026-09-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ernie-4.5-424b-a47b-vision","display_name":"ERNIE 4.5 VL 424B A47B","family":{"id":"ernie","vendor":"Baidu"},"generation":"4.5","size_class":"424b-a47b","params_total_b":423.53,"params_active_b":47,"params_exact":423526285184,"params_source":"hf_safetensors","vram_est":{"bf16_gb":847,"int8_gb":449,"int4_gb":254,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"baidu/ERNIE-4.5-VL-424B-A47B-PT","openrouter_id":"baidu/ernie-4.5-vl-424b-a47b","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["image","text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":123000,"release_date":"2025-06-30","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["ernie-4.5-vl-424b-a47b","ernie-4.5-vl-424b-a47b-pt"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.42,"completion_usd_per_m":1.25,"cache_read_usd_per_m":null,"context_length":123000,"source":"openrouter","source_model_id":"baidu/ernie-4.5-vl-424b-a47b","as_of":"2026-10-06","blended_usd_per_m":0.6275,"endpoint_expires_at":"2026-10-08"},"memory":[{"quant":"fp16","weights_gb":847.06,"kv_cache_gb_per_8k":1.8119,"min_serve_gb":891.225,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":847.162,"kv_cache_gb_per_8k":1.8119,"min_serve_gb":891.332,"method":"vendor","source_url":"https://huggingface.co/baidu/ERNIE-4.5-VL-424B-A47B-PT/tree/main"},{"quant":"fp8","weights_gb":448.942,"kv_cache_gb_per_8k":1.8119,"min_serve_gb":473.201,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":254.118,"kv_cache_gb_per_8k":1.8119,"min_serve_gb":268.636,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":448.942,"kv_cache_gb_per_8k":1.8119,"min_serve_gb":473.201,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#baidu/ernie-4.5-vl-424b-a47b","publisher":"OpenRouter (listed 2025-06-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"fugu-1-max","display_name":"Fugu Max","family":{"id":"fugu","vendor":"sakana"},"generation":"1","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"sakana/fugu-max","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-11","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["fugu-max"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.25,"context_length":1000000,"source":"openrouter","source_model_id":"sakana/fugu-max","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"terminal_bench","score_raw":89.5,"score_norm":99.41,"config":"2.1","trust":0,"source_type":"community","source_url":"https://ainformation.jp/article/10731","measured_date":"2026-09-14","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":89.5,"score_norm":99.41,"config":"2.1","trust":0,"source_type":"community","source_url":"https://news.dellows.jp/467866/","measured_date":"2026-09","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sakana/fugu-max","publisher":"OpenRouter (listed 2026-09-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"fugu-1-ultra","display_name":"Fugu Ultra","family":{"id":"fugu","vendor":"sakana"},"generation":"1","size_class":"frontier-ultra","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"sakana/fugu-ultra","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-06-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["fugu-ultra"],"intelligence":{"index":87.97,"inputs":[{"benchmark":"gpqa_diamond","score_raw":95.5,"score_norm":99.3,"trust":2,"source_url":"https://arxiv.org/html/2606.21228v1","config":"sakana fugu technical report, table 1 (model card)"},{"benchmark":"hle","score_raw":50,"score_norm":70.59,"trust":2,"source_url":"https://arxiv.org/html/2606.21228v1","config":"sakana fugu technical report, table 1 (model card)"},{"benchmark":"terminal_bench","score_raw":82.1,"score_norm":90.71,"trust":2,"source_url":"https://arxiv.org/html/2606.21228v1","config":"2.1, sakana fugu technical report, table 1 (model card)"}],"coverage":0.3174,"subset_bias":{"offset":-0.67,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":30,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"sakana/fugu-ultra","as_of":"2026-10-06","blended_usd_per_m":11.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"charxiv_reasoning","score_raw":86.6,"score_norm":null,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.5,"score_norm":99.3,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":50,"score_norm":70.59,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":92,"score_norm":null,"config":"6, sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"long_context_reasoning","score_raw":73.3,"score_norm":null,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":93.6,"score_norm":95.58,"config":"2, sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":58.7,"score_norm":null,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":73.7,"score_norm":100,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_banking","score_raw":20.6,"score_norm":null,"config":"sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.1,"score_norm":90.71,"config":"2.1, sakana fugu technical report, table 1 (model card)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2606.21228v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sakana/fugu-ultra","publisher":"OpenRouter (listed 2026-06-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"fugu-2-ultra","display_name":"Fugu Ultra v2","family":{"id":"fugu","vendor":"sakana"},"generation":"2","size_class":"frontier-ultra","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"sakana/fugu-ultra-v2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-11","release_status":"released","knowledge_cutoff":"2026-08-28","status":"current","supersedes":[],"superseded_by":null,"aliases":["fugu-ultra-v2"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":30,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"sakana/fugu-ultra-v2","as_of":"2026-10-06","blended_usd_per_m":11.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"chartography","score_raw":48.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://sakana.ai/fugu-max-release/","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":74.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://sakana.ai/fugu-max-release/","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.3,"score_norm":96.2,"config":"default routing, fixed gpqa diamond question set, last benchmark run 2026-09-30, openrouter independent run, representative-run accuracy","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/benchmarks/gpqa-diamond","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.5,"score_norm":99.3,"config":null,"trust":0,"source_type":"community","source_url":"https://news.yahoo.co.jp/articles/2e5dfb3d495ffed22aaaae1c20d0086c1071b1ae","measured_date":"2026-09-14","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":81,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://news.yahoo.co.jp/articles/2e5dfb3d495ffed22aaaae1c20d0086c1071b1ae","measured_date":"2026-09-14","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"swefish","score_raw":71.8,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://news.yahoo.co.jp/articles/2e5dfb3d495ffed22aaaae1c20d0086c1071b1ae","measured_date":"2026-09-14","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sakana/fugu-ultra-v2","publisher":"OpenRouter (listed 2026-09-11)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-06T06:08:58.742Z"},{"id":"gemini-2.5-deep-reasoning","display_name":"Gemini 2.5 Deep Think","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image","audio"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-08-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-2.5-deep-think"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://blog.google/products-and-platforms/products/gemini/gemini-2-5-deep-think/","publisher":"Google"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"gemini-2.5-pro","display_name":"Gemini 2.5 Pro","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-03","release_status":"released","knowledge_cutoff":"2025-01-31","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":53.62,"inputs":[{"benchmark":"aime","score_raw":88,"score_norm":85,"trust":3,"source_url":"https://arxiv.org/html/2507.06261v1","config":"aime 2025, single attempt, pass@1"},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.757, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-2-5-pro-2025-06-17-thinking, org=google, run: thinking 32k, system_type=cot"},{"benchmark":"arena_elo","score_raw":1445.6,"score_norm":73.97,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":86.4,"score_norm":86.48,"trust":3,"source_url":"https://arxiv.org/html/2507.06261v1","config":"single attempt, pass@1"},{"benchmark":"hle","score_raw":21.6,"score_norm":28.82,"trust":3,"source_url":"https://arxiv.org/html/2507.06261v1","config":"no tools, pass@1"},{"benchmark":"swe_bench_verified","score_raw":54.4,"score_norm":49.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.23, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":54.09,"score_norm":45.45,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.54093567251462 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":28.46,"score_norm":27.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.284644194756554 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7619,"subset_bias":{"offset":-2.05,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.125,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-2.5-pro","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":"2026-10-20"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.6,"score_norm":1.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":7.24,"score_norm":7.24,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":33.3,"score_norm":33.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":16.1,"score_norm":16.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.91,"score_norm":25.91,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":66,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.66 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aider_polyglot","score_raw":82.2,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":88,"score_norm":85,"config":"aime 2025, single attempt, pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87.67,"score_norm":84.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.876666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":41,"score_norm":null,"config":"cost_per_task_usd=0.4839, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gemini-2-5-pro-2025-06-17-thinking, org=google, run: thinking 16k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"config":"cost_per_task_usd=0.757, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-2-5-pro-2025-06-17-thinking, org=google, run: thinking 32k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1445.6,"score_norm":73.97,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1445.3,"score_norm":73.84,"config":"124807 votes, arena entry gemini-2.5-pro, leaderboard text-overall-style_control, rank 72, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1421.2,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1420.9,"score_norm":null,"config":"24963 votes, arena entry gemini-2.5-pro, document/overall, leaderboard document-overall-raw, rank 32, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1283.3,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1247.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1246.1,"score_norm":null,"config":"88746 votes, arena entry gemini-2.5-pro, leaderboard vision-overall-style_control, rank 44, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1227,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1225.7,"score_norm":null,"config":"2675 votes, arena entry gemini-2.5-pro, code/overall, leaderboard webdev-overall-raw, rank 111, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0257142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1096,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1158,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1230,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1124,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1146,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":8.45,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.08454000000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":89.2,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.4,"score_norm":86.48,"config":"single attempt, pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.44,"score_norm":83.72,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.844444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":21.6,"score_norm":28.82,"config":"no tools, pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.52,"score_norm":30.18,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.225208526413346 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":48.71,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.487074829931973 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":74.2,"score_norm":null,"config":"single attempt, pass@1","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":80.11,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.801058201058201 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu","score_raw":82,"score_norm":null,"config":"single attempt","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.91,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.749132947976879 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.82,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.428240740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":54,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":54.4,"score_norm":49.33,"config":"edition=swe-bench verified v1, runner=external, stderr=2.23, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":59.6,"score_norm":55.11,"config":"single attempt","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2507.06261v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":54.09,"score_norm":45.45,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.54093567251462 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.69,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0969072164948454 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":26.52,"score_norm":25.32,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.265151515151515 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.46,"score_norm":27.6,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.284644194756554 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro","publisher":"OpenRouter (listed 2025-06-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-2.5-pro-03-25","display_name":"gemini-2.5-pro-preview-03-25","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-pro-preview-05-06","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file","audio","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-05-07","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-2.5-pro-preview-03-25","gemini-2.5-pro-preview-05-06"],"intelligence":{"index":63.79,"inputs":[{"benchmark":"gpqa_diamond","score_raw":83.64,"score_norm":82.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":18.03,"score_norm":23.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":85.75,"score_norm":76.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","config":"artificial analysis independent evaluation"}],"coverage":0.3889,"subset_bias":{"offset":1.72,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"gpqa_diamond","score_raw":83.64,"score_norm":82.59,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.03,"score_norm":23.57,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":77.78,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":98,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.75,"score_norm":76.25,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro-preview-05-06","publisher":"OpenRouter (listed 2025-05-07)"}],"last_verified":"2026-09-13","last_enriched_at":null},{"id":"gemini-2.5-pro-06-05","display_name":"Gemini 2.5 Pro Preview 06-05","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-pro-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text","audio"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-06-05","release_status":"preview","knowledge_cutoff":"2025-01-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-2.5-pro-preview","gemini-2.5-pro-preview-06-05"],"intelligence":{"index":53.7,"inputs":[{"benchmark":"gpqa_diamond","score_raw":83.64,"score_norm":82.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: mar' 25, source value 0.836363636363636 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":19.93,"score_norm":26.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-05-06","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: may' 25, source value 0.1993 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":58.3,"score_norm":40.67,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2026-01-08 release, max thinking"}],"coverage":0.3889,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.125,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-2.5-pro-preview","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":23.43,"score_norm":23.43,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: mar' 25","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.64,"score_norm":82.59,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: mar' 25, source value 0.836363636363636 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":19.93,"score_norm":26.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: may' 25, source value 0.1993 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-05-06","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":58.3,"score_norm":40.67,"config":"livebench-2026-01-08 release, max thinking","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2026-01-08","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":77.78,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: mar' 25, source value 0.777777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":41.55,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: may' 25, scicode, source value 0.415509259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-pro-05-06","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-pro-preview","publisher":"OpenRouter (listed 2025-06-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3-deep-reasoning","display_name":"Gemini 3 Deep Think","family":{"id":"gemini","vendor":"Google"},"generation":"3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":null,"release_date":"2025-11-18","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-3-deep-think"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://blog.google/products-and-platforms/products/gemini/gemini-3/","publisher":"Google"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"gemini-3-pro-image","display_name":"Nano Banana Pro (Gemini 3 Pro Image)","family":{"id":"gemini","vendor":"Google"},"generation":"3","size_class":"frontier-image-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3-pro-image","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-06-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-3-pro-image-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":12,"cache_read_usd_per_m":0.2,"context_length":65536,"source":"openrouter","source_model_id":"google/gemini-3-pro-image-preview","as_of":"2026-10-06","blended_usd_per_m":4.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_graphicdesign","score_raw":1253,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-pro-image-preview","measured_date":null,"observed_at":"2026-09-26","admissible":false,"excluded":null},{"benchmark":"design_arena_image","score_raw":1246,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-pro-image-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_imageediting","score_raw":1254,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-pro-image-preview","measured_date":null,"observed_at":"2026-09-10","admissible":false,"excluded":null},{"benchmark":"design_arena_logo","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-pro-image-preview","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3-pro-image","publisher":"OpenRouter (listed 2026-06-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.1-flash-image","display_name":"Nano Banana 2 (Gemini 3.1 Flash Image)","family":{"id":"gemini","vendor":"Google"},"generation":"3.1","size_class":"frontier-image-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.1-flash-image","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-06-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["gemini-2.5-flash-image"],"superseded_by":null,"aliases":["gemini-3.1-flash-image-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":3,"cache_read_usd_per_m":null,"context_length":65536,"source":"openrouter","source_model_id":"google/gemini-3.1-flash-image-preview","as_of":"2026-10-06","blended_usd_per_m":1.125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_graphicdesign","score_raw":1264,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-image-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_image","score_raw":1279,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-image-preview","measured_date":null,"observed_at":"2026-09-10","admissible":false,"excluded":null},{"benchmark":"design_arena_logo","score_raw":1264,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-image-preview","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-image","publisher":"OpenRouter (listed 2026-06-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.1-flash-lite-image","display_name":"Nano Banana 2 Lite (Gemini 3.1 Flash Lite Image)","family":{"id":"gemini","vendor":"Google"},"generation":"3.1","size_class":"frontier-image-flash-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.1-flash-lite-image","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":65536,"release_date":"2026-06-30","release_status":"released","knowledge_cutoff":"2025-01-01","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":1.5,"cache_read_usd_per_m":null,"context_length":65536,"source":"openrouter","source_model_id":"google/gemini-3.1-flash-lite-image","as_of":"2026-10-06","blended_usd_per_m":0.5625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"character_editing_elo","score_raw":1026,"score_norm":null,"config":"no-thinking 1018, sxs human eval elo (+/-6), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"doodle_editing_elo","score_raw":990,"score_norm":null,"config":"no-thinking 985, sxs human eval elo (+/-7), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"general_editing_elo","score_raw":983,"score_norm":null,"config":"no-thinking 954, sxs human eval elo (+/-9), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"general_t2i_elo","score_raw":1059,"score_norm":null,"config":"no-thinking 1055, sxs human eval elo (+/-7), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multi_character_editing_elo","score_raw":1020,"score_norm":null,"config":"no-thinking 1034, sxs human eval elo (+/-8), thinking, up to 5 characters","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multi_input_editing_elo","score_raw":980,"score_norm":null,"config":"no-thinking 974, sxs human eval elo (+/-8), thinking, up to 5 inputs","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multi_product_editing_elo","score_raw":1029,"score_norm":null,"config":"no-thinking 1024, sxs human eval elo (+/-8), thinking, up to 14 products","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"object_environment_editing_elo","score_raw":996,"score_norm":null,"config":"no-thinking 993, sxs human eval elo (+/-9), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"social_media_trends_editing_elo","score_raw":996,"score_norm":null,"config":"no-thinking 968, sxs human eval elo (+/-13), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"social_media_trends_t2i_elo","score_raw":1030,"score_norm":null,"config":"no-thinking 1056, sxs human eval elo (+/-13), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"stylization_elo","score_raw":972,"score_norm":null,"config":"no-thinking 953, sxs human eval elo (+/-8), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"text_editing_elo","score_raw":961,"score_norm":null,"config":"no-thinking 968, sxs human eval elo (+/-10), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"visual_design_t2i_elo","score_raw":1047,"score_norm":null,"config":"no-thinking 1027, sxs human eval elo (+/-9), thinking","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-flash-lite-image/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-image","publisher":"OpenRouter (listed 2026-06-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.1-pro","display_name":"Gemini 3.1 Pro","family":{"id":"gemini","vendor":"Google"},"generation":"3.1","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.1-pro-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-02","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-3.1-pro-preview"],"intelligence":{"index":85.04,"inputs":[{"benchmark":"aime","score_raw":98.12,"score_norm":97.65,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.343, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":77.1,"score_norm":80.11,"trust":1,"source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","config":"arc prize verified; thinking high"},{"benchmark":"arena_elo","score_raw":1487,"score_norm":92.43,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":95.45,"score_norm":99.23,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.046, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":47.03,"score_norm":66.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.470342910101946 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":77,"score_norm":82.22,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench leaderboard row \"gemini 3.1 pro preview high\""},{"benchmark":"mmlu_pro","score_raw":90.99,"score_norm":84.98,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.284, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":78.8,"score_norm":76.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.83, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":70.79,"score_norm":77.4,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.124, task=overall, updated=2026-08-19"}],"coverage":1,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":12,"cache_read_usd_per_m":0.2,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.1-pro-preview","as_of":"2026-10-06","blended_usd_per_m":4.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":8.2,"score_norm":8.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":23.05,"score_norm":23.05,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":68.8,"score_norm":68.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.7,"score_norm":29.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":47.74,"score_norm":47.74,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.79 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":31.88,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":98.12,"score_norm":97.65,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.343, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":98.13,"score_norm":97.66,"config":"gemini 3.1 pro preview (02/26), overall accuracy, vals aime leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":41.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.4125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":32.01,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.320058997050148 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":33.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":77.1,"score_norm":80.11,"config":"arc prize verified; thinking high","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1487,"score_norm":92.43,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1486.3,"score_norm":92.12,"config":"99182 votes, arena entry gemini-3.1-pro-preview, leaderboard text-overall-style_control, rank 14, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1486.3,"score_norm":92.12,"config":"99182 votes, arena entry gemini-3.1-pro-preview, leaderboard text-overall-style_control, rank 14, run: gemini-3.1-pro-preview, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1443.9,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1445.3,"score_norm":null,"config":"45162 votes, arena entry gemini-3.1-pro-preview, document/overall, leaderboard document-overall-raw, rank 24, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1445.3,"score_norm":null,"config":"45162 votes, arena entry gemini-3.1-pro-preview, document/overall, leaderboard document-overall-raw, rank 24, raw (no style control), run: gemini-3.1-pro-preview, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1478.3,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1279.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1277.1,"score_norm":null,"config":"38674 votes, arena entry gemini-3.1-pro-preview, leaderboard vision-overall-style_control, rank 24, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1446.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1446.5,"score_norm":null,"config":"21573 votes, arena entry gemini-3.1-pro-preview, code/overall, leaderboard webdev-overall-raw, rank 48, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1446.5,"score_norm":null,"config":"21573 votes, arena entry gemini-3.1-pro-preview, code/overall, leaderboard webdev-overall-raw, rank 48, raw (no style control), run: gemini-3.1-pro-preview, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":37.54,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.37535846144401597 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":85.9,"score_norm":null,"config":"search + python + browse","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":17.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.177142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1250,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1114,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1226,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1112,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1064,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1281,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1250,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1247,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1067,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1215,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1236,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1153,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1090,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1110,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1109,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1285,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1285,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1119,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1259,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":42.17,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.42166517457475383 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1317,"score_norm":null,"config":"elo","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":23.25,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.23252999999999996 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.45,"score_norm":99.23,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.046, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.14,"score_norm":97.38,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.941414141414141 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.3,"score_norm":97.61,"config":"no tools; thinking high","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":58.85,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.588507743522941 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":47.03,"score_norm":66.22,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.470342910101946 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":51.4,"score_norm":72.65,"config":"search (blocklist) + code; thinking high","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":44.4,"score_norm":62.35,"config":"no tools; thinking high","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77.14,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.771428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":30.33,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, source value 0.303309120258273 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":77,"score_norm":82.22,"config":"global average, livebench leaderboard row \"gemini 3.1 pro preview high\"","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mcp_atlas","score_raw":69.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":15.56,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.155555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":90.99,"score_norm":84.98,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.284, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":91,"score_norm":85,"config":"em, high effort, third-party reported in deepseek-v4-pro model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro","measured_date":"2026-04-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":92.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":82.43,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.824277456647399 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.5,"score_norm":null,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle","score_raw":26.3,"score_norm":null,"config":"1m pointwise","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle","score_raw":84.9,"score_norm":null,"config":"128k average","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":58.91,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.58912037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":59,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":54.2,"score_norm":82,"config":"single attempt","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.8,"score_norm":76.44,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.83, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.6,"score_norm":78.44,"config":"single attempt","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":99.3,"score_norm":100,"config":"tau2/telecom, τ2-bench telecom","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":90.8,"score_norm":94.4,"config":"tau2/retail, τ2-bench retail","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":99.3,"score_norm":100,"config":"tau2/telecom","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":90.8,"score_norm":94.4,"config":"tau2/retail","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":21.44,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.214432989690722 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.79,"score_norm":77.4,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.124, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.79,"score_norm":57.4,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.537878787878788 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":73.78,"score_norm":80.92,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.737827715355805 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-pro-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":68.5,"score_norm":74.71,"config":"terminus-2 harness","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-1-pro/","measured_date":"2026-02-19","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.1-pro-customtools","display_name":"Gemini 3.1 Pro Preview Custom Tools","family":{"id":"gemini","vendor":"Google"},"generation":"3.1","size_class":"frontier-customtools-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.1-pro-preview-customtools","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","audio","image","video","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-02-25","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemini-3.1-pro-preview-customtools"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":12,"cache_read_usd_per_m":0.2,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.1-pro-preview-customtools","as_of":"2026-10-06","blended_usd_per_m":4.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-pro-preview-customtools","publisher":"OpenRouter (listed 2026-02-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.5-flash-lite","display_name":"Gemini 3.5 Flash-Lite","family":{"id":"gemini","vendor":"Google"},"generation":"3.5","size_class":"frontier-flash-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.5-flash-lite","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-07-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["gemini-2.5-flash-lite","gemini-3.1-flash-lite"],"superseded_by":null,"aliases":[],"intelligence":{"index":59.56,"inputs":[{"benchmark":"arc_agi_2","score_raw":10.28,"score_norm":5.87,"trust":3,"source_url":"https://arcprize.org/results/gemini-3-5-flash-lite","config":"cost_per_task_usd=0.13899638250000004, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-5-flash-lite, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1455,"score_norm":78.16,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":83.84,"score_norm":82.87,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.847, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":18.81,"score_norm":24.72,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gemini 3.5 flash-lite"},{"benchmark":"livebench","score_raw":63.94,"score_norm":53.2,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: high"},{"benchmark":"mmlu_pro","score_raw":85.84,"score_norm":76.4,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.344, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.938, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.991, task=overall, updated=2026-08-19"}],"coverage":0.873,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.03,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.5-flash-lite","as_of":"2026-10-06","blended_usd_per_m":0.85,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":14.3,"score_norm":14.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash-lite","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":27.16,"score_norm":27.16,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":49.3,"score_norm":49.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash-lite","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.2,"score_norm":22.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash-lite","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.44,"score_norm":37.44,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, source value 0.746666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":5.23,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":53.5,"score_norm":null,"config":"cost_per_task_usd=0.092850542, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gemini-3-5-flash-lite, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/gemini-3-5-flash-lite","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":10.28,"score_norm":5.87,"config":"cost_per_task_usd=0.13899638250000004, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-5-flash-lite, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/gemini-3-5-flash-lite","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":10.28,"score_norm":5.87,"config":"run gemini-3-5-flash-lite-high, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1455,"score_norm":78.16,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1457.2,"score_norm":79.15,"config":"17985 votes, arena entry gemini-3.5-flash-lite, leaderboard text-overall-style_control, rank 54, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1458.1,"score_norm":79.55,"config":"arena entry gemini-3.5-flash-lite, rank 52, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1448.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1264.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1269.9,"score_norm":null,"config":"3165 votes, arena entry gemini-3.5-flash-lite, leaderboard vision-overall-style_control, rank 29, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1439.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-30","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1449.1,"score_norm":null,"config":"228 votes, arena entry gemini-3.5-flash-lite, code/overall, leaderboard webdev-overall-raw, rank 47, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":32.66,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.3265910091801892 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":32,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, source value 0.31996500000000005 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.84,"score_norm":82.87,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.847, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.838,"score_norm":82.87,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.84,"score_norm":82.87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.838383838383838 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.84,"score_norm":82.87,"config":"artificial analysis eval, gemini 3.5 flash-lite","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.81,"score_norm":24.72,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.188137164040778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.81,"score_norm":24.72,"config":"artificial analysis eval, gemini 3.5 flash-lite","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":63.94,"score_norm":53.2,"config":"livebench-2026-06-25 release, run: high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":63.94,"score_norm":53.2,"config":"global average, livebench-2026-06-25 release, run gemini-3.5-flash-lite-high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.84,"score_norm":76.4,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.344, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.836,"score_norm":76.39,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":79.02,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, source value 0.790173410404624 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.86,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, scicode, source value 0.408564814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.938, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":17.53,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.175257731958763 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.991, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.187,"score_norm":53.16,"config":"high reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.535580524344569 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"config":"artificial analysis eval, gemini 3.5 flash-lite, terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash-lite","publisher":"OpenRouter (listed 2026-07-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.8-flash","display_name":"Gemini 3.8 Flash","family":{"id":"gemini","vendor":"Google"},"generation":"3.8","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.8-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"cloud_balanced","license":"Proprietary","is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["gemini-2.5-flash","gemini-3-flash","gemini-3.5-flash","gemini-3.6-flash","gemini-3.7-flash"],"superseded_by":null,"aliases":[],"intelligence":{"index":89.29,"inputs":[{"benchmark":"arena_elo","score_raw":1494.8,"score_norm":95.9,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":95.25,"score_norm":98.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","config":"artificial analysis harness, high reasoning effort"},{"benchmark":"hle","score_raw":47.82,"score_norm":67.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","config":"artificial analysis harness, high reasoning effort, no tools"},{"benchmark":"terminal_bench","score_raw":87.64,"score_norm":97.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","config":"artificial analysis harness, high reasoning effort, v2.1"}],"coverage":0.3968,"subset_bias":{"offset":0.78,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.75,"completion_usd_per_m":3.75,"cache_read_usd_per_m":0.075,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.8-flash","as_of":"2026-10-06","blended_usd_per_m":1.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":40.2,"score_norm":40.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.3,"score_norm":76.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.9,"score_norm":40.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81.33,"score_norm":null,"config":"artificial analysis harness, high reasoning effort, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1494.8,"score_norm":95.9,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1290.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1582.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":18.29,"score_norm":null,"config":"artificial analysis harness, high reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1310,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1077,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1310,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1275,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1242,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1319,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1273,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1233,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1178,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1291,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1323,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1230,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1308,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.25,"score_norm":98.94,"config":"artificial analysis harness, high reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":47.82,"score_norm":67.38,"config":"artificial analysis harness, high reasoning effort, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":85.61,"score_norm":null,"config":"artificial analysis harness, high reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":56.6,"score_norm":null,"config":"artificial analysis harness, high reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":44.95,"score_norm":null,"config":"artificial analysis harness, high reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.64,"score_norm":97.22,"config":"artificial analysis harness, high reasoning effort, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-8-flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/","publisher":"Google"},{"url":"https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.8-flash","publisher":"OpenRouter (listed 2026-09-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.8-flash-cyber","display_name":"Gemini 3.8 Flash Cyber","family":{"id":"gemini","vendor":"Google"},"generation":"3.8","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"cloud_frontier","license":"Proprietary","is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-02","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"cwe_bench","score_raw":47.2,"score_norm":null,"config":"antigravity agent harness with high thinking, computed by collinear ai (cwe-bench.com), pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/","measured_date":"2026-09-02","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"cwe_bench","score_raw":47.2,"score_norm":null,"config":"computed by collinear ai using antigravity agent harness with high thinking, official public leaderboard, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/cyber/","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"cwe_bench","score_raw":47.2,"score_norm":null,"config":"antigravity agent harness with high thinking, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_3-8_flash_cyber_model_evaluation.pdf","measured_date":"2026-09","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":86.2,"score_norm":null,"config":"final-submission setting, pass@1","trust":1,"source_type":"aggregator","source_url":"https://aireleasetracker.com/model/google/gemini-3.8-flash-cyber","measured_date":"2026-09-02","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":86.2,"score_norm":null,"config":"final-submission setting, gemini api default sampling, pass@1, self-computed with internal antigravity harness that is not cyber specialized, single attempt (no majority voting or parallel test-time compute)","trust":1,"source_type":"vendor_claim","source_url":"https://venturebeat.com/security/googles-gemini-3-8-flash-is-built-for-agents-while-its-cyber-twin-hunts-vulnerabilities","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":86.2,"score_norm":null,"config":"default gemini api sampling, final-submission setting, internal antigravity harness (not cyber-specialized), pass@1, single attempt","trust":1,"source_type":"vendor_claim","source_url":"https://storage.googleapis.com/deepmind-media/gemini/gemini_3-8_flash_cyber_model_evaluation.pdf","measured_date":"2026-09","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":86.2,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/cyber/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gray_swan_ipi_benchmark","score_raw":6,"score_norm":null,"config":"combined attack set, independently evaluated by gray swan, no computer use, transfer only","trust":1,"source_type":"aggregator","source_url":"https://hokai.io/hub/models/gemini-3.8-flash-cyber","measured_date":"2026-09-02","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"real_world_vulnerability_discovery_20_languages","score_raw":71,"score_norm":null,"config":"internal antigravity harness not cyber-specialized, pass@1 / single attempt, recall over 1200 recent confirmed historical vulnerabilities","trust":1,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/gemini-3-8-flash-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-10-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/","publisher":"Google"}],"last_verified":"2026-09-05","last_enriched_at":"2026-10-05T06:07:26.181Z"},{"id":"gemini-4-argon","display_name":"Gemini 4 Argon","family":{"id":"gemini","vendor":"Google"},"generation":"4","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":null,"release_date":"2026-09-30","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":53,"score_norm":53,"config":"high reasoning, intelligence index v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"agents_last_exam","score_raw":39.5,"score_norm":null,"config":"5-hour window, binary pass rate, default ale-claw harness, pass rate, safety filters enabled, self computed","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1525.2,"score_norm":100,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1679.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":51.3,"score_norm":null,"config":"private set, score, sourced from official zapier public leaderboard","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"chartography","score_raw":71.6,"score_norm":null,"config":"official surge leaderboard, without tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":68.17,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"cwe_bench_v1","score_raw":68,"score_norm":null,"config":"official public leaderboard, ranked on pass@1, ties broken on pass@4","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"cyberbench_v1_1","score_raw":77.86,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":77.9,"score_norm":null,"config":"1.1, highest thinking settings, pass@1, self computed using mini-swe agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":65.4,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"frontierswe_v2","score_raw":55,"score_norm":null,"config":"reported from proximal official public leaderboard","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"graphwalks","score_raw":84.2,"score_norm":null,"config":"200 problems, 256k to 1m, bfs (f1), self-computed","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"graphwalks","score_raw":99.7,"score_norm":null,"config":"650 items, bfs (f1), self-computed, up to 128k","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"harveys_legal_agent_benchmark","score_raw":19.58,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"harveys_legal_agent_benchmark","score_raw":19.6,"score_norm":null,"config":"sourced from vals ai","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"ioi","score_raw":100,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"labbench_2","score_raw":88.8,"score_norm":null,"config":"and r, internet access, linux terminal with pre-installed bioinfo tools, python, self computed","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"legal_bench","score_raw":88.3,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"lvbench","score_raw":91.7,"score_norm":null,"config":"1fps for gemini, self computed without tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":69.2,"score_norm":null,"config":"2.0, gemini cua harness, max step length 500, maxed over 3 runs with a single attempt per run, official 08.08 patch, offline subset partial score","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"posttrainbench","score_raw":45.3,"score_norm":null,"config":"10-hour budget on a single nvidia h100 gpu, opencode harness, self-computed, v1.1, weighted aggregate across four base models and seven benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"proofbench_v1_1","score_raw":99,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"riemannbench","score_raw":76,"score_norm":null,"config":"official surge public leaderboard","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":57.58,"score_norm":61.86,"config":"4.0, reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":57.4,"score_norm":61.65,"config":"4.0, highest scoring thinking level, pass@1, self computed","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":44.29,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science_0_1","score_raw":57.6,"score_norm":null,"config":"self computed with 6x verifier timeout","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vals_finance_agent_v2","score_raw":65.4,"score_norm":null,"config":"sourced from vals ai","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":68.9,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":68.9,"score_norm":null,"config":"gemini api highest thinking settings, pass@1, sourced from vals ai","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vals_rsi_index","score_raw":30.55,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench","score_raw":91.91,"score_norm":null,"config":"reasoning effort high, temperature 1.0","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/google_gemini-4-argon","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench","score_raw":91.9,"score_norm":null,"config":"reported from official leaderboard","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/gemini/","measured_date":"2026-09-30","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.llmreference.com/changelog/2026-09","publisher":"LLM Reference"},{"url":"https://benchlm.ai/model-updates","publisher":"BenchLM"},{"url":"https://en.wikipedia.org/wiki/List_of_large_language_models","publisher":"Wikipedia"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"gemma-2-27b-instruct","display_name":"Gemma 2 27B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"2","size_class":"27b","params_total_b":27.23,"params_active_b":27.23,"params_exact":27227128320,"params_source":"hf_safetensors","vram_est":{"bf16_gb":54.5,"int8_gb":28.9,"int4_gb":16.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.27,"expected_int8_gb":7.8,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-2-27b-it","openrouter_id":"google/gemma-2-27b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2024-07-13","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-2-27b-it"],"intelligence":{"index":10.08,"inputs":[{"benchmark":"arena_elo","score_raw":1289.4,"score_norm":4.35,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":40.91,"score_norm":22.41,"trust":2,"source_url":"https://huggingface.co/google/gemma-2-27b-it","config":"accuracy, inc. third-party run), sglang, wasp 0.3.0 harness (writer"},{"benchmark":"livebench","score_raw":38.2,"score_norm":0,"trust":3,"source_url":"https://livebench.ai/","config":"global average (overall), livebench-2024-11-25 release"},{"benchmark":"mmlu_pro","score_raw":44.51,"score_norm":7.52,"trust":3,"source_url":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard","config":"5-shot, bfloat16, huggingface open llm leaderboard v2, raw accuracy (leaderboard-normalised value 38.35)"}],"coverage":0.4762,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.65,"completion_usd_per_m":0.65,"cache_read_usd_per_m":null,"context_length":8192,"source":"openrouter","source_model_id":"google/gemma-2-27b-it","as_of":"2026-10-06","blended_usd_per_m":0.65,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":54.46,"kv_cache_gb_per_8k":null,"min_serve_gb":58.272,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":54.454,"kv_cache_gb_per_8k":null,"min_serve_gb":58.266,"method":"vendor","source_url":"https://huggingface.co/google/gemma-2-27b-it/tree/main"},{"quant":"fp8","weights_gb":28.864,"kv_cache_gb_per_8k":null,"min_serve_gb":31.396,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":16.338,"kv_cache_gb_per_8k":null,"min_serve_gb":18.427,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":28.864,"kv_cache_gb_per_8k":null,"min_serve_gb":31.396,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":13.22,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":27.23,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.33,"bw_limited_tok_s":13.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.98,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":27.23,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":10.03,"bw_limited_tok_s":6.02,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":68.74,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":27.23,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":123.03,"bw_limited_tok_s":73.82,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arena_elo","score_raw":1289.4,"score_norm":4.35,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1289.3,"score_norm":4.31,"config":"75754 votes, arena entry gemma-2-27b-it, leaderboard text-overall-style_control, rank 272, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"bbq_ambig","score_raw":85.99,"score_norm":null,"config":"1-shot top-1, gemma 2 it 27b","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-2-27b-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40.91,"score_norm":22.41,"config":"accuracy, inc. third-party run), sglang, wasp 0.3.0 harness (writer","trust":2,"source_type":"aggregator","source_url":"https://huggingface.co/google/gemma-2-27b-it","measured_date":"2026-04-16","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_main","score_raw":30.72,"score_norm":null,"config":"chain-of-thought, inc. third-party run), wasp harness (writer","trust":2,"source_type":"aggregator","source_url":"https://huggingface.co/google/gemma-2-27b-it","measured_date":"2026-04-16","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":38.2,"score_norm":0,"config":"global average (overall), livebench-2024-11-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":"2024-11-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":44.51,"score_norm":7.52,"config":"5-shot, bfloat16, huggingface open llm leaderboard v2, raw accuracy (leaderboard-normalised value 38.35)","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toxigen","score_raw":38.42,"score_norm":null,"config":"gemma 2 it 27b","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-2-27b-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":51.6,"score_norm":null,"config":"gemma 2 it 27b","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-2-27b-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemma-2-27b-it","publisher":"OpenRouter (listed 2024-07-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemma-3-1b","display_name":"Gemma 3 1B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"1b","params_total_b":1,"params_active_b":1,"params_exact":999885952,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2,"int8_gb":1.1,"int4_gb":0.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.11,"expected_int8_gb":2.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-3-1b-it","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":32000,"release_date":"2025-03-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-3-1b-it"],"intelligence":{"index":0.86,"inputs":[{"benchmark":"aime","score_raw":3.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-1b","config":"aa eval, aime 2025"},{"benchmark":"gpqa_diamond","score_raw":19.2,"score_norm":0,"trust":1,"source_url":"https://arxiv.org/abs/2503.19786","config":"gemma 3 1b it"},{"benchmark":"hle","score_raw":5.33,"score_norm":4.9,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-1b","config":"aa eval, no tools"},{"benchmark":"livebench","score_raw":14.4,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/abs/2505.09388","config":"gemma-3-1b-it, livebench 2024-11-25, measured by the qwen3 team, non-thinking"},{"benchmark":"mmlu_pro","score_raw":14.7,"score_norm":0,"trust":1,"source_url":"https://arxiv.org/abs/2503.19786","config":"gemma 3 1b it"},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-1b","config":"aa eval, tau2-bench telecom"}],"coverage":0.635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":2,"kv_cache_gb_per_8k":null,"min_serve_gb":3.04,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":2,"kv_cache_gb_per_8k":null,"min_serve_gb":3.04,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3-1b-it/tree/main"},{"quant":"fp8","weights_gb":1.06,"kv_cache_gb_per_8k":null,"min_serve_gb":2.1,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.6,"kv_cache_gb_per_8k":null,"min_serve_gb":1.64,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1.06,"kv_cache_gb_per_8k":null,"min_serve_gb":2.1,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":267.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":1,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":608,"bw_limited_tok_s":364.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":140.75,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":1,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":273,"bw_limited_tok_s":163.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":667.77,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":1,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3350,"bw_limited_tok_s":2010,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":3.33,"score_norm":0,"config":"aa eval, aime 2025","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bird_sql","score_raw":6.4,"score_norm":null,"config":"dev","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"facts_grounding","score_raw":36.4,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":34.2,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":19.2,"score_norm":0,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hiddenmath","score_raw":15.8,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":5.33,"score_norm":4.9,"config":"aa eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":14.4,"score_norm":0,"config":"gemma-3-1b-it, livebench 2024-11-25, measured by the qwen3 team, non-thinking","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":1.9,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":48,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":14.7,"score_norm":0,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"simpleqa","score_raw":2.2,"score_norm":null,"config":"gemma 3 1b it","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2503.19786","measured_date":"2025-03-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"config":"aa eval, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/google/gemma-3-1b-it","publisher":"Hugging Face"},{"url":"https://huggingface.co/blog/gemma3","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"gemma-3-270m","display_name":"Gemma 3 270M","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"270m","params_total_b":0.27,"params_active_b":0.27,"params_exact":268098176,"params_source":"hf_safetensors","vram_est":{"bf16_gb":0.5,"int8_gb":0.3,"int4_gb":0.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"google/gemma-3-270m-it","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":32000,"release_date":"2025-08-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-3-270m-it"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":0.54,"kv_cache_gb_per_8k":null,"min_serve_gb":1.551,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":0.536,"kv_cache_gb_per_8k":null,"min_serve_gb":1.547,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3-270m-it/tree/main"},{"quant":"fp8","weights_gb":0.286,"kv_cache_gb_per_8k":null,"min_serve_gb":1.297,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.162,"kv_cache_gb_per_8k":null,"min_serve_gb":1.173,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.286,"kv_cache_gb_per_8k":null,"min_serve_gb":1.297,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":574.67,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.27,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2251.85,"bw_limited_tok_s":1351.11,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":377.59,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.27,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1011.11,"bw_limited_tok_s":606.67,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":881.58,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.27,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":12407.41,"bw_limited_tok_s":7444.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc_c","score_raw":28.2,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"big_bench_hard","score_raw":26.7,"score_norm":null,"config":"few-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":37.7,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":51.2,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"piqa","score_raw":66.2,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":52.3,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3-270m-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/google/gemma-3-270m-it","publisher":"Hugging Face"},{"url":"https://developers.googleblog.com/en/introducing-gemma-3-270m/","publisher":"Google"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"gemma-3-27b-instruct","display_name":"Gemma 3 27B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"27b","params_total_b":27.43,"params_active_b":27.43,"params_exact":27432406640,"params_source":"hf_safetensors","vram_est":{"bf16_gb":54.9,"int8_gb":29.1,"int4_gb":16.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.49,"expected_int8_gb":14.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-3-27b-it","openrouter_id":"google/gemma-3-27b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-03-12","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-3-27b","gemma-3-27b-it"],"intelligence":{"index":20.12,"inputs":[{"benchmark":"aime","score_raw":20.67,"score_norm":0.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-27b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.206666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1365.4,"score_norm":38.23,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":42.83,"score_norm":25.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-27b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.428282828282828 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.4,"score_norm":3.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-27b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.044 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":67.5,"score_norm":45.83,"trust":2,"source_url":"https://arxiv.org/html/2503.19786v1","config":"gemma 3 technical report table 6 [raw: mmlu-pro], instruction-tuned (it)"},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-27b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.105263157894737 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-27b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.08,"completion_usd_per_m":0.45,"cache_read_usd_per_m":0.04,"context_length":131072,"source":"openrouter","source_model_id":"google/gemma-3-27b-it","as_of":"2026-10-06","blended_usd_per_m":0.1725,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":54.86,"kv_cache_gb_per_8k":null,"min_serve_gb":58.7,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":54.865,"kv_cache_gb_per_8k":null,"min_serve_gb":58.705,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3-27b-it/tree/main"},{"quant":"fp8","weights_gb":29.076,"kv_cache_gb_per_8k":null,"min_serve_gb":31.627,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":16.458,"kv_cache_gb_per_8k":null,"min_serve_gb":18.555,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":29.076,"kv_cache_gb_per_8k":null,"min_serve_gb":31.627,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":13.12,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":27.43,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.17,"bw_limited_tok_s":13.3,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.94,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":27.43,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9.95,"bw_limited_tok_s":5.97,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":68.27,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":27.43,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":122.13,"bw_limited_tok_s":73.28,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.1,"score_norm":0.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-27b-it","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":0.27,"score_norm":0.27,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":10.1,"score_norm":10.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-27b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.9,"score_norm":4.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-27b-it","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.36,"score_norm":7.36,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":6.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.0633333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":20.67,"score_norm":0.84,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.206666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":20.8,"score_norm":1,"config":"'gemma 3 27b non-thinking' column [raw: aime 2026 no tools], aime 2026, gemma 4 technical report table 5, no tools, non-thinking","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1365.4,"score_norm":38.23,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1365.9,"score_norm":38.45,"config":"47563 votes, arena entry gemma-3-27b-it, leaderboard text-overall-style_control, rank 182, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1159.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1159.5,"score_norm":null,"config":"17216 votes, arena entry gemma-3-27b-it, leaderboard vision-overall-style_control, rank 93, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.83,"score_norm":25.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.428282828282828 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.4,"score_norm":24.51,"config":"diamond, gemma 3 technical report table 6 [raw: gpqa diamond], instruction-tuned (it), no tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2503.19786v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.4,"score_norm":3.53,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.044 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":31.84,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.318367346938776 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":13.65,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.136507936507936 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":67.5,"score_norm":45.83,"config":"gemma 3 technical report table 6 [raw: mmlu-pro], instruction-tuned (it)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2503.19786v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":48.03,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.480346820809249 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":21.18,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.211805555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":11.38,"score_norm":10.63,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.105263157894737 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":0.82,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00824742268041237 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.49,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4,"score_norm":0,"config":"'gemma 3 27b non-thinking' column [raw: terminal bench hard], gemma 4 technical report table 5, non-thinking, terminal-bench hard","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/google/gemma-3-27b-it","publisher":"Hugging Face"},{"url":"https://huggingface.co/blog/gemma3","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemma-3-4b-instruct","display_name":"Gemma 3 4B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"4b","params_total_b":4.3,"params_active_b":4.3,"params_exact":4300079472,"params_source":"hf_safetensors","vram_est":{"bf16_gb":8.6,"int8_gb":4.6,"int4_gb":2.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.89,"expected_int8_gb":4,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-3-4b-it","openrouter_id":"google/gemma-3-4b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_edge","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-03-12","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-3-4b","gemma-3-4b-it"],"intelligence":{"index":4.36,"inputs":[{"benchmark":"aime","score_raw":12.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-4b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.126666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1303.2,"score_norm":10.51,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":29.09,"score_norm":5.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-4b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.290909090909091 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-4b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0529 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":4.97,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-4b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0497076023391813 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0.37,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-4b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5238,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.1,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"google/gemma-3-4b-it","as_of":"2026-10-06","blended_usd_per_m":0.0625,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":8.6,"kv_cache_gb_per_8k":null,"min_serve_gb":9.772,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":8.6,"kv_cache_gb_per_8k":null,"min_serve_gb":9.772,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3-4b-it/tree/main"},{"quant":"fp8","weights_gb":4.558,"kv_cache_gb_per_8k":null,"min_serve_gb":5.73,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.58,"kv_cache_gb_per_8k":null,"min_serve_gb":3.752,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.558,"kv_cache_gb_per_8k":null,"min_serve_gb":5.73,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":78.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":141.4,"bw_limited_tok_s":84.84,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":36.7,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":63.49,"bw_limited_tok_s":38.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":318.54,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":779.07,"bw_limited_tok_s":467.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":2.7,"score_norm":2.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-4b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.8,"score_norm":4.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-4b-it","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":1,"score_norm":1,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":6.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.0666666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":12.67,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.126666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1303.2,"score_norm":10.51,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1303.3,"score_norm":10.55,"config":"4171 votes, arena entry gemma-3-4b-it, leaderboard text-overall-style_control, rank 265, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.09,"score_norm":5.76,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.290909090909091 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0529 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":28.3,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.282993197278912 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":11.22,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.112169312169312 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":29.94,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.299421965317919 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":7.29,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.0729166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":4.97,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0497076023391813 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":0.41,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00412371134020619 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.37,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemma-3-4b-it","publisher":"OpenRouter (listed 2025-03-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemma-3-n-e4b-instruct","display_name":"gemma-3n-e4b-it","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"frontier","params_total_b":7.85,"params_active_b":7.85,"params_exact":7849978192,"params_source":"hf_safetensors","vram_est":{"bf16_gb":15.7,"int8_gb":8.3,"int4_gb":4.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.75,"expected_int8_gb":6.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-3n-E4B-it","openrouter_id":"google/gemma-3n-e4b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2025-05-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-3n-e4b-it"],"intelligence":{"index":7.58,"inputs":[{"benchmark":"aime","score_raw":14.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","config":"aime 2025, artificial analysis harness"},{"benchmark":"arena_elo","score_raw":1317.4,"score_norm":16.83,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":29.6,"score_norm":6.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","config":"artificial analysis harness"},{"benchmark":"hle","score_raw":4.49,"score_norm":3.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","config":"artificial analysis harness, no tools"},{"benchmark":"mmlu_pro","score_raw":50.6,"score_norm":17.67,"trust":1,"source_url":"https://huggingface.co/google/gemma-3n-E4B-it","config":"0-shot accuracy, card row 'mmlu (pro)', e4b it"},{"benchmark":"tau_bench","score_raw":4.97,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","config":"artificial analysis harness, tau2"},{"benchmark":"terminal_bench","score_raw":0.75,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","config":"artificial analysis harness, v2.1"}],"coverage":0.6428,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":15.7,"kv_cache_gb_per_8k":null,"min_serve_gb":17.014,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":15.7,"kv_cache_gb_per_8k":null,"min_serve_gb":17.014,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3n-E4B-it/tree/main"},{"quant":"fp8","weights_gb":8.321,"kv_cache_gb_per_8k":null,"min_serve_gb":9.635,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.71,"kv_cache_gb_per_8k":null,"min_serve_gb":6.024,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.321,"kv_cache_gb_per_8k":null,"min_serve_gb":9.635,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":44.41,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":7.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":77.45,"bw_limited_tok_s":46.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":20.44,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":7.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34.78,"bw_limited_tok_s":20.87,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":203.85,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":7.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":426.75,"bw_limited_tok_s":256.05,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":3.2,"score_norm":3.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3n-e4b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":14.33,"score_norm":0,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":11.6,"score_norm":0,"config":"0-shot accuracy, aime 2025, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1317.4,"score_norm":16.83,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1318.2,"score_norm":17.19,"config":"22624 votes, arena entry gemma-3n-e4b-it, leaderboard text-overall-style_control, rank 244, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":64.5,"score_norm":null,"config":"0-shot accuracy, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.6,"score_norm":6.48,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":23.7,"score_norm":0,"config":"0-shot, e4b it, relaxedaccuracy/accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.49,"score_norm":3.66,"config":"artificial analysis harness, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":75,"score_norm":null,"config":"0-shot pass@1, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":27.89,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":14.6,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":25.7,"score_norm":null,"config":"0-shot pass@1, e4b it, v5","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":63.6,"score_norm":null,"config":"3-shot pass@1, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":60.7,"score_norm":null,"config":"0-shot accuracy, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":64.9,"score_norm":null,"config":"0-shot accuracy, e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":50.6,"score_norm":17.67,"config":"0-shot accuracy, card row 'mmlu (pro)', e4b it","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-3n-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":26.24,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":4.97,"score_norm":0,"config":"artificial analysis harness, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.75,"score_norm":0,"config":"artificial analysis harness, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3n-e4b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemma-3n-e4b-it","publisher":"OpenRouter (listed 2025-05-20)"}],"last_verified":"2026-08-25","last_enriched_at":null},{"id":"gemma-4-12b","display_name":"Gemma 4 12B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"12b","params_total_b":11.96,"params_active_b":11.96,"params_exact":11959730224,"params_source":"hf_safetensors","vram_est":{"bf16_gb":23.9,"int8_gb":12.7,"int4_gb":7.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"google/gemma-4-12B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2026-06","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":23.92,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":25.586,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":23.92,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":25.586,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-12B/tree/main"},{"quant":"fp8","weights_gb":12.678,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":14.148,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.176,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":8.646,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":12.678,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":14.148,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":29.6,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":50.84,"bw_limited_tok_s":30.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.51,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.83,"bw_limited_tok_s":13.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":143.88,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":280.1,"bw_limited_tok_s":168.06,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/google/gemma-4-12B","publisher":"Hugging Face"},{"url":"https://developers.googleblog.com/gemma-4-12b-the-developer-guide","publisher":"Google"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"gemma-4-12b-unified-instruct","display_name":"Gemma 4 12B Unified","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"12b","params_total_b":11.96,"params_active_b":11.96,"params_exact":11959730224,"params_source":"hf_safetensors","vram_est":{"bf16_gb":23.9,"int8_gb":12.7,"int4_gb":7.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.21,"expected_int8_gb":40.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-4-12B-it","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["gemma-3-12b-instruct"],"superseded_by":null,"aliases":["gemma-4-12b-it","gemma-4-12b-unified"],"intelligence":{"index":51.17,"inputs":[{"benchmark":"aime","score_raw":77.5,"score_norm":71.88,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-12B","config":"12b unified column, 2026, instruction-tuned, no tools"},{"benchmark":"gpqa_diamond","score_raw":75.25,"score_norm":70.77,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-12b","config":"artificial analysis harness, reasoning mode"},{"benchmark":"hle","score_raw":15.66,"score_norm":20.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-12b","config":"artificial analysis harness, no tools, reasoning mode"},{"benchmark":"mmlu_pro","score_raw":77.2,"score_norm":62,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-12B","config":"12b unified column, instruction-tuned (it) checkpoint, vendor cross-size table"},{"benchmark":"tau_bench","score_raw":36.26,"score_norm":21.68,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-12b","config":"artificial analysis harness, reasoning mode, tau2"},{"benchmark":"terminal_bench","score_raw":27.34,"score_norm":26.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-12b","config":"artificial analysis harness, reasoning mode, v2.1"}],"coverage":0.5635,"subset_bias":{"offset":1.17,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":23.92,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":25.586,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":23.92,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":25.586,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-12B-it/tree/main"},{"quant":"fp8","weights_gb":12.678,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":14.148,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.176,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":8.646,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":12.678,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":14.148,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":29.6,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":50.84,"bw_limited_tok_s":30.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.51,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.83,"bw_limited_tok_s":13.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":143.88,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":11.96,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":280.1,"bw_limited_tok_s":168.06,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":22,"score_norm":22,"config":"9 evals, composite index (reasoning variant), native index scale, not percent, out of ~140 models ranked","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":63.67,"score_norm":null,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":77.5,"score_norm":71.88,"config":"12b unified column, 2026, instruction-tuned, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":77.5,"score_norm":71.88,"config":"aime 2026, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":53,"score_norm":null,"config":"12b unified column, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":53,"score_norm":null,"config":"instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":1659,"score_norm":50.53,"config":"12b unified column, instruction-tuned, native elo rating scale, not percent","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"covost","score_raw":38.5,"score_norm":null,"config":"12b unified column, excludes chinese language (vendor footnote *), native scale, not rescaled, speech-translation bleu-style score","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"fleurs","score_raw":0.069,"score_norm":null,"config":"12b unified column, audio wer-style metric, excludes chinese language (vendor footnote *), lower is better, native scale, not rescaled","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.25,"score_norm":70.77,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.8,"score_norm":75.77,"config":"12b unified column, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.8,"score_norm":75.77,"config":"instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":15.66,"score_norm":20.09,"config":"artificial analysis harness, no tools, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.2,"score_norm":4.71,"config":"12b unified column, humanity's last exam, instruction-tuned, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.2,"score_norm":4.71,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.54,"score_norm":null,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":72,"score_norm":null,"config":"12b unified column, instruction-tuned, v6 edition","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":72,"score_norm":null,"config":"v6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":79.7,"score_norm":null,"config":"12b unified column, instruction-tuned, vision+math","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":79.7,"score_norm":null,"config":"vision","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"medxpertqa_mm","score_raw":48.7,"score_norm":null,"config":"12b unified column, instruction-tuned, multimodal medical qa","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":77.2,"score_norm":62,"config":"12b unified column, instruction-tuned (it) checkpoint, vendor cross-size table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":77.2,"score_norm":62,"config":"gemma 4 12b unified column, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":83.4,"score_norm":null,"config":"12b unified column, instruction-tuned, multilingual mmlu","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmmlu","score_raw":83.4,"score_norm":null,"config":"instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.65,"score_norm":null,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.1,"score_norm":null,"config":"12b unified column, instruction-tuned, vision benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.1,"score_norm":null,"config":"vision","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":43.4,"score_norm":30.39,"config":"8 needle 128k, average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mrcr_v2_8_needle_128k_average","score_raw":43.4,"score_norm":null,"config":"128k context, 12b unified column, 8-needle, average, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"omnidocbench_1_5","score_raw":0.164,"score_norm":null,"config":"12b unified column, average edit distance, lower is better, native 0-1 scale, not percent - do not rescale","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":36.26,"score_norm":21.68,"config":"artificial analysis harness, reasoning mode, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":69,"score_norm":65.33,"config":"12b unified column, average over 3 domains, instruction-tuned, tau2/average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":69,"score_norm":65.33,"config":"average over 3 runs, tau2","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-12B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":18.18,"score_norm":15.51,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":27.34,"score_norm":26.28,"config":"artificial analysis harness, reasoning mode, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-12b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/gemma4","publisher":"Hugging Face"},{"url":"https://huggingface.co/google/gemma-4-12B","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"gemma-4-26b-a4b-instruct","display_name":"Gemma 4 26B-A4B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"26b-a4b","params_total_b":25.81,"params_active_b":4,"params_exact":25805936206,"params_source":"hf_safetensors","vram_est":{"bf16_gb":51.6,"int8_gb":27.4,"int4_gb":15.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":8.83,"expected_int8_gb":242,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-4-26B-A4B-it","openrouter_id":"google/gemma-4-26b-a4b-it","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-4-26b-a4b","gemma-4-26b-a4b-it"],"intelligence":{"index":60.91,"inputs":[{"benchmark":"aime","score_raw":88.3,"score_norm":85.38,"trust":2,"source_url":"https://arxiv.org/html/2607.02770v2","config":"26b-a4b column [raw: aime 2026 no tools], aime 2026, gemma 4 technical report table 5, no tools"},{"benchmark":"arena_elo","score_raw":1437.5,"score_norm":70.36,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":79.19,"score_norm":76.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.791919191919192 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":19.32,"score_norm":25.47,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.193234476367006 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.6,"score_norm":71,"trust":2,"source_url":"https://arxiv.org/html/2607.02770v2","config":"26b-a4b column [raw: mmlu pro], gemma 4 technical report table 5"},{"benchmark":"tau_bench","score_raw":43.57,"score_norm":31.43,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.435672514619883 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":38.95,"score_norm":39.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.389513108614232 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":2.78,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.0765,"completion_usd_per_m":0.255,"cache_read_usd_per_m":0.0425,"context_length":262144,"source":"openrouter","source_model_id":"google/gemma-4-26b-a4b-it","as_of":"2026-10-06","blended_usd_per_m":0.121125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":51.62,"kv_cache_gb_per_8k":0.3775,"min_serve_gb":54.579,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":51.612,"kv_cache_gb_per_8k":0.3775,"min_serve_gb":54.57,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-26B-A4B-it/tree/main"},{"quant":"fp8","weights_gb":27.359,"kv_cache_gb_per_8k":0.3775,"min_serve_gb":29.104,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":15.486,"kv_cache_gb_per_8k":0.3775,"min_serve_gb":16.864,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":27.359,"kv_cache_gb_per_8k":0.3775,"min_serve_gb":29.104,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":83.58,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":152,"bw_limited_tok_s":91.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":37.18,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":68.25,"bw_limited_tok_s":38.61,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":334.44,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":837.5,"bw_limited_tok_s":502.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":2.9,"score_norm":2.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-10-05","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":11.04,"score_norm":11.04,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":39.3,"score_norm":39.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-26b-a4b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":16.7,"score_norm":16.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26,"score_norm":26,"config":"aa intelligence index v4.1.1, composite of 9 evals, it variant, native index scale, not percent, thinking on","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26.07,"score_norm":26.07,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":61.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.616666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":88.3,"score_norm":85.38,"config":"26b-a4b column [raw: aime 2026 no tools], aime 2026, gemma 4 technical report table 5, no tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":88.3,"score_norm":85.38,"config":"2026, 26b-a4b column, it, no tools, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1437.5,"score_norm":70.36,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1438.2,"score_norm":70.68,"config":"5809 votes, arena entry significant-otter-text, leaderboard text-overall-style_control, rank 84, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1241.9,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1241.6,"score_norm":null,"config":"22719 votes, arena entry significant-otter-vision, leaderboard vision-overall-style_control, rank 46, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1358.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1361.7,"score_norm":null,"config":"1443 votes, arena entry significant-otter-code, code/overall, leaderboard webdev-overall-raw, rank 81, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_text_elo","score_raw":1438,"score_norm":null,"config":"5, 809 votes, 95% ci +/-8, arena (lmarena/arena.ai) text leaderboard, overall category, rank 84","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-27","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":64.8,"score_norm":null,"config":"26b-a4b column, it, micro avg, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":1718,"score_norm":54,"config":"it, native elo scale, not percent, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":13.43,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.134315 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.8,"score_norm":71.55,"config":"cloudflare provider endpoint, openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.3,"score_norm":70.85,"config":"nextbit provider endpoint, openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.7,"score_norm":71.41,"config":"openrouter hosted eval, parasail provider endpoint","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.2,"score_norm":73.52,"config":"openrouter hosted eval, siliconflow provider endpoint","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":72,"score_norm":66.2,"config":"auto-routing endpoint (mixed providers), openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.19,"score_norm":76.32,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.791919191919192 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.3,"score_norm":80.7,"config":"26b-a4b column [raw: gpqa diamond], diamond, gemma 4 technical report table 5, no tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.3,"score_norm":80.7,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.3,"score_norm":80.7,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-26B-A4B-it","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"graphwalks","score_raw":72.6,"score_norm":null,"config":"context <128k, f1, it, table 9, without thinking","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":19.32,"score_norm":25.47,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.193234476367006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.7,"score_norm":9.85,"config":"26b-a4b column (separate 'hle with search' row = 17.2) [raw: hle], gemma 4 technical report table 5, no tools / no search","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.2,"score_norm":22.35,"config":"humanity's last exam with search tool, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":8.7,"score_norm":9.85,"config":"humanity's last exam, it, no tools, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.2,"score_norm":22.35,"config":"with search","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-26B-A4B-it","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":72.45,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.724489795918367 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":72,"score_norm":null,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":98.5,"score_norm":null,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"infographicvqa","score_raw":77.8,"score_norm":null,"config":"it, low resolution (nmax=280 vision tokens), table 12, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"infographicvqa","score_raw":89.3,"score_norm":null,"config":"it, max resolution (1120 vision tokens), table 6, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":23.63,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.236346516007533 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":77.1,"score_norm":null,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"loft_text_retrieval","score_raw":66.3,"score_norm":null,"config":"128k context, it, recall@k, table 9, without thinking","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":80.3,"score_norm":null,"config":"it, low resolution (nmax=280 vision tokens), table 12, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":82.4,"score_norm":null,"config":"it, max resolution (1120 vision tokens), table 6, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"medxpertqa_mm","score_raw":55.7,"score_norm":null,"config":"it, low resolution (nmax=280 vision tokens), table 12, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"medxpertqa_mm","score_raw":58.1,"score_norm":null,"config":"it, max resolution (1120 vision tokens), table 6, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.6,"score_norm":71,"config":"26b-a4b column [raw: mmlu pro], gemma 4 technical report table 5","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.6,"score_norm":71,"config":"26b-a4b column, gemma 4 technical report table 5, it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.6,"score_norm":71,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-26B-A4B-it","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":86.3,"score_norm":null,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.25,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.692485549132948 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.2,"score_norm":null,"config":"it, low resolution (nmax=280 vision tokens), table 12, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.8,"score_norm":null,"config":"it, max resolution (1120 vision tokens), table 6, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle_128k_average","score_raw":44.1,"score_norm":null,"config":"128k, 8-needle, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtob_eng_kgv","score_raw":48.9,"score_norm":null,"config":"chrf metric, it, table 9, without thinking, ~256k context (full book)","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtob_eng_kgv","score_raw":50,"score_norm":null,"config":"chrf metric, it, table 9, without thinking, ~128k context (half book)","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtob_kgv_eng","score_raw":42.7,"score_norm":null,"config":"chrf metric, it, table 9, without thinking, ~256k context (full book)","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtob_kgv_eng","score_raw":45,"score_norm":null,"config":"chrf metric, it, table 9, without thinking, ~128k context (half book)","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"omnidocbench_1_5","score_raw":0.269,"score_norm":null,"config":"edit-distance metric, low resolution (nmax=280), lower is better, native 0-1 scale, not percent, table 12","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"omnidocbench_1_5","score_raw":0.149,"score_norm":null,"config":"edit-distance metric, lower is better, max resolution, native 0-1 scale, not percent - do not rescale, table 6","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler","score_raw":89.8,"score_norm":null,"config":"128k context, accuracy, it, table 9, without thinking","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler","score_raw":97.3,"score_norm":null,"config":"32k context, accuracy, it, table 9, without thinking","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.05,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.400462962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40,"score_norm":null,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":66.1,"score_norm":61.47,"config":"cloudflare provider endpoint, openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":69.5,"score_norm":66,"config":"nextbit provider endpoint, openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":66.9,"score_norm":62.53,"config":"auto-routing endpoint (mixed providers), openrouter hosted eval","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-26b-a4b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":43.57,"score_norm":31.43,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.435672514619883 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":43,"score_norm":30.67,"config":"domain=telecom sub-score, it, table 5, tau2/telecom, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":85.5,"score_norm":87.33,"config":"domain=retail sub-score, it, table 5, tau2/retail, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76,"score_norm":74.67,"config":"domain=airline sub-score, it, table 5, tau2/airline, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":68.2,"score_norm":64.27,"config":"google hf model card main comparison table [raw: tau2 (average over 3)], published average over airline/retail/telecom, tau2-bench","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-26B-A4B-it","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":11.96,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.119587628865979 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":25,"score_norm":23.53,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.25 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38.95,"score_norm":39.94,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.389513108614232 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":14,"score_norm":10.59,"config":"26b-a4b column [raw: terminal bench hard], gemma 4 technical report table 5, terminal-bench hard","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":14,"score_norm":10.59,"config":"26b-a4b column, it, table 5, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/pdf/2607.02770","measured_date":"2026-06-19","observed_at":"2026-09-02","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/gemma4","publisher":"Hugging Face"},{"url":"https://huggingface.co/google/gemma-4-26B-A4B-it","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemma-4-31b-instruct","display_name":"Gemma 4 31B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"31b","params_total_b":31.27,"params_active_b":31.27,"params_exact":31273088876,"params_source":"hf_safetensors","vram_est":{"bf16_gb":62.5,"int8_gb":33.1,"int4_gb":18.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.85,"expected_int8_gb":61.4,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-4-31B-it","openrouter_id":"google/gemma-4-31b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-4-31b","gemma-4-31b-it"],"intelligence":{"index":64.68,"inputs":[{"benchmark":"aime","score_raw":89.2,"score_norm":86.5,"trust":2,"source_url":"https://arxiv.org/html/2607.02770v1","config":"aime 2026, gemma 4 technical report table 5 (gemma 4 31b column) [raw: aime 2026 no tools], no tools"},{"benchmark":"arena_elo","score_raw":1452.8,"score_norm":77.18,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-31b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":23.63,"score_norm":31.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-31b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.236329935125116 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"trust":2,"source_url":"https://arxiv.org/html/2607.02770v1","config":"gemma 4 technical report table 5 (gemma 4 31b column) [raw: mmlu pro], no tools, thinking mode"},{"benchmark":"swe_bench_verified","score_raw":53.92,"score_norm":48.8,"trust":3,"source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","config":"not the full swe-bench verified set - not comparable to full-set numbers, vals.ai-run subset"},{"benchmark":"tau_bench","score_raw":65.5,"score_norm":60.67,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-31b-non-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.654970760233918 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":43.45,"score_norm":45.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-4-31b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.434456928838951 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.09,"completion_usd_per_m":0.34,"cache_read_usd_per_m":0.05,"context_length":262144,"source":"openrouter","source_model_id":"google/gemma-4-31b-it","as_of":"2026-10-06","blended_usd_per_m":0.15250000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":62.54,"kv_cache_gb_per_8k":1.5099,"min_serve_gb":67.177,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":62.546,"kv_cache_gb_per_8k":1.5099,"min_serve_gb":67.183,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-31B-it/tree/main"},{"quant":"fp8","weights_gb":33.146,"kv_cache_gb_per_8k":1.5099,"min_serve_gb":36.313,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.762,"kv_cache_gb_per_8k":1.5099,"min_serve_gb":21.272,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":33.146,"kv_cache_gb_per_8k":1.5099,"min_serve_gb":36.313,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":22.8,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":31.27,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":38.89,"bw_limited_tok_s":23.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.21,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":31.27,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.73,"bw_limited_tok_s":5.24,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":60.4,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":31.27,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":107.13,"bw_limited_tok_s":64.28,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":4.2,"score_norm":4.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-31b-it","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":14.38,"score_norm":14.38,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":43.4,"score_norm":43.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-31b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.7,"score_norm":14.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-4-31b-it","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":30,"score_norm":30,"config":"aa intelligence index v4.1.1, composite index scale, not percent","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.69,"score_norm":29.69,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":89.2,"score_norm":86.5,"config":"aime 2026, gemma 4 technical report table 5 (gemma 4 31b column) [raw: aime 2026 no tools], no tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":89.2,"score_norm":86.5,"config":"2026, it, no tools, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1452.8,"score_norm":77.18,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1451.3,"score_norm":76.52,"config":"5899 votes, arena entry pteronura-text, leaderboard text-overall-style_control, rank 62, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1424.5,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1423.7,"score_norm":null,"config":"10577 votes, arena entry pteronura-document, document/overall, leaderboard document-overall-raw, rank 30, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1262.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1260.1,"score_norm":null,"config":"33890 votes, arena entry pteronura-vision, leaderboard vision-overall-style_control, rank 33, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1364.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1363.6,"score_norm":null,"config":"8965 votes, arena entry pteronura-code, code/overall, leaderboard webdev-overall-raw, rank 80, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_text_elo","score_raw":1451,"score_norm":null,"config":"5, 898 votes, 95% ci +/-8, listed as 'gemma-4-31b', overall text arena leaderboard, rank 64 overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard/text","measured_date":"2026-08-27","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":74.4,"score_norm":null,"config":"it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"case_law","score_raw":52.63,"score_norm":null,"config":"edition v2, vals.ai","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":2150,"score_norm":79.41,"config":"it, native elo scale, not percent, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"corp_fin","score_raw":59.67,"score_norm":null,"config":"edition v2, shared-max-context split, vals.ai","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":28.35,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.28349746344374815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"finance_agent","score_raw":50.79,"score_norm":null,"config":"edition v1.1, vals.ai","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":15.53,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.155305 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.5,"score_norm":83.8,"config":"as served via openrouter, not vendor headline eval, provider-side quality check, provider=together","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-31b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.3,"score_norm":83.52,"config":"diamond, gemma 4 technical report table 5 (gemma 4 31b column) [raw: gpqa diamond], no tools, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.3,"score_norm":83.52,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.3,"score_norm":83.52,"config":"it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"graphwalks","score_raw":82.3,"score_norm":null,"config":"<128k context, f1, table 9","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":47.23,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.47228253003329 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":23.63,"score_norm":31.81,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.236329935125116 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":19.5,"score_norm":25.74,"config":"gemma 4 31b column [raw: hle no tools], humanity's last exam, no tools (a separate 'hle with search' row reports 26.5), table 5","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":26.5,"score_norm":36.03,"config":"humanity's last exam, it, thinking mode, with search","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":19.5,"score_norm":25.74,"config":"humanity's last exam, it, no tools, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.58,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.75578231292517 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":76,"score_norm":null,"config":"gemma 4 technical report table 5, it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":98.9,"score_norm":null,"config":"gemma 4 technical report table 5, it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"infographicvqa","score_raw":92,"score_norm":null,"config":"max resolution, table 6, thinking mode, vision","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":37.29,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.372881355932203 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":80,"score_norm":null,"config":"6, it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"loft_text_retrieval","score_raw":79.5,"score_norm":null,"config":"128k context, recall@k, table 9","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":85.6,"score_norm":null,"config":"it, thinking mode, vision benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"medxpertqa_mm","score_raw":61.3,"score_norm":null,"config":"it, thinking mode, vision benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"config":"gemma 4 technical report table 5 (gemma 4 31b column) [raw: mmlu pro], no tools, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"config":"dense 31b, it, thinking mode, vendor cross-size table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":88.4,"score_norm":null,"config":"it, multilingual mmlu, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.41,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.734104046242775 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":76.9,"score_norm":null,"config":"it, thinking mode, vision benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mortgagetax","score_raw":61.37,"score_norm":null,"config":"vals.ai task","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle_128k_average","score_raw":66.4,"score_norm":null,"config":"128k context, 8-needle, average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtob_eng_kgv","score_raw":52.9,"score_norm":null,"config":"chrf, half book (~128k context), table 9","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"omnidocbench_1_5","score_raw":0.131,"score_norm":null,"config":"average edit distance, lower is better, native 0-1 scale, not percent - do not rescale","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler","score_raw":96.4,"score_norm":null,"config":"128k context, accuracy, table 9","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler","score_raw":96.8,"score_norm":null,"config":"32k context, accuracy, table 9","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"sage","score_raw":55.03,"score_norm":null,"config":"ranked #1 on this task at time of measurement, vals.ai sage task","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.4,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.434027777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43,"score_norm":null,"config":"gemma 4 technical report table 5, it, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2607.02770","measured_date":"2026-07-02","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":53.92,"score_norm":48.8,"config":"not the full swe-bench verified set - not comparable to full-set numbers, vals.ai-run subset","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":78.5,"score_norm":78,"config":"as served via openrouter, provider-side quality check, provider=crusoe","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/google/gemma-4-31b-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":65.5,"score_norm":60.67,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.654970760233918 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76.9,"score_norm":75.87,"config":"agentic tool-use, average over 3, it, tau2/average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-31B-it","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76.9,"score_norm":75.87,"config":"also mirrored on huggingface.co/google/gemma-4-31b-it) [raw: tau2 (average over 3)], published average over airline/retail/telecom (printed verbatim on the model card, tau-2-bench","trust":1,"source_type":"vendor_claim","source_url":"https://ai.google.dev/gemma/docs/core/model_card_4","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":14.85,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.148453608247423 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":36.36,"score_norm":36.89,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.363636363636364 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.45,"score_norm":45.24,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.434456928838951 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-4-31b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":36,"score_norm":36.47,"config":"gemma 4 31b column (row: 36.0 / 14.0 / 18.0 / 8.0 / 3.0 / 4.0) [raw: terminal bench hard], gemma 4 technical report table 5, terminal-bench hard","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.02770v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"vals_index","score_raw":38.94,"score_norm":null,"config":"vals.ai composite index across legal/finance/coding task suite","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"vals_multimodal_index","score_raw":45.12,"score_norm":null,"config":"vals.ai composite multimodal index","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/google_gemma-4-31b-it","measured_date":"2026-04-09","observed_at":"2026-09-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/gemma4","publisher":"Hugging Face"},{"url":"https://huggingface.co/google/gemma-4-12B","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemma-4-e2b","display_name":"Gemma 4 E2B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"frontier","params_total_b":5.12,"params_active_b":5.12,"params_exact":5123178051,"params_source":"hf_safetensors","vram_est":{"bf16_gb":10.2,"int8_gb":5.4,"int4_gb":3.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.22,"expected_int8_gb":17.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-4-E2B-it","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-4-e2b-it"],"intelligence":{"index":24.96,"inputs":[{"benchmark":"aime","score_raw":37.5,"score_norm":21.88,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E2B-it","config":"2026, no tools"},{"benchmark":"gpqa_diamond","score_raw":43.4,"score_norm":25.92,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E2B-it","config":null},{"benchmark":"mmlu_pro","score_raw":60,"score_norm":33.33,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E2B-it","config":null},{"benchmark":"tau_bench","score_raw":24.5,"score_norm":6,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E2B-it","config":"tau2/average"}],"coverage":0.4049,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":10.24,"kv_cache_gb_per_8k":0.0566,"min_serve_gb":11.297,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":10.247,"kv_cache_gb_per_8k":0.0566,"min_serve_gb":11.304,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-E2B-it/tree/main"},{"quant":"fp8","weights_gb":5.427,"kv_cache_gb_per_8k":0.0566,"min_serve_gb":6.484,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":3.072,"kv_cache_gb_per_8k":0.0566,"min_serve_gb":4.129,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":5.427,"kv_cache_gb_per_8k":0.0566,"min_serve_gb":6.484,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":66.51,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":5.12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":118.75,"bw_limited_tok_s":71.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":31,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":5.12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":53.32,"bw_limited_tok_s":31.99,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":281.91,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":5.12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":654.3,"bw_limited_tok_s":392.58,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":37.5,"score_norm":21.88,"config":"2026, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":21.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":633,"score_norm":0,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":43.4,"score_norm":25.92,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":44,"score_norm":null,"config":"6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":52.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":60,"score_norm":33.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":67.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":44.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle_128k","score_raw":19.1,"score_norm":null,"config":"average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":24.5,"score_norm":6,"config":"tau2/average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E2B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemma/docs/core/model_card_4","publisher":"Google"},{"url":"https://huggingface.co/google/gemma-4-E2B-it","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"gemma-4-e4b","display_name":"Gemma 4 E4B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"4","size_class":"frontier","params_total_b":8,"params_active_b":8,"params_exact":7996156490,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.65,"expected_int8_gb":30.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"google/gemma-4-E4B-it","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gemma-4-e4b-it"],"intelligence":{"index":41.97,"inputs":[{"benchmark":"aime","score_raw":42.5,"score_norm":28.13,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E4B-it","config":"aime 2026, no tools"},{"benchmark":"gpqa_diamond","score_raw":58.6,"score_norm":47.32,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E4B-it","config":null},{"benchmark":"mmlu_pro","score_raw":69.4,"score_norm":49,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E4B-it","config":null},{"benchmark":"tau_bench","score_raw":42.2,"score_norm":29.6,"trust":1,"source_url":"https://huggingface.co/google/gemma-4-E4B-it","config":"average over 3, tau2"}],"coverage":0.4049,"subset_bias":{"offset":13.01,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16,"kv_cache_gb_per_8k":0.1552,"min_serve_gb":17.155,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":15.993,"kv_cache_gb_per_8k":0.1552,"min_serve_gb":17.148,"method":"vendor","source_url":"https://huggingface.co/google/gemma-4-E4B-it/tree/main"},{"quant":"fp8","weights_gb":8.48,"kv_cache_gb_per_8k":0.1552,"min_serve_gb":9.635,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.8,"kv_cache_gb_per_8k":0.1552,"min_serve_gb":5.955,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.48,"kv_cache_gb_per_8k":0.1552,"min_serve_gb":9.635,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":76,"bw_limited_tok_s":45.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":20.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34.13,"bw_limited_tok_s":20.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.8,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":418.75,"bw_limited_tok_s":251.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":42.5,"score_norm":28.13,"config":"aime 2026, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bigbench_extra_hard","score_raw":33.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"codeforces_elo","score_raw":940,"score_norm":8.24,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":58.6,"score_norm":47.32,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":52,"score_norm":null,"config":"v6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":59.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"medxpertqa_mm","score_raw":28.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.4,"score_norm":49,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":76.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":52.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":25.4,"score_norm":7.01,"config":"8 needle 128k, average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":42.2,"score_norm":29.6,"config":"average over 3, tau2","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/google/gemma-4-E4B-it","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemma/docs/core/model_card_4","publisher":"Google"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"glm-0414-z1-32b-reasoning","display_name":"GLM-Z1-32B-0414","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"0414","size_class":"32b","params_total_b":32.57,"params_active_b":32.57,"params_exact":32566081536,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.1,"int8_gb":34.5,"int4_gb":19.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"THUDM/GLM-Z1-32B-0414","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-z1-32b-0414"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":65.14,"kv_cache_gb_per_8k":0.5117,"min_serve_gb":68.909,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.132,"kv_cache_gb_per_8k":0.5117,"min_serve_gb":68.9,"method":"vendor","source_url":"https://huggingface.co/THUDM/GLM-Z1-32B-0414/tree/main"},{"quant":"fp8","weights_gb":34.524,"kv_cache_gb_per_8k":0.5117,"min_serve_gb":36.762,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.542,"kv_cache_gb_per_8k":0.5117,"min_serve_gb":21.054,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.524,"kv_cache_gb_per_8k":0.5117,"min_serve_gb":36.762,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32.57,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":37.33,"bw_limited_tok_s":22.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32.57,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.38,"bw_limited_tok_s":5.03,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":58.13,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32.57,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":102.86,"bw_limited_tok_s":61.71,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":63.6,"score_norm":54.5,"config":"aime25, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":90.6,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":70.5,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.1,"score_norm":57.89,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":84.5,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":59.1,"score_norm":null,"config":"2408-2501, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"omni_math","score_raw":68.4,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":52.6,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"sysbench","score_raw":81.2,"score_norm":null,"config":"isr","trust":1,"source_type":"vendor_claim","source_url":"https://raw.githubusercontent.com/zai-org/GLM-4/main/resources/Bench-Z1-32B.png","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.deeplearning.ai/the-batch/zhipu-ai-builds-smaller-open-models-to-rival-deepseeks","publisher":"DeepLearning.AI"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"glm-4-32b-0414-instruct","display_name":"GLM-4-32B-0414","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4","size_class":"32b","params_total_b":32,"params_active_b":32,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":64,"int8_gb":33.9,"int4_gb":19.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"THUDM/GLM-4-32B-0414","openrouter_id":"z-ai/glm-4-32b","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2025-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4-32b","glm-4-32b-0414"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":64,"kv_cache_gb_per_8k":null,"min_serve_gb":68.48,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":33.92,"kv_cache_gb_per_8k":null,"min_serve_gb":36.896,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.2,"kv_cache_gb_per_8k":null,"min_serve_gb":21.48,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":33.92,"kv_cache_gb_per_8k":null,"min_serve_gb":36.896,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":22.29,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":38,"bw_limited_tok_s":22.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.09,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.53,"bw_limited_tok_s":5.12,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":59.1,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":104.69,"bw_limited_tok_s":62.81,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"bfcl_v3","score_raw":69.6,"score_norm":null,"config":"overall","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hotpotqa","score_raw":63.8,"score_norm":null,"config":"averaged over 3 runs, search/click tools, ~500 sampled cases","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":87.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":88.1,"score_norm":null,"config":"averaged over 3 runs, search/click tools, ~500 sampled cases","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":27.2,"score_norm":19.11,"config":"60 iterations, no yarn, openhands v0.29.1 framework","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":30.7,"score_norm":23,"config":"agentless v1.5.0 framework","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":33.8,"score_norm":26.44,"config":"moatless v0.0.3 framework","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":51.2,"score_norm":41.6,"config":"airline","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":68.7,"score_norm":64.93,"config":"retail","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4-32B-0414","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.deeplearning.ai/the-batch/zhipu-ai-builds-smaller-open-models-to-rival-deepseeks","publisher":"DeepLearning.AI"},{"url":"https://github.com/zai-org/GLM-4","publisher":"GitHub"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"glm-4-9b-0414-instruct","display_name":"GLM-4-9B-0414","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4","size_class":"9b","params_total_b":9.4,"params_active_b":9.4,"params_exact":9400279040,"params_source":"hf_safetensors","vram_est":{"bf16_gb":18.8,"int8_gb":10,"int4_gb":5.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-4-9B-0414","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4-9b-0414"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":18.8,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":20.136,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":18.801,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":20.136,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4-9B-0414/tree/main"},{"quant":"fp8","weights_gb":9.964,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":11.3,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.64,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":6.975,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.964,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":11.3,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":37.36,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":9.4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":64.68,"bw_limited_tok_s":38.81,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":17.13,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":9.4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":29.04,"bw_limited_tok_s":17.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":176.16,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":9.4,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":356.38,"bw_limited_tok_s":213.83,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/zai-org/GLM-4","publisher":"GitHub"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"glm-4.5-355b-a32b","display_name":"GLM-4.5","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.5","size_class":"355b-a32b","params_total_b":358.34,"params_active_b":32,"params_exact":358337791296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":717,"int8_gb":380,"int4_gb":215,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.6,"expected_int8_gb":227,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-4.5","openrouter_id":"z-ai/glm-4.5","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07-28","release_status":"released","knowledge_cutoff":"2024-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4.5","glm-4.5-355b"],"intelligence":{"index":58.89,"inputs":[{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.332, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1411.3,"score_norm":58.69,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":72.22,"score_norm":66.51,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.262, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":14.4,"score_norm":18.24,"trust":3,"source_url":"https://arxiv.org/html/2508.06471v1","config":null},{"benchmark":"mmlu_pro","score_raw":81.22,"score_norm":68.7,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.378, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":64.2,"score_norm":60.22,"trust":3,"source_url":"https://arxiv.org/html/2508.06471v1","config":null},{"benchmark":"tau_bench","score_raw":79.7,"score_norm":79.6,"trust":3,"source_url":"https://arxiv.org/html/2508.06471v1","config":"retail"},{"benchmark":"terminal_bench","score_raw":37.5,"score_norm":38.24,"trust":3,"source_url":"https://arxiv.org/html/2508.06471v1","config":null}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":2.2,"cache_read_usd_per_m":0.11,"context_length":131072,"source":"openrouter","source_model_id":"z-ai/glm-4.5","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":"2026-12-31"},"memory":[{"quant":"fp16","weights_gb":716.68,"kv_cache_gb_per_8k":3.087,"min_serve_gb":755.601,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":716.681,"kv_cache_gb_per_8k":3.087,"min_serve_gb":755.602,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.5/tree/main"},{"quant":"fp8","weights_gb":379.84,"kv_cache_gb_per_8k":3.087,"min_serve_gb":401.919,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":215.004,"kv_cache_gb_per_8k":3.087,"min_serve_gb":228.841,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":379.84,"kv_cache_gb_per_8k":3.087,"min_serve_gb":401.919,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.8,"score_norm":12.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.75,"score_norm":19.75,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":51.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.516666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.332, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":91,"score_norm":88.75,"config":"aime 2024","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":73.67,"score_norm":67.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.736666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1411.3,"score_norm":58.69,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1411.1,"score_norm":58.6,"config":"24403 votes, arena entry glm-4.5, leaderboard text-overall-style_control, rank 131, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"browsecomp","score_raw":26.4,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1185,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1167,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1158,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1095,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":72.22,"score_norm":66.51,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.262, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.1,"score_norm":76.2,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.18,"score_norm":74.9,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.781818181818182 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.4,"score_norm":18.24,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.97,"score_norm":16.13,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.129749768303985 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":44.08,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.440816326530612 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":72.9,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":73.76,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.737566137566138 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":98.2,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.22,"score_norm":68.7,"config":"edition=mmlu pro v1, runner=platform, stderr=0.378, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.6,"score_norm":74.33,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":34.84,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.34837962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":64.2,"score_norm":60.22,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":79.7,"score_norm":79.6,"config":"retail","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":42.98,"score_norm":30.64,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.429824561403509 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.5,"score_norm":38.24,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":21.97,"score_norm":19.96,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.21969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4.5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://z.ai/blog/glm-4.5","publisher":"Zhipu AI"},{"url":"https://huggingface.co/zai-org/GLM-4.5","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5","publisher":"OpenRouter (listed 2025-07-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.5-air-106b-a12b","display_name":"GLM-4.5-Air","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.5","size_class":"106b-a12b","params_total_b":110.47,"params_active_b":12,"params_exact":110468824832,"params_source":"hf_safetensors","vram_est":{"bf16_gb":221,"int8_gb":117,"int4_gb":66.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.49,"expected_int8_gb":174,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-4.5-Air","openrouter_id":"z-ai/glm-4.5-air","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07-28","release_status":"released","knowledge_cutoff":"2024-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4.5-air","glm-4.5-air-106b"],"intelligence":{"index":50.05,"inputs":[{"benchmark":"aime","score_raw":80.67,"score_norm":75.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5-air","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.806666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1373.3,"score_norm":41.75,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":73.33,"score_norm":68.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5-air","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":7.04,"score_norm":7.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5-air","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0704355885078777 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":81.4,"score_norm":69,"trust":2,"source_url":"https://arxiv.org/html/2508.06471v1","config":"glm-4.5 tech report (arxiv 2508.06471) table 4 'reasoning benchmarks', glm-4.5-air column (col 2 of glm-4.5 / glm-4.5-air / o3 / claude opus 4 / gemini 2.5 pro / deepseek r1 0528 / qwen3 235b 2507 / g [raw: mmlu-pro]"},{"benchmark":"swe_bench_verified","score_raw":57.6,"score_norm":52.89,"trust":2,"source_url":"https://arxiv.org/html/2508.06471v1","config":"glm-4.5 tech report table 5, glm-4.5-air column (col 2 of glm-4.5 / glm-4.5-air / o3 / gpt-4.1 / claude opus 4 / claude sonnet 4 / gemini 2.5 pro / deepseek r1 0528 / kimi k2). resolved rate. [raw: swe-bench verified]"},{"benchmark":"tau_bench","score_raw":46.49,"score_norm":35.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5-air","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.464912280701754 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":20.45,"score_norm":18.18,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5-air","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.204545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.13,"completion_usd_per_m":0.85,"cache_read_usd_per_m":0.025,"context_length":131072,"source":"openrouter","source_model_id":"z-ai/glm-4.5-air","as_of":"2026-10-06","blended_usd_per_m":0.31,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":220.94,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":233.531,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":220.94,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":233.531,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.5-Air/tree/main"},{"quant":"fp8","weights_gb":117.098,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":124.496,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":66.282,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":71.14,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":117.098,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":124.496,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":12.71,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.75,"bw_limited_tok_s":12.87,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":250.94,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":558.33,"bw_limited_tok_s":335,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.1,"score_norm":11.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":16.66,"score_norm":16.66,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":45.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.456666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":80.67,"score_norm":75.84,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.806666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":82,"score_norm":77.5,"config":"aime 2025, glm-4.5-air row (columns: aime24 / aime25 / lcb v6 / gpqa / hle / mmlu-pro). s [raw: aime 2025], no tools. third-party evaluation in the intellect-3 technical report (prime intellect), table 2 'evaluations'","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2512.16144v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1373.3,"score_norm":41.75,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1372.7,"score_norm":41.48,"config":"31190 votes, arena entry glm-4.5-air, leaderboard text-overall-style_control, rank 177, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1138,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1143,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1195,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1107,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1070,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1130,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1153,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.33,"score_norm":68.07,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75,"score_norm":70.42,"config":"glm-4.5 tech report table 4, glm-4.5-air column. table header says 'gpqa' without naming a subset, so this is very likely the diamond [raw: gpqa], the paper's science-rl ablation figure references 'gpqa-diamond'","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.04,"score_norm":7.41,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0704355885078777 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.6,"score_norm":12.65,"config":"glm reports hle alongside other no-tool reasoning evals. [raw: humanity's last exam (hle)], glm-4.5 tech report table 4, glm-4.5-air column. tool-use setting not specified in the table","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":37.55,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.375510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":68.36,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.683597883597884 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.4,"score_norm":69,"config":"glm-4.5 tech report (arxiv 2508.06471) table 4 'reasoning benchmarks', glm-4.5-air column (col 2 of glm-4.5 / glm-4.5-air / o3 / claude opus 4 / gemini 2.5 pro / deepseek r1 0528 / qwen3 235b 2507 / g [raw: mmlu-pro]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":30.56,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.305555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":57.6,"score_norm":52.89,"config":"glm-4.5 tech report table 5, glm-4.5-air column (col 2 of glm-4.5 / glm-4.5-air / o3 / gpt-4.1 / claude opus 4 / claude sonnet 4 / gemini 2.5 pro / deepseek r1 0528 / kimi k2). resolved rate. [raw: swe-bench verified]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":46.49,"score_norm":35.32,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.464912280701754 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":77.9,"score_norm":77.2,"config":"glm-4.5-air column. no combined retail+airline average is published for glm-4.5-air anywhere in the report, tau-bench retail domain only. glm-4.5 tech report table 3, the airline domain score is 60.8. [raw: tau-bench (retail)]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":20.45,"score_norm":18.18,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.204545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5-air","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":30,"score_norm":29.41,"config":"glm-4.5 tech report table 5, glm-4.5-air column. paper states: 'we use the terminus framework and standard function calling rather than direct prompting for evaluation.' version number not stated (rep [raw: terminal-bench]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2508.06471v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/zai-org/GLM-4.5","publisher":"Hugging Face"},{"url":"https://z.ai/blog/glm-4.5","publisher":"Zhipu AI"},{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5-air","publisher":"OpenRouter (listed 2025-07-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.5-v","display_name":"GLM 4.5V","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.5","size_class":"frontier","params_total_b":107.71,"params_active_b":null,"params_exact":107710933120,"params_source":"hf_safetensors","vram_est":{"bf16_gb":215,"int8_gb":114,"int4_gb":64.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.98,"expected_int8_gb":111,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-4.5V","openrouter_id":"z-ai/glm-4.5v","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2025-08-11","release_status":"released","knowledge_cutoff":"2024-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4.5v"],"intelligence":{"index":35.31,"inputs":[{"benchmark":"aime","score_raw":73,"score_norm":66.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1352.4,"score_norm":32.43,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":68.38,"score_norm":61.1,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.683838383838384 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.3,"score_norm":6.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0630213160333642 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":22.51,"score_norm":3.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.225146198830409 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-5v","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-2.92,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":1.8,"cache_read_usd_per_m":0.11,"context_length":65536,"source":"openrouter","source_model_id":"z-ai/glm-4.5v","as_of":"2026-10-06","blended_usd_per_m":0.8999999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":215.42,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":227.734,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":215.424,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":227.739,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.5V/tree/main"},{"quant":"fp8","weights_gb":114.173,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":121.425,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":64.626,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":69.401,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":114.173,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":121.425,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":1.43,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":107.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2.53,"bw_limited_tok_s":1.43,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":35.98,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":107.71,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":62.2,"bw_limited_tok_s":37.32,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.6,"score_norm":7.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5v","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9,"score_norm":9,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":73,"score_norm":66.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1352.4,"score_norm":32.43,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1353.3,"score_norm":32.84,"config":"4968 votes, arena entry glm-4.5v, leaderboard text-overall-style_control, rank 195, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1152.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1155.4,"score_norm":null,"config":"3284 votes, arena entry glm-4.5v, leaderboard vision-overall-style_control, rank 98, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":68.38,"score_norm":61.1,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.683838383838384 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.3,"score_norm":6.32,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0630213160333642 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":34.22,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.342176870748299 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":60.42,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.604232804232804 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":50.46,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.504624277456647 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":22.11,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.221064814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":22.51,"score_norm":3.35,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.225146198830409 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-5v","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.5v","publisher":"OpenRouter (listed 2025-08-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.6-v-106b","display_name":"GLM-4.6V","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.6","size_class":"106b","params_total_b":107.71,"params_active_b":106,"params_exact":107710933120,"params_source":"hf_safetensors","vram_est":{"bf16_gb":215,"int8_gb":114,"int4_gb":64.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-4.6V","openrouter_id":"z-ai/glm-4.6v","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-12-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4.6-v","glm-4.6v"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":0.9,"cache_read_usd_per_m":0.05,"context_length":131072,"source":"openrouter","source_model_id":"z-ai/glm-4.6v","as_of":"2026-10-06","blended_usd_per_m":0.44999999999999996,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":215.42,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":227.734,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":215.424,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":227.739,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.6V/tree/main"},{"quant":"fp8","weights_gb":114.173,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":121.425,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":64.626,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":69.401,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":114.173,"kv_cache_gb_per_8k":1.5435,"min_serve_gb":121.425,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":1.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":106,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2.58,"bw_limited_tok_s":1.46,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":36.54,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":106,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":63.21,"bw_limited_tok_s":37.92,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.2,"score_norm":11.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6v","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1378.7,"score_norm":44.16,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1161.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.marktechpost.com/2025/12/09/zhipu-ai-releases-glm-4-6v-a-128k-context-vision-language-model-with-native-tool-calling","publisher":"MarkTechPost"},{"url":"https://venturebeat.com/technology/z-ai-debuts-open-source-glm-4-6v-a-native-tool-calling-vision-model-for","publisher":"VentureBeat"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.6-v-flash-9b","display_name":"GLM-4.6V-Flash","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.6","size_class":"9b","params_total_b":9,"params_active_b":9,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":18,"int8_gb":9.5,"int4_gb":5.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"MIT","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-12-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-4.6v-flash"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":18,"kv_cache_gb_per_8k":null,"min_serve_gb":19.36,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":9.54,"kv_cache_gb_per_8k":null,"min_serve_gb":10.9,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.4,"kv_cache_gb_per_8k":null,"min_serve_gb":6.76,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.54,"kv_cache_gb_per_8k":null,"min_serve_gb":10.9,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":38.95,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":9,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":67.56,"bw_limited_tok_s":40.53,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":17.87,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":9,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":30.33,"bw_limited_tok_s":18.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":182.56,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":9,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":372.22,"bw_limited_tok_s":223.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://venturebeat.com/technology/z-ai-debuts-open-source-glm-4-6v-a-native-tool-calling-vision-model-for","publisher":"VentureBeat"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"glm-4.7-flash","display_name":"GLM 4.7 Flash","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.7","size_class":"frontier-flash","params_total_b":31.22,"params_active_b":null,"params_exact":31221488576,"params_source":"hf_safetensors","vram_est":{"bf16_gb":62.4,"int8_gb":33.1,"int4_gb":18.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":4.98,"expected_int8_gb":165,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-4.7-Flash","openrouter_id":"z-ai/glm-4.7-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":200000,"release_date":"2026-01-19","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":48.1,"inputs":[{"benchmark":"aime","score_raw":91.6,"score_norm":89.5,"trust":1,"source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","config":"aime 2025, card's recommended eval params temperature 1.0 class reasoning settings [raw: aime 25], glm-4.7-flash column, no tools, official z.ai model card table, thinking mode"},{"benchmark":"arena_elo","score_raw":1364.9,"score_norm":38.01,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":58.08,"score_norm":46.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.580808080808081 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":7.6,"score_norm":8.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0759962928637627 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":59.2,"score_norm":54.67,"trust":1,"source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","config":"card's recommended eval params: temperature 0.7, glm-4.7-flash column, instance-set size not stated on card [raw: swe-bench verified], max_new_tokens 16384, official z.ai model card table, top_p 1.0"},{"benchmark":"tau_bench","score_raw":98.83,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.988304093567251 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":21.97,"score_norm":19.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.21969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6826,"subset_bias":{"offset":-0.8,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.0605,"completion_usd_per_m":0.4,"cache_read_usd_per_m":null,"context_length":200000,"source":"openrouter","source_model_id":"z-ai/glm-4.7-flash","as_of":"2026-10-06","blended_usd_per_m":0.145375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":62.44,"kv_cache_gb_per_8k":0.4435,"min_serve_gb":66.006,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":62.444,"kv_cache_gb_per_8k":0.4435,"min_serve_gb":66.01,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash/tree/main"},{"quant":"fp8","weights_gb":33.093,"kv_cache_gb_per_8k":0.4435,"min_serve_gb":35.191,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.732,"kv_cache_gb_per_8k":0.4435,"min_serve_gb":20.176,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":33.093,"kv_cache_gb_per_8k":0.4435,"min_serve_gb":35.191,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":22.84,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":31.22,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":38.95,"bw_limited_tok_s":23.37,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.92,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":31.22,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.74,"bw_limited_tok_s":4.95,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":60.49,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":31.22,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":107.3,"bw_limited_tok_s":64.38,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":14.9,"score_norm":14.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23.28,"score_norm":23.28,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":40.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.406666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.6,"score_norm":89.5,"config":"aime 2025, card's recommended eval params temperature 1.0 class reasoning settings [raw: aime 25], glm-4.7-flash column, no tools, official z.ai model card table, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1364.9,"score_norm":38.01,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1367.3,"score_norm":39.08,"config":"11747 votes, arena entry glm-4.7-flash-fireworks, leaderboard text-overall-style_control, rank 181, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1136,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1183,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1132,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1148,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1037,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1214,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1201,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":58.08,"score_norm":46.59,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.580808080808081 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.2,"score_norm":70.7,"config":"glm-4.7-flash column, no tools [raw: gpqa], reasoning on, row labeled exactly \"gpqa\" on the official model card (diamond subset not explicitly stated)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.6,"score_norm":8.24,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0759962928637627 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.4,"score_norm":18.24,"config":"glm-4.7-flash column, humanity's last exam, no-tools/text-only not explicitly stated on card [raw: hle], official model card table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":60.82,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.608163265306122 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":33.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.336805555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":59.2,"score_norm":54.67,"config":"card's recommended eval params: temperature 0.7, glm-4.7-flash column, instance-set size not stated on card [raw: swe-bench verified], max_new_tokens 16384, official z.ai model card table, top_p 1.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.83,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.988304093567251 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":79.5,"score_norm":79.33,"config":"no per-domain breakdown given), preserved thinking mode recommended by card for multi-turn agentic evals, tau-squared bench as published on the official model card (single headline number, temperature [raw: τ2-bench]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7-Flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":21.97,"score_norm":19.96,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.21969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7-flash","publisher":"OpenRouter (listed 2026-01-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5-turbo","display_name":"GLM 5 Turbo","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"z-ai/glm-5-turbo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":202752,"release_date":"2026-03-15","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":65.52,"inputs":[{"benchmark":"gpqa_diamond","score_raw":84.75,"score_norm":84.15,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, glm-5-turbo"},{"benchmark":"hle","score_raw":27.76,"score_norm":37.88,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, glm-5-turbo"},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-turbo","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-turbo","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-1.46,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.2,"completion_usd_per_m":4,"cache_read_usd_per_m":0.24,"context_length":202752,"source":"openrouter","source_model_id":"z-ai/glm-5-turbo","as_of":"2026-10-06","blended_usd_per_m":1.9,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":26.6,"score_norm":26.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.05,"score_norm":39.05,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":66.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.666666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1264,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1166,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1268,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1261,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1205,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1270,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1270,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.75,"score_norm":84.15,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.847474747474747 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.75,"score_norm":84.15,"config":"artificial analysis eval, glm-5-turbo","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.76,"score_norm":37.88,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.277571825764597 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.76,"score_norm":37.88,"config":"artificial analysis eval, glm-5-turbo","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.2,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.731972789115646 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.63,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.436342592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5-turbo","publisher":"OpenRouter (listed 2026-03-15)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5-v-turbo","display_name":"GLM 5V Turbo","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"z-ai/glm-5v-turbo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":202752,"release_date":"2026-04-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-5v-turbo"],"intelligence":{"index":60.15,"inputs":[{"benchmark":"arena_elo","score_raw":1433.5,"score_norm":68.58,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":80.91,"score_norm":78.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.809090909090909 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":17.15,"score_norm":22.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.171455050973123 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-1.19,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.2,"completion_usd_per_m":4,"cache_read_usd_per_m":0.24,"context_length":202752,"source":"openrouter","source_model_id":"z-ai/glm-5v-turbo","as_of":"2026-10-06","blended_usd_per_m":1.9,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":23.5,"score_norm":23.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":35.35,"score_norm":35.35,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.656666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1433.5,"score_norm":68.58,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1433.2,"score_norm":68.45,"config":"9380 votes, arena entry glm-5v-turbo, leaderboard text-overall-style_control, rank 92, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1415.6,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1417.2,"score_norm":null,"config":"4894 votes, arena entry glm-5v-turbo, document/overall, leaderboard document-overall-raw, rank 33, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1423,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1230.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1229.8,"score_norm":null,"config":"32772 votes, arena entry glm-5v-turbo, leaderboard vision-overall-style_control, rank 54, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1399.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1399.8,"score_norm":null,"config":"2183 votes, arena entry glm-5v-turbo, code/overall, leaderboard webdev-overall-raw, rank 65, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1225,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1102,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1134,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1171,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1267,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1131,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1234,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1208,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1147,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1231,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":957,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1120,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1131,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1164,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1165,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1217,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1117,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.91,"score_norm":78.75,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.809090909090909 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.9,"score_norm":78.73,"config":"artificial analysis gpqa diamond run, mirrored by benchlm, rank 89, row reads 'glm-5v-turbo — z.ai · closed — 80.9%'. distinct row from sibling glm-5-turbo (84.7%). [raw: aa-gpqa diamond (artificial analysis gpqa diamond)]","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/aaGpqaDiamond","measured_date":null,"observed_at":"2026-08-19","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":17.15,"score_norm":22.28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.171455050973123 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.1,"score_norm":22.21,"config":"artificial analysis hle accuracy snapshot, between gemini 3.5 flash-lite 18.8% and mercury 2 17.1%. [raw: aa-hle (artificial analysis humanity's last exam)], no tools (benchlm lists 'hle w/ tools' as a separate leaderboard), rank 89","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/aahle","measured_date":null,"observed_at":"2026-08-19","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":61.09,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.610884353741497 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":72.77,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.727745664739884 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.52,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.435185185185185 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.5,"score_norm":100,"config":"page does not state whether it is a domain average or a single domain. tied at 98.5% wi [raw: τ2-bench (artificial analysis τ2-bench)], rank 7 on benchlm's 'τ2-bench tool-agent-user evaluation' page, row source label 'artificial analysis τ2-bench'","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/tau2-bench","measured_date":null,"observed_at":"2026-08-19","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5v-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5v-turbo","publisher":"OpenRouter (listed 2026-04-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.3","display_name":"GLM 5.3","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.3","size_class":"frontier","params_total_b":753.33,"params_active_b":null,"params_exact":753329940480,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1507,"int8_gb":799,"int4_gb":452,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.71,"expected_int8_gb":565,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-5.3","openrouter_id":"z-ai/glm-5.3","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["glm-4.6","glm-4.7","glm-5","glm-5.1","glm-5.2"],"superseded_by":null,"aliases":[],"intelligence":{"index":83.15,"inputs":[{"benchmark":"arena_elo","score_raw":1486.9,"score_norm":92.38,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"3751 votes, arena entry glm-5.3, leaderboard text-overall-style_control, rank 13, run: max, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":88.13,"score_norm":88.92,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.851, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":42.26,"score_norm":59.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.422613531047266 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.77,"score_norm":77.95,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.335, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":95.4,"score_norm":94.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.938, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":71.54,"score_norm":78.28,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=1.982, task=overall, updated=2026-08-19"}],"coverage":0.6746,"subset_bias":{"offset":1.14,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.07,"completion_usd_per_m":7,"cache_read_usd_per_m":0.065,"context_length":1048576,"source":"openrouter","source_model_id":"z-ai/glm-5.3","as_of":"2026-10-06","blended_usd_per_m":1.8025,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1506.66,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1582.729,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":798.53,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.193,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":755.632,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":794.15,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-5.3/tree/main"},{"quant":"int4","weights_gb":451.998,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":475.334,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":798.53,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.193,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":53.1,"score_norm":53.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":59.1,"score_norm":59.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":74.8,"score_norm":74.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":44.8,"score_norm":44.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":59.51,"score_norm":59.51,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.763333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":14.3,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1486.9,"score_norm":92.38,"config":"3751 votes, arena entry glm-5.3, leaderboard text-overall-style_control, rank 13, run: max, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1598.6,"score_norm":null,"config":"1916 votes, arena entry glm-5.3, code/overall, leaderboard webdev-overall-raw, rank 8, raw (no style control), run: max, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":19.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.191428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1361,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1219,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1316,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1330,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1189,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-09-11","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1191,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1255,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1297,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1326,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":63.45,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.6345 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.13,"score_norm":88.92,"config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.851, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.72,"score_norm":93.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.917171717171717 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.26,"score_norm":59.21,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.422613531047266 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":62.5,"score_norm":88.97,"config":"with tools.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"mmlu_pro","score_raw":86.77,"score_norm":77.95,"config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.335, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":56.48,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, scicode, source value 0.564814814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95.4,"score_norm":94.89,"config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.938, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":50.31,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.503092783505155 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":71.54,"score_norm":78.28,"config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=1.982, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":83.9,"score_norm":92.82,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.838951310861423 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.2,"score_norm":97.88,"config":"2.1, harness: claude code","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3","publisher":"OpenRouter (listed 2026-08-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.3-flash","display_name":"GLM 5.3 Flash","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.3","size_class":"frontier-flash","params_total_b":321.32,"params_active_b":18,"params_exact":321323031390,"params_source":"hf_safetensors","vram_est":{"bf16_gb":643,"int8_gb":341,"int4_gb":193,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.35,"expected_int8_gb":459,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"zai-org/GLM-5.3-Flash","openrouter_id":"z-ai/glm-5.3-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-26","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["glm-5.3-flash-320b"],"intelligence":{"index":78.84,"inputs":[{"benchmark":"arena_elo","score_raw":1473,"score_norm":86.19,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"livebench","score_raw":71.6,"score_norm":70.22,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":86.06,"score_norm":76.77,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"accuracy +/-0.34, vals.ai independent eval"},{"benchmark":"terminal_bench","score_raw":84.3,"score_norm":93.29,"trust":1,"source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","config":"terminal bench 2.1"}],"coverage":0.365,"subset_bias":{"offset":0.93,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.5,"cache_read_usd_per_m":0.03,"context_length":1048576,"source":"openrouter","source_model_id":"z-ai/glm-5.3-flash","as_of":"2026-10-06","blended_usd_per_m":0.2375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":642.64,"kv_cache_gb_per_8k":null,"min_serve_gb":675.492,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":340.599,"kv_cache_gb_per_8k":null,"min_serve_gb":358.349,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":328.337,"kv_cache_gb_per_8k":null,"min_serve_gb":345.474,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash/tree/main"},{"quant":"int4","weights_gb":192.792,"kv_cache_gb_per_8k":null,"min_serve_gb":203.152,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":340.599,"kv_cache_gb_per_8k":null,"min_serve_gb":358.349,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":50.9,"score_norm":50.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.5,"score_norm":71.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-08-29","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":41.8,"score_norm":41.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"agents_last_exam","score_raw":26.3,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1473,"score_norm":86.19,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1587.9,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1278.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1615.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":48.8,"score_norm":null,"config":"v1.0.6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":63.4,"score_norm":null,"config":"mini-swe-agent harness, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1333,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1294,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1324,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1280,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"extractbench","score_raw":80.75,"score_norm":null,"config":"mean","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":55.3,"score_norm":78.38,"config":"full set, hle w/ tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":55.3,"score_norm":78.38,"config":"full set, gpt-5.6-luna judge, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"livebench","score_raw":71.6,"score_norm":70.22,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2026-06-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.06,"score_norm":76.77,"config":"accuracy +/-0.34, vals.ai independent eval","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.3,"score_norm":93.29,"config":"terminal bench 2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.3,"score_norm":93.29,"config":"claude code 2.1.207 harness, terminal-bench 2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.3-Flash","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flash","publisher":"OpenRouter (listed 2026-08-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.3-flashx","display_name":"GLM 5.3 FlashX","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"z-ai/glm-5.3-flashx","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.37,"completion_usd_per_m":1.25,"cache_read_usd_per_m":0.09,"context_length":1048576,"source":"openrouter","source_model_id":"z-ai/glm-5.3-flashx","as_of":"2026-10-06","blended_usd_per_m":0.59,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-flashx","publisher":"OpenRouter (listed 2026-09-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.3-prime","display_name":"GLM 5.3 Prime","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"z-ai/glm-5.3-prime","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2.8,"completion_usd_per_m":8.8,"cache_read_usd_per_m":0.56,"context_length":1000000,"source":"openrouter","source_model_id":"z-ai/glm-5.3-prime","as_of":"2026-10-06","blended_usd_per_m":4.3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.3-prime","publisher":"OpenRouter (listed 2026-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-3.5-turbo","display_name":"GPT-3.5 Turbo","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3.5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-3.5-turbo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":16385,"release_date":"2023-05-28","release_status":"released","knowledge_cutoff":"2021-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":6.92,"inputs":[{"benchmark":"arena_elo","score_raw":1225,"score_norm":0,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"66207 votes, arena entry gpt-3.5-turbo-0125, leaderboard text-overall-style_control, rank 313, run: gpt-3.5-turbo-0125, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":30.56,"score_norm":7.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.287, task=overall, updated=2026-08-19"},{"benchmark":"mmlu_pro","score_raw":46.19,"score_norm":10.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","config":"artificial analysis independent evaluation"}],"coverage":0.3572,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":1.5,"cache_read_usd_per_m":null,"context_length":16385,"source":"openrouter","source_model_id":"openai/gpt-3.5-turbo","as_of":"2026-10-06","blended_usd_per_m":0.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":10.7,"score_norm":10.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":5.5,"score_norm":5.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":3.2,"score_norm":3.2,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1225,"score_norm":0,"config":"66207 votes, arena entry gpt-3.5-turbo-0125, leaderboard text-overall-style_control, rank 313, run: gpt-3.5-turbo-0125, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":30.56,"score_norm":7.83,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.287, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.7,"score_norm":6.62,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.7,"score_norm":6.62,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.296969696969697 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"math_500","score_raw":44.07,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":46.19,"score_norm":10.32,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-35-turbo","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo","publisher":"OpenRouter (listed 2023-05-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-3.5-turbo-0613","display_name":"GPT-3.5 Turbo (older v0613)","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3.5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-3.5-turbo-0613","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":4095,"release_date":"2024-01-25","release_status":"released","knowledge_cutoff":"2021-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":2,"cache_read_usd_per_m":null,"context_length":4095,"source":"openrouter","source_model_id":"openai/gpt-3.5-turbo-0613","as_of":"2026-10-06","blended_usd_per_m":1.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_elo","score_raw":1112,"score_norm":0,"config":"bradley-terry mle, chatbot arena, dec 2023 leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://www.lmsys.org/blog/2023-12-07-leaderboard/","measured_date":"2023-12-07","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo-0613","publisher":"OpenRouter (listed 2024-01-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-3.5-turbo-16k","display_name":"GPT-3.5 Turbo 16k","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3.5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-3.5-turbo-16k","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":16385,"release_date":"2023-08-28","release_status":"released","knowledge_cutoff":"2021-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":4,"cache_read_usd_per_m":null,"context_length":16385,"source":"openrouter","source_model_id":"openai/gpt-3.5-turbo-16k","as_of":"2026-10-06","blended_usd_per_m":3.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo-16k","publisher":"OpenRouter (listed 2023-08-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-3.5-turbo-instruct","display_name":"GPT-3.5 Turbo Instruct","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3.5","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-3.5-turbo-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":4095,"release_date":"2023-09-28","release_status":"released","knowledge_cutoff":"2021-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.5,"completion_usd_per_m":2,"cache_read_usd_per_m":null,"context_length":4095,"source":"openrouter","source_model_id":"openai/gpt-3.5-turbo-instruct","as_of":"2026-10-06","blended_usd_per_m":1.625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-3.5-turbo-instruct","publisher":"OpenRouter (listed 2023-09-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4","display_name":"GPT-4","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":8191,"release_date":"2023-05-28","release_status":"released","knowledge_cutoff":"2021-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":10.77,"inputs":[{"benchmark":"arena_elo","score_raw":1287.4,"score_norm":3.46,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"54173 votes, arena entry gpt-4-0314, leaderboard text-overall-style_control, rank 274, run: gpt-4-0314, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":34.95,"score_norm":14.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.349494949494949 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":45.1,"score_norm":11.33,"trust":3,"source_url":"https://livebench.ai/table_2024_08_31.csv","config":"global average of the 6 category averages, leaderboard row gpt-4-0613, livebench 2024-08-31 release"}],"coverage":0.3572,"subset_bias":null},"pricing":{"prompt_usd_per_m":30,"completion_usd_per_m":60,"cache_read_usd_per_m":null,"context_length":8191,"source":"openrouter","source_model_id":"openai/gpt-4","as_of":"2026-10-06","blended_usd_per_m":37.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":13.1,"score_norm":13.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.78,"score_norm":6.78,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ai2_reasoning_challenge","score_raw":96.3,"score_norm":null,"config":"25-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1287.4,"score_norm":3.46,"config":"54173 votes, arena entry gpt-4-0314, leaderboard text-overall-style_control, rank 274, run: gpt-4-0314, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"drop","score_raw":80.9,"score_norm":null,"config":"3-shot, f1","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":34.95,"score_norm":14.01,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.349494949494949 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gsm_8k","score_raw":92,"score_norm":null,"config":"5-shot cot, training-set contamination flagged by the report","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":95.3,"score_norm":null,"config":"10-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":67,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":33.2,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.331972789115646 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":45.1,"score_norm":11.33,"config":"global average of the 6 category averages, leaderboard row gpt-4-0613, livebench 2024-08-31 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2024_08_31.csv","measured_date":"2024-08-31","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu","score_raw":86.4,"score_norm":null,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":87.5,"score_norm":null,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2303.08774v6","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4","publisher":"OpenRouter (listed 2023-05-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4-turbo","display_name":"GPT-4 Turbo","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4-turbo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-04-09","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-4-turbo-preview"],"intelligence":{"index":21.02,"inputs":[{"benchmark":"arena_elo","score_raw":1324.3,"score_norm":19.91,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"98114 votes, arena entry gpt-4-turbo-2024-04-09, leaderboard text-overall-style_control, rank 233, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"hle","score_raw":3.13,"score_norm":1.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-turbo","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0313 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":49.6,"score_norm":21.33,"trust":2,"source_url":"https://arxiv.org/html/2406.19314v2","config":"global average, gpt-4-turbo-2024-04-09, table 2"},{"benchmark":"mmlu_pro","score_raw":63.71,"score_norm":39.52,"trust":3,"source_url":"https://github.com/TIGER-AI-Lab/MMLU-Pro","config":"official mmlu-pro leaderboard"}],"coverage":0.4285,"subset_bias":null},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":30,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"openai/gpt-4-turbo","as_of":"2026-10-06","blended_usd_per_m":15,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":21.5,"score_norm":21.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4-turbo","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7,"score_norm":7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4-turbo","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.69,"score_norm":7.69,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1324.3,"score_norm":19.91,"config":"98114 votes, arena entry gpt-4-turbo-2024-04-09, leaderboard text-overall-style_control, rank 233, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1112.2,"score_norm":null,"config":"13391 votes, arena entry gpt-4-turbo-2024-04-09, leaderboard vision-overall-style_control, rank 115, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.13,"score_norm":1.66,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0313 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":49.6,"score_norm":21.33,"config":"global average, gpt-4-turbo-2024-04-09, table 2","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2406.19314v2","measured_date":"2025-04-18","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":29.1,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.291005291005291 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":63.71,"score_norm":39.52,"config":"official mmlu-pro leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://github.com/TIGER-AI-Lab/MMLU-Pro","measured_date":"2024-10-10","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":31.94,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.319444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-turbo","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4-turbo","publisher":"OpenRouter (listed 2024-04-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5-image","display_name":"GPT-5 Image","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-image","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-image","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-10-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":10,"cache_read_usd_per_m":1.25,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5-image","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_graphicdesign","score_raw":1175,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_image","score_raw":1194,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image","measured_date":null,"observed_at":"2026-09-10","admissible":false,"excluded":null},{"benchmark":"design_arena_logo","score_raw":1204,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image","publisher":"OpenRouter (listed 2025-10-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5-image-mini","display_name":"GPT-5 Image Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-image-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-image-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-10-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":2,"cache_read_usd_per_m":0.25,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5-image-mini","as_of":"2026-10-06","blended_usd_per_m":2.375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_graphicdesign","score_raw":1168,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image-mini","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_image","score_raw":1186,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_logo","score_raw":1210,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-image-mini","publisher":"OpenRouter (listed 2025-10-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.1-codex-max","display_name":"GPT-5.1-Codex-Max","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.1","size_class":"frontier-codex-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.1-codex-max","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-11-19","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":74.89,"inputs":[{"benchmark":"livebench","score_raw":75.6,"score_norm":79.11,"trust":3,"source_url":"https://livebench.ai/table_2025_12_23.csv","config":"global average of the 7 category averages, leaderboard row gpt-5.1-codex-max (default effort), livebench 2025-12-23 release"},{"benchmark":"swe_bench_verified","score_raw":77.9,"score_norm":75.44,"trust":1,"source_url":"https://venturebeat.com/technology/openai-debuts-gpt-5-1-codex-max-coding-model-and-it-already-completed-a-24","config":"extra-high reasoning effort, n=500, with compaction"},{"benchmark":"terminal_bench","score_raw":58.1,"score_norm":62.47,"trust":1,"source_url":"https://venturebeat.com/technology/openai-debuts-gpt-5-1-codex-max-coding-model-and-it-already-completed-a-24","config":"extra-high reasoning effort, n=89, v2.0, with compaction"}],"coverage":0.3254,"subset_bias":{"offset":-1.66,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.125,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.1-codex-max","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_lcr","score_raw":69,"score_norm":null,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86,"score_norm":85.92,"config":"probe","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5.1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86,"score_norm":85.92,"config":"artificial analysis harness (aa-gpqa diamond)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":25.7,"score_norm":34.85,"config":"artificial analysis harness (aa-hle)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":75.6,"score_norm":79.11,"config":"global average of the 7 category averages, leaderboard row gpt-5.1-codex-max (default effort), livebench 2025-12-23 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2025_12_23.csv","measured_date":"2025-12-23","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":72.5,"score_norm":null,"config":"artificial analysis harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.2,"score_norm":null,"config":"artificial analysis harness (aa-scicode)","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.9,"score_norm":75.44,"config":"extra-high reasoning effort, n=500, with compaction","trust":1,"source_type":"vendor_claim","source_url":"https://venturebeat.com/technology/openai-debuts-gpt-5-1-codex-max-coding-model-and-it-already-completed-a-24","measured_date":"2025-11-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_lancer_ic_swe","score_raw":79.9,"score_norm":null,"config":"extra-high reasoning effort, with compaction","trust":1,"source_type":"vendor_claim","source_url":"https://venturebeat.com/technology/openai-debuts-gpt-5-1-codex-max-coding-model-and-it-already-completed-a-24","measured_date":"2025-11-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":83,"score_norm":84,"config":"tau2","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-5-1-codex-max","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":58.1,"score_norm":62.47,"config":"extra-high reasoning effort, n=89, v2.0, with compaction","trust":1,"source_type":"vendor_claim","source_url":"https://venturebeat.com/technology/openai-debuts-gpt-5-1-codex-max-coding-model-and-it-already-completed-a-24","measured_date":"2025-11-19","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-max","publisher":"OpenRouter (listed 2025-12-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.1-codex-mini","display_name":"GPT-5.1-Codex-Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.1","size_class":"frontier-codex-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.1-codex-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-11-13","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":60.09,"inputs":[{"benchmark":"aime","score_raw":91.67,"score_norm":89.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.916666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":81.31,"score_norm":79.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.813131313131313 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":18.49,"score_norm":24.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.184893419833179 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":62.87,"score_norm":57.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.628654970760234 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":3.51,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":2,"cache_read_usd_per_m":0.03,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.1-codex-mini","as_of":"2026-10-06","blended_usd_per_m":0.6875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":20.4,"score_norm":20.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.33,"score_norm":31.33,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.65 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.67,"score_norm":89.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.916666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1243.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1243.7,"score_norm":null,"config":"1133 votes, arena entry gpt-5.1-codex-mini, code/overall, leaderboard webdev-overall-raw, rank 106, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1003,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1115,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1099,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1105,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1105,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":977,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1088,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1117,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.31,"score_norm":79.31,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.813131313131313 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.49,"score_norm":24.25,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.184893419833179 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.89,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.67891156462585 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":83.6,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.835978835978836 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.02,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.690173410404624 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.59,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.425925925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":62.87,"score_norm":57.16,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.628654970760234 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex-mini","publisher":"OpenRouter (listed 2025-11-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.2-instruct","display_name":"GPT-5.2 Chat","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.2","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.2-chat","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2025-12-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-5.2-chat"],"intelligence":{"index":52.47,"inputs":[{"benchmark":"aime","score_raw":51,"score_norm":38.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","config":"2025, artificial analysis independent evaluation"},{"benchmark":"arena_elo","score_raw":1476.2,"score_norm":87.61,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"34339 votes, arena entry gpt-5.2-chat-latest, leaderboard text-overall-style_control, rank 24, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":71.21,"score_norm":65.08,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":8.02,"score_norm":8.85,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":81.36,"score_norm":68.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","config":"artificial analysis independent evaluation"},{"benchmark":"tau_bench","score_raw":46.49,"score_norm":35.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","config":"artificial analysis independent evaluation, tau2/telecom"}],"coverage":0.5953,"subset_bias":{"offset":3.46,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.75,"completion_usd_per_m":14,"cache_read_usd_per_m":0.175,"context_length":128000,"source":"openrouter","source_model_id":"openai/gpt-5.2-chat","as_of":"2026-10-06","blended_usd_per_m":4.8125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_lcr","score_raw":45.67,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":51,"score_norm":38.75,"config":"2025, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.2,"score_norm":87.61,"config":"34339 votes, arena entry gpt-5.2-chat-latest, leaderboard text-overall-style_control, rank 24, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1278.2,"score_norm":null,"config":"15370 votes, arena entry gpt-5.2-chat-latest, leaderboard vision-overall-style_control, rank 23, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.57,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":86.8,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.21,"score_norm":65.08,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.02,"score_norm":8.85,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":47.41,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":66.88,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.36,"score_norm":68.93,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":65.84,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":46.49,"score_norm":35.32,"config":"artificial analysis independent evaluation, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-chat","publisher":"OpenRouter (listed 2025-12-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.3-codex","display_name":"GPT-5.3-Codex","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.3","size_class":"frontier-codex","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.3-codex","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2026-02-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["gpt-5-codex","gpt-5.1-codex","gpt-5.2-codex"],"superseded_by":null,"aliases":[],"intelligence":{"index":77.06,"inputs":[{"benchmark":"gpqa_diamond","score_raw":91.52,"score_norm":93.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.915151515151515 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":42.49,"score_norm":59.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.424930491195551 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":78,"score_norm":75.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.854, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":85.96,"score_norm":87.95,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.859649122807018 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":53.03,"score_norm":56.51,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.53030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5239,"subset_bias":{"offset":-0.35,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.75,"completion_usd_per_m":14,"cache_read_usd_per_m":0.175,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.3-codex","as_of":"2026-10-06","blended_usd_per_m":4.8125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":32.5,"score_norm":32.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45.51,"score_norm":45.51,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":78.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.783333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":10.87,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=xhigh, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1408.4,"score_norm":null,"config":"2526 votes, arena entry gpt-5.3-codex, code/overall, leaderboard webdev-overall-raw, rank 61, raw (no style control), run: codex-harness, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":16.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.168571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1023,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1084,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1162,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1172,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":981,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1124,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1051,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1128,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1025,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1169,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.52,"score_norm":93.69,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.915151515151515 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.49,"score_norm":59.54,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.424930491195551 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.37,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.753741496598639 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.5,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.784971098265896 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.24,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.532407407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78,"score_norm":75.56,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.854, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":85.96,"score_norm":87.95,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.859649122807018 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.03,"score_norm":56.51,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.53030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-3-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.3-codex","publisher":"OpenRouter (listed 2026-02-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.4-image-2","display_name":"GPT-5.4 Image 2","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.4","size_class":"frontier-image","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.4-image-2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":272000,"release_date":"2026-04-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":8,"completion_usd_per_m":15,"cache_read_usd_per_m":2,"context_length":272000,"source":"openrouter","source_model_id":"openai/gpt-5.4-image-2","as_of":"2026-10-06","blended_usd_per_m":9.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-image-2","publisher":"OpenRouter (listed 2026-04-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.4-mini","display_name":"GPT-5.4 Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.4","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.4-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2026-03-17","release_status":"released","knowledge_cutoff":"2025-08-31","status":"current","supersedes":["gpt-4-o-mini","gpt-4.1-mini","gpt-5-mini"],"superseded_by":null,"aliases":[],"intelligence":{"index":65.64,"inputs":[{"benchmark":"aime","score_raw":95.62,"score_norm":94.53,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.409, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":18.9,"score_norm":15.44,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.75, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-mini-reasoning, org=openai, run: xhigh, system_type=cot"},{"benchmark":"arena_elo","score_raw":1447.1,"score_norm":74.64,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":83.08,"score_norm":81.8,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=2.457, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":28.13,"score_norm":38.43,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gpt-5.4 mini (xhigh)"},{"benchmark":"livebench","score_raw":66.37,"score_norm":58.6,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run gpt-5.4-mini-xhigh"},{"benchmark":"mmlu_pro","score_raw":84.55,"score_norm":74.25,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.359, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":73,"score_norm":70,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.987, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":83.33,"score_norm":84.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":54.68,"score_norm":58.45,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=4.556, task=overall, updated=2026-08-19"}],"coverage":1,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.75,"completion_usd_per_m":4.5,"cache_read_usd_per_m":0.075,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.4-mini","as_of":"2026-10-06","blended_usd_per_m":1.6875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":17.9,"score_norm":17.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-mini","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":31.54,"score_norm":31.54,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":56.1,"score_norm":56.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.1,"score_norm":24.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-mini","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.94,"score_norm":40.94,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.62,"score_norm":94.53,"config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.409, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":15,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.15 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":28.17,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.281710914454277 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":63.67,"score_norm":null,"config":"cost_per_task_usd=0.51, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-4-mini-reasoning, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":18.9,"score_norm":15.44,"config":"cost_per_task_usd=0.75, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-mini-reasoning, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":18.9,"score_norm":15.44,"config":"run gpt-5-4-mini-xhigh, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1447.1,"score_norm":74.64,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1448.2,"score_norm":75.13,"config":"59502 votes, arena entry gpt-5.4-mini-high, leaderboard text-overall-style_control, rank 69, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1448.2,"score_norm":75.13,"config":"arena entry gpt-5.4-mini-high, rank 67, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1250.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1251.8,"score_norm":null,"config":"24061 votes, arena entry gpt-5.4-mini-high, leaderboard vision-overall-style_control, rank 39, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1397,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1397.1,"score_norm":null,"config":"11294 votes, arena entry gpt-5.4-mini-high, code/overall, leaderboard webdev-overall-raw, rank 68, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":10,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.1 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":34.62,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.3461653237839451 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":33.59,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.335875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.08,"score_norm":81.8,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=2.457, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.08,"score_norm":81.8,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.47,"score_norm":87.99,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.874747474747475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.47,"score_norm":87.99,"config":"artificial analysis eval, gpt-5.4 mini (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":60.75,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.607468519322623 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":28.13,"score_norm":38.43,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.281278962001854 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28.13,"score_norm":38.43,"config":"artificial analysis eval, gpt-5.4 mini (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.27,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.73265306122449 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":35.22,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, it-bench sre, reasoning=true, run: xhigh, source value 0.352165725047081 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":66.37,"score_norm":58.6,"config":"livebench-2026-06-25 release, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":66.37,"score_norm":58.6,"config":"global average, livebench-2026-06-25 release, run gpt-5.4-mini-xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":3.89,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.0388888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.55,"score_norm":74.25,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.359, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.554,"score_norm":74.26,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.29,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.732947976878613 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":49.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.498842592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73,"score_norm":70,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.987, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73,"score_norm":70,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":83.33,"score_norm":84.44,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":25.57,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.255670103092783 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":54.68,"score_norm":58.45,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=4.556, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":54.682,"score_norm":58.45,"config":"high reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.27,"score_norm":55.61,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.522727272727273 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.18,"score_norm":63.74,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.591760299625468 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.18,"score_norm":63.74,"config":"artificial analysis eval, gpt-5.4 mini (xhigh), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-mini","publisher":"OpenRouter (listed 2026-03-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.4-nano","display_name":"GPT-5.4 Nano","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.4","size_class":"frontier-nano","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.4-nano","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2026-03-17","release_status":"released","knowledge_cutoff":"2025-08-31","status":"current","supersedes":["gpt-4.1-nano","gpt-5-nano"],"superseded_by":null,"aliases":[],"intelligence":{"index":58.58,"inputs":[{"benchmark":"aime","score_raw":88.75,"score_norm":85.94,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.99, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":5.69,"score_norm":0.77,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.16, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-nano-reasoning, org=openai, run: xhigh, system_type=cot"},{"benchmark":"arena_elo","score_raw":1401.3,"score_norm":54.23,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.53,"score_norm":73.99,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.417, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":28.3,"score_norm":38.68,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","config":"artificial analysis run, gpt-5.4 nano (xhigh), no tools, page data value 0.282669138 rendered as 28.3% [raw: humanity's last exam]"},{"benchmark":"livebench","score_raw":69.58,"score_norm":65.73,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh"},{"benchmark":"mmlu_pro","score_raw":77.17,"score_norm":61.95,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.433, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":69.8,"score_norm":66.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.055, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":76.02,"score_norm":74.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.760233918128655 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":41.57,"score_norm":43.02,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.973, task=overall, updated=2026-08-19"}],"coverage":1,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":1.25,"cache_read_usd_per_m":0.02,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.4-nano","as_of":"2026-10-06","blended_usd_per_m":0.4625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":16,"score_norm":16,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-nano","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":29.67,"score_norm":29.67,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":56.1,"score_norm":56.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-nano","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.7,"score_norm":20.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-nano","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.71,"score_norm":39.71,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.72 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":88.75,"score_norm":85.94,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.99, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":26.67,"score_norm":8.34,"config":"aime 2025, layerlens stratix run submitted 2026-03-17, no tools [raw: aime 2025], reasoning effort not disclosed","trust":2,"source_type":"aggregator","source_url":"https://layerlens.ai/blog/gpt-5-4-benchmark-review","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":24.93,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.249262536873156 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":51.5,"score_norm":null,"config":"cost_per_task_usd=0.13, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-4-nano-reasoning, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":5.69,"score_norm":0.77,"config":"cost_per_task_usd=0.16, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-nano-reasoning, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1401.3,"score_norm":54.23,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1402.1,"score_norm":54.59,"config":"58551 votes, arena entry gpt-5.4-nano-high, leaderboard text-overall-style_control, rank 141, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1199.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1201.4,"score_norm":null,"config":"23917 votes, arena entry gpt-5.4-nano-high, leaderboard vision-overall-style_control, rank 74, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":9.25,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.0925306122448979 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":31.48,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.3148313936138466 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":30.24,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.30239499999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.53,"score_norm":73.99,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.417, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.72,"score_norm":79.89,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.817171717171717 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.7,"score_norm":79.86,"config":"artificial analysis intelligence index run, gpt-5.4 nano (xhigh), page data value 0.817171717 rendered as 81.7% [raw: gpqa diamond]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":52.24,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.522362136343899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":28.27,"score_norm":38.63,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.282669138090825 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28.3,"score_norm":38.68,"config":"artificial analysis run, gpt-5.4 nano (xhigh), no tools, page data value 0.282669138 rendered as 28.3% [raw: humanity's last exam]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.92,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.759183673469388 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":24.39,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, it-bench sre, reasoning=true, run: xhigh, source value 0.243879472693032 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":69.58,"score_norm":65.73,"config":"livebench-2026-06-25 release, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":5,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.05 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":77.17,"score_norm":61.95,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.433, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":35.61,"score_norm":0,"config":"layerlens stratix independent evaluation submitted 2026-03-17, reasoning effort not disclosed (likely low/default) — far below vendor-tier expectations, see notes [raw: mmlu pro]","trust":2,"source_type":"aggregator","source_url":"https://layerlens.ai/blog/gpt-5-4-benchmark-review","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":65.38,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.653757225433526 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.46875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.8,"score_norm":66.44,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.055, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69,"score_norm":65.56,"config":"comparable to gpt 5 codex (6 [raw: swe-bench verified], it scored 69.0%, stated in the page's model-update text (\"on swe-bench verified, vals ai independent run via official openai api on model snapshot gpt-5.4-nano-2026-03-17","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/openai_gpt-5.4-nano-2026-03-17","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76.02,"score_norm":74.69,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.760233918128655 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.5,"score_norm":96.67,"config":"telecom domain only (no published average), xhigh reasoning effort [raw: τ2-bench (telecom)]","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-4-mini-and-nano/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":27.42,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.274226804123711 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":41.57,"score_norm":43.02,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.973, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":42.42,"score_norm":44.02,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.424242424242424 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.67,"score_norm":65.49,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.606741573033708 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.7,"score_norm":65.53,"config":"artificial analysis harness, gpt-5.4 nano (xhigh), page data value 0.606741573 rendered as 60.7% [raw: terminal-bench v2.1], terminal-bench version 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-nano","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-nano","publisher":"OpenRouter (listed 2026-03-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.5","display_name":"GPT-5.5","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-04-23","release_status":"released","knowledge_cutoff":"2025-12-01","status":"current","supersedes":["gpt-4-o","gpt-4.1","gpt-5","gpt-5.1","gpt-5.2","gpt-5.4"],"superseded_by":null,"aliases":[],"intelligence":{"index":84.84,"inputs":[{"benchmark":"arc_agi_2","score_raw":85,"score_norm":88.89,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=1.87, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-5-2026-04-22-thinking, org=openai, run: xhigh, system_type=cot"},{"benchmark":"arena_elo","score_raw":1476.7,"score_norm":87.84,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.18,"score_norm":96.03,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.288, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":45.78,"score_norm":64.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.457831325301205 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":80.19,"score_norm":89.31,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh effort"},{"benchmark":"mmlu_pro","score_raw":88.14,"score_norm":80.23,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.318, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":82.6,"score_norm":80.67,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.695, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":93.86,"score_norm":98.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.93859649122807 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":76.4,"score_norm":84,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.649, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":30,"cache_read_usd_per_m":0.5,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.5","as_of":"2026-10-06","blended_usd_per_m":11.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":36.4,"score_norm":36.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":47.41,"score_norm":47.41,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":74.9,"score_norm":74.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.4,"score_norm":38.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":56.31,"score_norm":56.31,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.79 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":20.52,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=xhigh, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":50,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.5 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":37.68,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.376843657817109 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":95,"score_norm":null,"config":"cost_per_task_usd=0.73, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-5-2026-04-22-thinking, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":85,"score_norm":88.89,"config":"cost_per_task_usd=1.87, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-5-2026-04-22-thinking, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":85,"score_norm":88.89,"config":"verified; vendor-reported (gpt-5.5, high reasoning)","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.43,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=openai-gpt-5-5-2026-04-23-thinking, org=openai, run: high, system_type=cot, total_cost_usd=10000","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.7,"score_norm":87.84,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1481.9,"score_norm":90.15,"config":"59545 votes, arena entry gpt-5.5-high-text, leaderboard text-overall-style_control, rank 17, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1484.5,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1484.7,"score_norm":null,"config":"16816 votes, arena entry gpt-5.5-high-document, document/overall, leaderboard document-overall-raw, rank 7, raw (no style control), run: high, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1281,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1284.9,"score_norm":null,"config":"20727 votes, arena entry gpt-5.5-vision, leaderboard vision-overall-style_control, rank 14, run: gpt-5.5, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1457.5,"score_norm":null,"config":"13672 votes, arena entry gpt-5.5-webdev, code/overall, leaderboard webdev-overall-raw, rank 46, raw (no style control), run: codex-harness, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":42.1,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.42103132391668596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"bixbench","score_raw":80.5,"score_norm":null,"config":"bioinformatics; vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":84.4,"score_norm":null,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":27.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.271428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":81.8,"score_norm":null,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1212,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1179,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1084,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1150,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1177,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1266,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1088,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1173,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1067,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1139,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1157,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1152,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1235,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1110,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1263,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":46.64,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.4664279319606088 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"expert_swe","score_raw":73.1,"score_norm":null,"config":"openai internal eval; vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"frontiermath","score_raw":35.4,"score_norm":64.36,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"frontiermath_tiers_1_3","score_raw":51.7,"score_norm":null,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":84.9,"score_norm":null,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":49.51,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.49511 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.18,"score_norm":96.03,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.288, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.6,"score_norm":96.62,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.54,"score_norm":96.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.935353535353535 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":86.28,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.862787668258793 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":41.4,"score_norm":57.94,"config":"no tools; vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":45.78,"score_norm":64.38,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.457831325301205 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.85,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.758503401360544 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":45.81,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, it-bench sre, reasoning=true, run: xhigh, source value 0.458097928436911 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":80.19,"score_norm":89.31,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mcp_atlas","score_raw":75.3,"score_norm":null,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":17.78,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.177777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.14,"score_norm":80.23,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.318, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":81.16,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=medium, mmmu-pro, reasoning=true, run: medium, source value 0.811560693641619 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":74,"score_norm":70.13,"config":"512k-1m context bucket","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"osworld","score_raw":78.7,"score_norm":null,"config":"vendor-reported, verified","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":56.13,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.561342592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":58.6,"score_norm":89.33,"config":"vendor-reported, single pass","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":82.6,"score_norm":80.67,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.695, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98,"score_norm":100,"config":"no prompt tuning, tau2/telecom, vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":93.86,"score_norm":98.48,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.93859649122807 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":38.97,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.389690721649485 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":76.4,"score_norm":84,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.649, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"vendor-reported","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/everything-you-need-to-know-about-gpt-5-5","measured_date":"2026-04-23","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.61,"score_norm":65.42,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.27,"score_norm":93.26,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.842696629213483 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://help.openai.com/en/articles/9624314-model-release-notes","publisher":"OpenAI"},{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5","publisher":"OpenRouter (listed 2026-04-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.5-pro","display_name":"GPT-5.5 Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.5-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-04-24","release_status":"released","knowledge_cutoff":"2025-12-01","status":"current","supersedes":["gpt-5-pro","gpt-5.2-pro","gpt-5.4-pro"],"superseded_by":null,"aliases":[],"intelligence":{"index":83.45,"inputs":[{"benchmark":"arc_agi_2","score_raw":84.58,"score_norm":88.42,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=10.51, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-5-pro-2026-04-23-thinking, org=openai, run: high, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":93.92,"score_norm":97.07,"trust":3,"source_url":"https://epoch.ai/data/benchmarks.csv","config":"epoch ai independent run, gpt-5.5-pro-pre-release, xhigh reasoning effort"},{"benchmark":"hle","score_raw":43.1,"score_norm":60.44,"trust":2,"source_url":"https://benchmarklist.com/models/openai-gpt-5.5-pro/","config":"no tools (with-tools score 57.2%), openai gpt-5.5 system card"}],"coverage":0.3492,"subset_bias":{"offset":-1.68,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":30,"completion_usd_per_m":180,"cache_read_usd_per_m":null,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.5-pro","as_of":"2026-10-06","blended_usd_per_m":67.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arc_agi_1","score_raw":96.5,"score_norm":null,"config":"cost_per_task_usd=4.53, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-5-pro-2026-04-23-thinking, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":84.58,"score_norm":88.42,"config":"cost_per_task_usd=10.51, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-5-pro-2026-04-23-thinking, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":84.58,"score_norm":88.42,"config":"arc prize verified, gpt-5.5 pro (high), semi-private eval","trust":2,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/openai-gpt-5.5-pro/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":30.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.305714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-5-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.92,"score_norm":97.07,"config":"epoch ai independent run, gpt-5.5-pro-pre-release, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://epoch.ai/data/benchmarks.csv","measured_date":"2026-04-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.1,"score_norm":60.44,"config":"no tools (with-tools score 57.2%), openai gpt-5.5 system card","trust":2,"source_type":"aggregator","source_url":"https://benchmarklist.com/models/openai-gpt-5.5-pro/","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.5-pro","publisher":"OpenRouter (listed 2026-04-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-luna","display_name":"GPT-5.6 Luna","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-luna","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-luna","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":78.79,"inputs":[{"benchmark":"arc_agi_2","score_raw":59.54,"score_norm":60.6,"trust":3,"source_url":"https://arcprize.org/results/openai-gpt-5-6-luna","config":"cost_per_task_usd=0.67, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-luna, org=openai, run: max, system_type=cot"},{"benchmark":"arena_elo","score_raw":1453.1,"score_norm":77.32,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.736, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":39.48,"score_norm":55.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.394810009267841 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.04,"score_norm":76.73,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.35, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":93,"score_norm":92.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=1.142, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":71.3,"score_norm":68.4,"trust":2,"source_url":"https://openrouter.ai/openai/gpt-5.6-luna","config":"openrouter exacto, provider=openai"},{"benchmark":"terminal_bench","score_raw":79.03,"score_norm":87.09,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=0.991, task=overall, updated=2026-08-19"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.02,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-luna","as_of":"2026-10-06","blended_usd_per_m":0.45,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":42.1,"score_norm":42.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-luna","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":46.9,"score_norm":46.9,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":75,"score_norm":null,"config":"artificial analysis coding agent index, luna (max)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.4,"score_norm":71.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-luna","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.3,"score_norm":37.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-luna","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":52.32,"score_norm":52.32,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":78.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.783333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"agents_last_exam","score_raw":50.3,"score_norm":null,"config":"openai gpt-5.6 launch data (2026-07-09); no reasoning-effort noted for luna","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":35.84,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.358407079646018 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":88,"score_norm":null,"config":"cost_per_task_usd=0.32, eval=arc-agi-1 semi-private (v1_semi_private), model_group=openai-gpt-5-6-luna, org=openai, run: max, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-luna","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":59.54,"score_norm":60.6,"config":"cost_per_task_usd=0.67, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-luna, org=openai, run: max, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-luna","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.18,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=openai-gpt-5-6-luna, org=openai, run: max, system_type=cot, total_cost_usd=3189.42","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-luna","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1453.1,"score_norm":77.32,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1450.8,"score_norm":76.29,"config":"20693 votes, arena entry gpt-5.6-luna-xhigh-text, leaderboard text-overall-style_control, rank 63, run: xhigh, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1456.9,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1462.3,"score_norm":null,"config":"1888 votes, arena entry gpt-5.6-luna-xhigh-document, document/overall, leaderboard document-overall-raw, rank 19, raw (no style control), run: xhigh, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1259.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1252.6,"score_norm":null,"config":"5155 votes, arena entry gpt-5.6-luna-xhigh-vision, leaderboard vision-overall-style_control, rank 38, run: xhigh, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":42.24,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.4224225837450071 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":83.3,"score_norm":null,"config":"openai gpt-5.6 launch data","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.205714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":67.2,"score_norm":null,"config":"deepswe v1.1; openai gpt-5.6 launch data","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":53.91,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.53915 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.736, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.7,"score_norm":89.72,"config":"openrouter exacto, provider=openai","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.11,"score_norm":93.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.911111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.3,"score_norm":94.79,"config":"openai gpt-5.6 launch data (vendor-reported); cf. openrouter exacto 88.7","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":87.9,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.878998407873788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":55.7,"score_norm":null,"config":"openai gpt-5.6 launch data","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":39.48,"score_norm":55.12,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.394810009267841 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"it_bench_sre","score_raw":40.32,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, reasoning=true, run: max, source value 0.403201506591337 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":19.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.194444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.04,"score_norm":76.73,"config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.35, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.55,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, mmmu-pro, reasoning=true, run: max, source value 0.785549132947977 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":41.3,"score_norm":27.66,"config":"mrcr v2, 8 needle, 512k-1m context","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":45.6,"score_norm":null,"config":"osworld 2.0; openai gpt-5.6 launch data","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.55,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, scicode, source value 0.525462962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":62.7,"score_norm":96.17,"config":"openai gpt-5.6 launch data (2026-07-09)","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":93,"score_norm":92.22,"config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=1.142, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":71.3,"score_norm":68.4,"config":"openrouter exacto, provider=openai","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":31.13,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.311340206185567 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":79.03,"score_norm":87.09,"config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=0.991, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.9,"score_norm":89.29,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.808988764044944 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-luna","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.7,"score_norm":93.76,"config":"terminal-bench 2.1; openai gpt-5.6 launch data (2026-07-09)","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://developers.openai.com/api/docs/models","publisher":"OpenAI"},{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-luna","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-luna-pro","display_name":"GPT-5.6 Luna Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-luna-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-luna-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.02,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-luna-pro","as_of":"2026-10-06","blended_usd_per_m":0.45,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"config":"1 run, auto-routing, openrouter provider benchmark","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.4,"score_norm":92.11,"config":"1 run, openai first-party endpoint, openrouter provider benchmark","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":70,"score_norm":66.67,"config":"1 run, auto-routing, tau_bench_verified_airline","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":68.67,"score_norm":64.89,"config":"1 run, openai first-party endpoint, tau_bench_verified_airline","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/openai/gpt-5.6-luna-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-luna-pro","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-sol","display_name":"GPT-5.6 Sol","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-sol","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-sol","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-5.6-sol-reasoning"],"intelligence":{"index":90,"inputs":[{"benchmark":"arc_agi_2","score_raw":92.5,"score_norm":97.22,"trust":3,"source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","config":"cost_per_task_usd=1.44, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-sol, org=openai, run: max, system_type=cot"},{"benchmark":"arena_elo","score_raw":1483.9,"score_norm":91.05,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":95.2,"score_norm":98.87,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.074, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":49.49,"score_norm":69.84,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gpt-5.6 sol (max)"},{"benchmark":"livebench","score_raw":81.05,"score_norm":91.22,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run gpt-5.6-sol-max"},{"benchmark":"mmlu_pro","score_raw":89.1,"score_norm":81.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.308, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":96.2,"score_norm":95.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.856, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":85.09,"score_norm":86.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.850877192982456 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=1.35, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-sol","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":50.2,"score_norm":50.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-sol","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":57.78,"score_norm":57.78,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":80,"score_norm":null,"config":"codex harness, max reasoning effort, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":77.4,"score_norm":77.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-sol","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":47,"score_norm":47,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-sol","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":60.93,"score_norm":60.93,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":77.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.776666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":21.97,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":47.5,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":97.5,"score_norm":null,"config":"cost_per_task_usd=0.4, eval=arc-agi-1 semi-private (v1_semi_private), model_group=openai-gpt-5-6-sol, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":96.5,"score_norm":null,"config":"max reasoning effort; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":92.5,"score_norm":97.22,"config":"cost_per_task_usd=1.44, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-sol, org=openai, run: max, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":92.5,"score_norm":97.22,"config":"run openai-gpt-5-6-sol-max, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":92.5,"score_norm":97.22,"config":"max reasoning effort; arc prize verified, semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":7.78,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=openai-gpt-5-6-sol, org=openai, run: max, system_type=cot, total_cost_usd=25064.11","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":7.78,"score_norm":null,"config":"max reasoning effort; semi-private set, official harness","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1483.9,"score_norm":91.05,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1482.3,"score_norm":90.33,"config":"19541 votes, arena entry gpt-5.6-sol-xhigh-text, leaderboard text-overall-style_control, rank 16, run: xhigh, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1481.1,"score_norm":89.8,"config":"arena entry gpt-5.6-sol-xhigh, rank 19, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1482.7,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1478.7,"score_norm":null,"config":"1152 votes, arena entry gpt-5.6-sol-xhigh-document, document/overall, leaderboard document-overall-raw, rank 11, raw (no style control), run: xhigh, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1257.3,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1284.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1280.9,"score_norm":null,"config":"5164 votes, arena entry gpt-5.6-sol-xhigh-vision, leaderboard vision-overall-style_control, rank 19, run: xhigh, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":51.19,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.5118769513495001 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":90.4,"score_norm":null,"config":"sol standard (92.2 with sol ultra); secondary report of openai launch table","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":32.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.322857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":42.91,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.42912563413906296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":61.15,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.61148 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.2,"score_norm":98.87,"config":"edition=gpqa diamond v1, reasoning_effort=max, runner=platform, stderr=1.074, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":95.202,"score_norm":98.88,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.14,"score_norm":97.38,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.941414141414141 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.14,"score_norm":97.38,"config":"artificial analysis eval, gpt-5.6 sol (max)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.6,"score_norm":98.03,"config":"as published in openai launch table (secondary report)","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":87.18,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.871761470545665 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"healthbench","score_raw":57,"score_norm":null,"config":"length-adjusted score (unadjusted 55.6)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"healthbench_hard","score_raw":33.1,"score_norm":null,"config":"length-adjusted score (unadjusted 31.1)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":49.49,"score_norm":69.84,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.494902687673772 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":49.49,"score_norm":69.84,"config":"artificial analysis eval, gpt-5.6 sol (max)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":72.65,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, reasoning=true, run: max, source value 0.726530612244898 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":56.21,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, reasoning=true, run: max, source value 0.562146892655367 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":81.05,"score_norm":91.22,"config":"global average, livebench-2026-06-25 release, run gpt-5.6-sol-max","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":26.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.261111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.1,"score_norm":81.83,"config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.308, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.1,"score_norm":81.83,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":83.41,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, mmmu-pro, reasoning=true, run: max, source value 0.834104046242775 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":62.6,"score_norm":null,"config":"2.0, secondary report of openai launch table","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":56.94,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.569444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":64.6,"score_norm":99.33,"config":"secondary report of openai launch table","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":96.2,"score_norm":95.78,"config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.856, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":96.2,"score_norm":95.78,"config":"max reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":85.09,"score_norm":86.79,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.850877192982456 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":44.33,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.443298969072165 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=1.35, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.768,"score_norm":95.02,"config":"max reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":65.91,"score_norm":71.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.659090909090909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":89.51,"score_norm":99.42,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.895131086142322 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-sol-xhigh","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.01,"score_norm":97.66,"config":"artificial analysis eval, gpt-5.6 sol (max), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.8,"score_norm":98.59,"config":"sol standard (91.9 with sol ultra); secondary report of openai launch table","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://developers.openai.com/api/docs/models","publisher":"OpenAI"},{"url":"https://openai.com/index/gpt-5-6/","publisher":"OpenAI"},{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-sol","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-sol-pro","display_name":"GPT-5.6 Sol Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-sol-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-sol-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-sol-pro","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-sol-pro","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-terra","display_name":"GPT-5.6 Terra","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-terra","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-terra","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":84.26,"inputs":[{"benchmark":"arc_agi_2","score_raw":83.9,"score_norm":87.67,"trust":3,"source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","config":"cost_per_task_usd=1.09, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-terra, org=openai, run: max, system_type=cot"},{"benchmark":"arena_elo","score_raw":1465.5,"score_norm":82.84,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":90.91,"score_norm":92.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.906, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":42.91,"score_norm":60.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.429101019462465 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":77.9,"score_norm":84.22,"trust":3,"source_url":"https://livebench.ai/","config":"agentic coding 54.9, coding 78.2, data analysis 79. [raw: livebench (livebench-2026-06-25) overall], global/overall average, livebench-2026-06-25 release, mathematics 94.9, row \"gpt-5.6 terra max effort\" (category splits on same row: reasoning 90.6"},{"benchmark":"mmlu_pro","score_raw":86.66,"score_norm":77.77,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.333, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":95.4,"score_norm":94.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.938, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":86.26,"score_norm":88.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.862573099415205 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=2.247, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":12,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-terra","as_of":"2026-10-06","blended_usd_per_m":4.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":43.2,"score_norm":43.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-terra","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":50.2,"score_norm":50.2,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":77,"score_norm":null,"config":"gpt-5.6 terra (max reasoning), v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/gpt-5-6-has-landed","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.7,"score_norm":76.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-terra","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":42.1,"score_norm":42.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-terra","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":56.58,"score_norm":56.58,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.796666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.05,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"agents_last_exam","score_raw":50.4,"score_norm":null,"config":"gpt-5.6 terra","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":38.94,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.389380530973451 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":96.5,"score_norm":null,"config":"cost_per_task_usd=0.55, eval=arc-agi-1 semi-private (v1_semi_private), model_group=openai-gpt-5-6-terra, org=openai, run: max, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":94,"score_norm":null,"config":"xhigh reasoning; public eval (400 tasks); arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":96.5,"score_norm":null,"config":"public eval (400 tasks); max reasoning; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":83.9,"score_norm":87.67,"config":"cost_per_task_usd=1.09, eval=arc-agi-2 semi-private (v2_semi_private), model_group=openai-gpt-5-6-terra, org=openai, run: max, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":67.1,"score_norm":69,"config":"public eval (120 tasks); high reasoning; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":83.9,"score_norm":87.67,"config":"public eval (120 tasks); max reasoning; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":74.2,"score_norm":76.89,"config":"xhigh reasoning; public eval (120 tasks)","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.8,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=openai-gpt-5-6-terra, org=openai, run: max, system_type=cot, total_cost_usd=7918.31","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.65,"score_norm":null,"config":"xhigh reasoning; semi-private; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.8,"score_norm":null,"config":"semi-private; max reasoning; arc prize verified","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-5-6-terra","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1465.5,"score_norm":82.84,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1465.4,"score_norm":82.8,"config":"20216 votes, arena entry gpt-5.6-terra-xhigh-text, leaderboard text-overall-style_control, rank 44, run: xhigh, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1471.6,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1479.2,"score_norm":null,"config":"1969 votes, arena entry gpt-5.6-terra-xhigh-document, document/overall, leaderboard document-overall-raw, rank 10, raw (no style control), run: xhigh, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1268.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1265.5,"score_norm":null,"config":"5094 votes, arena entry gpt-5.6-terra-xhigh-vision, leaderboard vision-overall-style_control, rank 30, run: xhigh, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":45.61,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.4561448924929106 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":87.5,"score_norm":null,"config":"gpt-5.6 terra, browsing tools on","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":30,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.3 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":69.6,"score_norm":null,"config":"1.1, gpt-5.6 terra","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":53.82,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.53815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1593,"score_norm":null,"config":"elo; openai published gpt-5.6 eval table","trust":0,"source_type":"community","source_url":"https://www.marktechpost.com/2026/07/09/openai-releases-gpt-5-6-a-three-tier-model-family-with-programmatic-tool-calling/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"genebench_pro","score_raw":23.3,"score_norm":null,"config":"gpt-5.6 terra","trust":0,"source_type":"community","source_url":"https://anycap.ai/page/en-US/ai/gpt-5-6-benchmarks-sol-terra-luna","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.91,"score_norm":92.83,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.906, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.53,"score_norm":95.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.925252525252525 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.5,"score_norm":95.07,"config":"\"gpt-5.6 terra (max)\" = max reasoning effort, diamond, independently run by artificial analysis (rank 12 of 30 shown) [raw: gpqa diamond], no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.9,"score_norm":95.63,"config":"gpt-5.6 terra, default reasoning; as published in openai gpt-5.6 launch comparison table","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":85.18,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.851787523520046 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"healthbench","score_raw":57,"score_norm":null,"config":"length-adjusted (unadjusted 58.7, mean 2285 chars)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"healthbench_consensus","score_raw":95.1,"score_norm":null,"config":"length-adjusted (unadjusted 95.2, mean 2247 chars)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"healthbench_hard","score_raw":32.7,"score_norm":null,"config":"length-adjusted (unadjusted 34.3, mean 2199 chars)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":57.7,"score_norm":null,"config":"length-adjusted (unadjusted 62.4, mean 3618 chars)","trust":1,"source_type":"vendor_claim","source_url":"https://deploymentsafety.openai.com/gpt-5-6-preview/gpt-5-6-preview.pdf","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":42.91,"score_norm":60.16,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.429101019462465 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.9,"score_norm":60.15,"config":"\"gpt-5.6 terra (max)\" = max reasoning effort, independently run by artificial analysis (12th of 31 shown, tied with grok 4.6 (high) at 42.9%). text-only vs full split not stated on the page. [raw: humanity's last exam]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.22,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, reasoning=true, run: max, source value 0.712244897959184 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":51.04,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, reasoning=true, run: max, source value 0.510357815442561 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":77.9,"score_norm":84.22,"config":"agentic coding 54.9, coding 78.2, data analysis 79. [raw: livebench (livebench-2026-06-25) overall], global/overall average, livebench-2026-06-25 release, mathematics 94.9, row \"gpt-5.6 terra max effort\" (category splits on same row: reasoning 90.6","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":31.67,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.316666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.66,"score_norm":77.77,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.333, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.69,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, mmmu-pro, reasoning=true, run: max, source value 0.806936416184971 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":50.2,"score_norm":null,"config":"2.0, gpt-5.6 terra","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.94,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, scicode, source value 0.539351851851852 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":63.4,"score_norm":97.33,"config":"gpt-5.6 terra, default reasoning","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95.4,"score_norm":94.89,"config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=0.938, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":86.26,"score_norm":88.35,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.862573099415205 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":40.21,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.402061855670103 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=2.247, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":62.88,"score_norm":68.09,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.628787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra-xhigh","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.01,"score_norm":97.66,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.880149812734082 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-6-terra","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88,"score_norm":97.65,"config":"\"gpt-5.6 terra (max)\" = max reasoning effort, independently run by artificial analysis (tied 4th-6th at 88.0%) [raw: terminal-bench v2.1], terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.4,"score_norm":96.94,"config":"gpt-5.6 terra, default reasoning","trust":0,"source_type":"community","source_url":"https://www.cometapi.com/gpt-5-6-models-explained-benchmarks-access/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"toolathlon","score_raw":53.1,"score_norm":null,"config":"openai published gpt-5.6 eval table","trust":0,"source_type":"community","source_url":"https://www.marktechpost.com/2026/07/09/openai-releases-gpt-5-6-a-three-tier-model-family-with-programmatic-tool-calling/","measured_date":"2026-07-09","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://developers.openai.com/api/docs/models","publisher":"OpenAI"},{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-terra","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.6-terra-pro","display_name":"GPT-5.6 Terra Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.6","size_class":"frontier-terra-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.6-terra-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-07-09","release_status":"released","knowledge_cutoff":"2026-02-16","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":12,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.6-terra-pro","as_of":"2026-10-06","blended_usd_per_m":4.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.6-terra-pro","publisher":"OpenRouter (listed 2026-07-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6-astra","display_name":"GPT-6 Astra","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-astra","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-astra","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":"Proprietary","is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-03","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":93.7,"inputs":[{"benchmark":"arc_agi_2","score_raw":95,"score_norm":100,"trust":1,"source_url":"https://openai.com/index/gpt-6-astra/","config":null},{"benchmark":"gpqa_diamond","score_raw":96,"score_norm":100,"trust":1,"source_url":"https://openai.com/index/gpt-6-astra/","config":"maximum at any effort, research environment or api"},{"benchmark":"livebench","score_raw":82.16,"score_norm":93.69,"trust":1,"source_url":"https://www.datalearner.com/en/ai-models/pretrained-models/gpt-6-astra/analysis","config":"max"},{"benchmark":"terminal_bench","score_raw":57.9,"score_norm":62.24,"trust":1,"source_url":"https://openai.com/index/gpt-6-astra/","config":"4.0, agentic terminal tasks, maximum at any effort"}],"coverage":0.4047,"subset_bias":{"offset":4.33,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":50,"cache_read_usd_per_m":1,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-astra","as_of":"2026-10-06","blended_usd_per_m":20,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":51,"score_norm":51,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6-astra","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":67,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.9,"score_norm":76.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6-astra","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":52.7,"score_norm":52.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6-astra","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":61.2,"score_norm":61.2,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"agents_last_exam","score_raw":59.3,"score_norm":null,"config":"maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":98.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":98.5,"score_norm":null,"config":"maximum at any effort","trust":0,"source_type":"community","source_url":"https://the-decoder.com/gpt-6-astra-is-the-first-model-making-openai-willing-to-declare-the-agi-era/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":95,"score_norm":100,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-10-04","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":95,"score_norm":100,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-6-astra","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":95,"score_norm":100,"config":"maximum at any effort","trust":0,"source_type":"community","source_url":"https://the-decoder.com/gpt-6-astra-is-the-first-model-making-openai-willing-to-declare-the-agi-era/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":99.9,"score_norm":null,"config":"high reasoning, openai responses api / provider adapter harness","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":99.9,"score_norm":null,"config":"openai announcement table","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":62.7,"score_norm":null,"config":"arc prize standard harness, max reasoning, semi-private test set","trust":0,"source_type":"community","source_url":"https://www.deeplearning.ai/the-batch/gpt-6-astra-is-a-star","measured_date":"2026-09-11","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_3_standard_harness","score_raw":62.71,"score_norm":null,"config":"max reasoning, semi-private, standard harness","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/openai-gpt-6-astra","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":41.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":41.4,"score_norm":null,"config":"maximum at any effort","trust":0,"source_type":"community","source_url":"https://the-decoder.com/gpt-6-astra-is-the-first-model-making-openai-willing-to-declare-the-agi-era/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"benchcad","score_raw":95.9,"score_norm":null,"config":"geometric-overlap score, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":91.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":91.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":91.5,"score_norm":null,"config":"maximum at any effort","trust":0,"source_type":"community","source_url":"https://the-decoder.com/gpt-6-astra-is-the-first-model-making-openai-willing-to-declare-the-agi-era/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":74.1,"score_norm":null,"config":"1.1, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":74.1,"score_norm":null,"config":"v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"exploitbench","score_raw":100,"score_norm":null,"config":"maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontiercode_1_1_extended_score","score_raw":64.5,"score_norm":null,"config":"developer message similar to codex, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontiercode_1_1_main_score","score_raw":53.3,"score_norm":null,"config":"developer message similar to codex, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"frontiermath","score_raw":97.6,"score_norm":100,"config":"tier 4 (v2)","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"frontiermath_tier_4","score_raw":97.6,"score_norm":null,"config":"2, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":96,"score_norm":100,"config":"maximum at any effort, research environment or api","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":96,"score_norm":100,"config":"openai launch table","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":57.2,"score_norm":81.18,"config":"maximum at any effort, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":57.2,"score_norm":81.18,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":54.7,"score_norm":77.5,"config":"artificial analysis harness (aa-hle), no tools","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-6-astra","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle_diamond","score_raw":82.9,"score_norm":null,"config":"reasoning high, with tools (web+code)","trust":3,"source_type":"paper","source_url":"https://lastexam.ai/blog/hle-diamond","measured_date":"2026-09-22","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"hle_diamond","score_raw":59.9,"score_norm":null,"config":"reasoning high, without tools","trust":3,"source_type":"paper","source_url":"https://lastexam.ai/blog/hle-diamond","measured_date":"2026-09-22","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":82.16,"score_norm":93.69,"config":"max","trust":1,"source_type":"aggregator","source_url":"https://www.datalearner.com/en/ai-models/pretrained-models/gpt-6-astra/analysis","measured_date":null,"observed_at":"2026-10-04","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":86.9,"score_norm":null,"config":"artificial analysis harness","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/gpt-6-astra","measured_date":"2026-09-04","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":72.6,"score_norm":null,"config":"osworld 2.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld_2_0_v2026_08_08_offline_set_partial_score","score_raw":72.6,"score_norm":null,"config":"latency simulation ~40 min/task, offline set, partial score","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"screenspot_pro","score_raw":92.7,"score_norm":null,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"screenspot_pro","score_raw":92.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"sre_bench","score_raw":99.2,"score_norm":null,"config":"within four attempts","trust":1,"source_type":"vendor_claim","source_url":"https://archive.fo/2026.09.03-222053/https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"sre_bench","score_raw":88,"score_norm":null,"config":"single attempt / pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://archive.fo/2026.09.03-222053/https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":57.9,"score_norm":62.24,"config":"4.0, agentic terminal tasks, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":57.7,"score_norm":62,"config":"v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":64.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.vellum.ai/blog/gpt-6-astra-benchmarks-explained","measured_date":"2026-09-03","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science_0_1","score_raw":64.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science_0_1","score_raw":64.6,"score_norm":null,"config":"code and terminal tools, maximum at any effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/gpt-6-astra/","measured_date":"2026-09-03","observed_at":"2026-09-28","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openai.com/index/gpt-6-astra/","publisher":"OpenAI"},{"url":"https://openai.com/products/release-notes/","publisher":"OpenAI"},{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-astra","publisher":"OpenRouter (listed 2026-09-04)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-04T06:04:13.466Z"},{"id":"gpt-6-astra-pro","display_name":"GPT-6 Astra Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-astra-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-astra-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":50,"cache_read_usd_per_m":1,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-astra-pro","as_of":"2026-10-06","blended_usd_per_m":20,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"kernelbench_cuda","score_raw":27.6,"score_norm":null,"config":"3/4 problems, cli 0.140.0), openai gpt-6-astra-pro through openrouter (codex harness, peak fraction of board roofline (hard / most cuda), rtx pro 6000, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-06","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"kernelbench_cuda_canonical_board_rtx_pro_6000","score_raw":27.6,"score_norm":null,"config":"3/4 problems pass, codex harness cli 0.140.0, openrouter openai/gpt-6-astra-pro, peak fraction of board roofline (hard / most cuda), unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-07","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"kernelbench_cuda_glm_5_2_fused_moe_rtx_pro_6000","score_raw":9.45,"score_norm":null,"config":"codex cli 0.140.0, openrouter openai/gpt-6-astra-pro, publishable peak fraction of board roofline, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"kernelbench_cuda_grid_mingru_sps_rtx_pro_6000","score_raw":68.37,"score_norm":null,"config":"codex cli 0.140.0 harness, openrouter openai/gpt-6-astra-pro, publishable peak fraction of board roofline, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-07","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"kernelbench_cuda_megaqwen_decode_rtx_pro_6000","score_raw":4.9,"score_norm":null,"config":"codex cli 0.140.0 harness, openrouter openai/gpt-6-astra-pro, publishable peak fraction of board roofline, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-07","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"kernelbench_grid_mingru_sps","score_raw":68.37,"score_norm":null,"config":"audit-time isolated regrade 2026-09-07, openrouter (codex cli 0.140.0 harness), rtx pro 6000, trusted post-session grade, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-07","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"kernelbench_mega","score_raw":24.8,"score_norm":null,"config":"1/1, cli 0.140.0, codex harness, post-session trusted grade 24.8188x geomean, rtx pro 6000, speedup vs torch baseline, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-06","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"kernelbench_mega_kimi_linear_decode_rtx_pro_6000","score_raw":24.8,"score_norm":null,"config":"canonical board, codex harness cli 0.140.0, openrouter openai/gpt-6-astra-pro, publishable audited-clean speedup vs torch baseline, unlimited budget, xhigh effort","trust":0,"source_type":"community","source_url":"https://kernelbench.com/models/gpt-6-astra-pro","measured_date":"2026-09-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"official_a","score_raw":97.5,"score_norm":null,"config":"157 tests, openai/gpt-6-astra-pro:batch via openrouter batch api, reasoning.effort=low, smf-bench official a strict_v01, temperature omitted, thinking-off analogue","trust":0,"source_type":"community","source_url":"https://x.com/MichaelGannotti/article/2097740315960774735","measured_date":"2026-09-09","observed_at":"2026-10-04","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-astra-pro","publisher":"OpenRouter (listed 2026-09-04)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-04T06:04:04.747Z"},{"id":"gpt-6-luna","display_name":"GPT-6 Luna","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-luna","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-luna","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.5,"cache_read_usd_per_m":0.01,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-luna","as_of":"2026-10-06","blended_usd_per_m":0.2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":38.1,"score_norm":38.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6-luna","measured_date":null,"observed_at":"2026-10-02","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-luna","publisher":"OpenRouter (listed 2026-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6-luna-pro","display_name":"GPT-6 Luna Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-luna-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-luna-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.5,"cache_read_usd_per_m":0.01,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-luna-pro","as_of":"2026-10-06","blended_usd_per_m":0.2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-luna-pro","publisher":"OpenRouter (listed 2026-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6-sol","display_name":"GPT-6 Sol","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-sol","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-sol","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-sol","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":47.6,"score_norm":47.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6-sol","measured_date":null,"observed_at":"2026-10-02","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-sol","publisher":"OpenRouter (listed 2026-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6-sol-pro","display_name":"GPT-6 Sol Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6","size_class":"frontier-sol-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6-sol-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.2,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6-sol-pro","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6-sol-pro","publisher":"OpenRouter (listed 2026-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6.1-sol","display_name":"GPT-6.1 Sol","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6.1","size_class":"frontier-sol","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6.1-sol","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-29","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.1,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6.1-sol","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":51.8,"score_norm":51.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-6.1-sol","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6.1-sol","publisher":"OpenRouter (listed 2026-09-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-6.1-sol-pro","display_name":"GPT-6.1 Sol Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"6.1","size_class":"frontier-sol-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-6.1-sol-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-09-29","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":10,"cache_read_usd_per_m":0.1,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-6.1-sol-pro","as_of":"2026-10-06","blended_usd_per_m":4,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-6.1-sol-pro","publisher":"OpenRouter (listed 2026-09-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o3","display_name":"OpenAI o3","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3","size_class":"frontier-o","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-04-16","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":["gpt-o1"],"superseded_by":null,"aliases":["o3","o3-2025-04-16"],"intelligence":{"index":59.11,"inputs":[{"benchmark":"aime","score_raw":85.28,"score_norm":81.6,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.389, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":6.53,"score_norm":1.7,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.8339, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1431.6,"score_norm":67.73,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":84.09,"score_norm":83.23,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.861, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":20.05,"score_norm":26.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.2005 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":85.59,"score_norm":75.98,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.345, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":62.3,"score_norm":58.11,"trust":3,"source_url":"https://epoch.ai/models/o3","config":"epoch ai evaluation, o3 at medium reasoning effort"},{"benchmark":"tau_bench","score_raw":80.7,"score_norm":80.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.807017543859649 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":37.12,"score_norm":37.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.371212121212121 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":8,"cache_read_usd_per_m":0.5,"context_length":200000,"source":"openrouter","source_model_id":"openai/o3","as_of":"2026-10-06","blended_usd_per_m":3.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":20.2,"score_norm":20.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.1,"score_norm":31.1,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":85.28,"score_norm":81.6,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.389, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":88.33,"score_norm":85.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.883333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":88.9,"score_norm":86.13,"config":"2025, o3 (no tools)","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/2025id_/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":"2025-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":60.83,"score_norm":null,"config":"cost_per_task_usd=0.5002, eval=arc-agi-1 semi-private (v1_semi_private), model_group=o3, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":6.53,"score_norm":1.7,"config":"cost_per_task_usd=0.8339, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":2.9,"score_norm":0,"config":"arc-agi-2 semi-private eval, o3-medium","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/blog/analyzing-o3-with-arc-agi","measured_date":"2025-04-22","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1431.6,"score_norm":67.73,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1430.7,"score_norm":67.33,"config":"59888 votes, arena entry o3-2025-04-16, leaderboard text-overall-style_control, rank 95, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1215.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1217,"score_norm":null,"config":"46249 votes, arena entry o3-2025-04-16, leaderboard vision-overall-style_control, rank 63, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1026,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1200,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1047,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1023,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1043,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.09,"score_norm":83.23,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.861, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.73,"score_norm":81.31,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.827272727272727 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.3,"score_norm":82.11,"config":"o3 (no tools)","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/2025id_/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":"2025-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":20.05,"score_norm":26.54,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.2005 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":20.32,"score_norm":26.94,"config":"o3 (no tools)","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/2025id_/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":"2025-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.43,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.714285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":80.85,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.808465608465608 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.59,"score_norm":75.98,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.345, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.6,"score_norm":76,"config":"o3-2025-04-16, vals.ai independent evaluation","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":70.06,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.700578034682081 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.97,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.409722222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":62.3,"score_norm":58.11,"config":"epoch ai evaluation, o3 at medium reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://epoch.ai/models/o3","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.1,"score_norm":65.67,"config":"256k max context, internal tool scaffold, n=477 verified subset, pass@1 averaged over 4 tries","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/2025id_/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":"2025-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":80.7,"score_norm":80.93,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.807017543859649 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":70.4,"score_norm":67.2,"config":"averaged across 5 runs, no custom tools or prompting, o3-high, tau-bench retail","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/2025id_/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":"2025-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.12,"score_norm":37.79,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.371212121212121 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.1,"score_norm":37.76,"config":"artificial analysis intelligence index, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o3","publisher":"OpenRouter (listed 2025-04-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o3-mini-high","display_name":"OpenAI o3 Mini High","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3","size_class":"frontier-o-mini-high","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o3-mini-high","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-02-12","release_status":"released","knowledge_cutoff":"2023-10-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["o3-mini-high","o3-mini-high-2025-01-31"],"intelligence":{"index":45.94,"inputs":[{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"trust":3,"source_url":"https://matharena.ai/?comp=aime--aime_2025","config":"accuracy column 86.67%, aime 2025, leaderboard row \"o3-mini (high)\", no tools (matharena standard harness) [raw: aime 2025]"},{"benchmark":"arc_agi_2","score_raw":3,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"2025-01-31, arc-agi-2 column = 3.0% (arc-agi-1 column = 34.5% on the same row), cost/task $0.547 [raw: arc-agi-2], openai, row \"o3-mini (high)\", system type cot"},{"benchmark":"arena_elo","score_raw":1363.6,"score_norm":37.43,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.27,"score_norm":73.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini-high","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.772727272727273 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":12.04,"score_norm":14.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini-high","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.1204 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":75.88,"score_norm":79.73,"trust":2,"source_url":"https://epoch.ai/data/external_benchmarks/live_bench.csv","config":"epoch ai mirror of the livebench.ai leaderboard (o3-mini rows have s [raw: livebench], global average, livebench version livebench-2024-11-25, row id \"o3-mini-2025-01-31-high\" / model version \"o3-mini-2025-01-31_high\""},{"benchmark":"tau_bench","score_raw":31.29,"score_norm":15.05,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini-high","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.312865497076023 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini-high","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.3,"peers":8,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.1,"completion_usd_per_m":4.4,"cache_read_usd_per_m":0.55,"context_length":200000,"source":"openrouter","source_model_id":"openai/o3-mini-high","as_of":"2026-10-06","blended_usd_per_m":1.9250000000000003,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.9,"score_norm":0.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3-mini-high","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.72,"score_norm":1.72,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":16.3,"score_norm":16.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3-mini-high","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11,"score_norm":11,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3-mini-high","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.7,"score_norm":15.7,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":42,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.42 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"config":"accuracy column 86.67%, aime 2025, leaderboard row \"o3-mini (high)\", no tools (matharena standard harness) [raw: aime 2025]","trust":3,"source_type":"leaderboard","source_url":"https://matharena.ai/?comp=aime--aime_2025","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":3,"score_norm":0,"config":"2025-01-31, arc-agi-2 column = 3.0% (arc-agi-1 column = 34.5% on the same row), cost/task $0.547 [raw: arc-agi-2], openai, row \"o3-mini (high)\", system type cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1363.6,"score_norm":37.43,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1363.5,"score_norm":37.38,"config":"18589 votes, arena entry o3-mini-high, leaderboard text-overall-style_control, rank 184, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.27,"score_norm":73.62,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.772727272727273 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77,"score_norm":73.24,"config":"0.77 +/- 0.03), 198-question diamond subse [raw: gpqa diamond], epoch ai internal run, high reasoning effort, model version \"o3-mini-2025-01-31_high\", raw best score 0.7702020202020202 (choice scorer, task \"gpqa diamond\"","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/data/benchmarks.csv","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.04,"score_norm":14.76,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.1204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":13.4,"score_norm":16.76,"config":"calibration error 80% [raw: humanity's last exam (hle)], evaluated on text-only subset), no tools, row \"o3-mini (high)*\", text-only subset (asterisk = not multi-modal","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2501.14249v11","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.14,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.671428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":75.88,"score_norm":79.73,"config":"epoch ai mirror of the livebench.ai leaderboard (o3-mini rows have s [raw: livebench], global average, livebench version livebench-2024-11-25, row id \"o3-mini-2025-01-31-high\" / model version \"o3-mini-2025-01-31_high\"","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/data/external_benchmarks/live_bench.csv","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":73.44,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.734391534391534 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.79,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.397856419092825 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":31.29,"score_norm":15.05,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.312865497076023 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.15,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0515463917525773 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.49,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o3-mini-high","publisher":"OpenRouter (listed 2025-02-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o3-pro","display_name":"OpenAI o3 Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3","size_class":"frontier-o-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o3-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-06-10","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":["gpt-o1-pro"],"superseded_by":null,"aliases":["o3-pro","o3-pro-2025-06-10"],"intelligence":{"index":63.82,"inputs":[{"benchmark":"aime","score_raw":90,"score_norm":87.5,"trust":1,"source_url":"https://help.openai.com/en/articles/9624314-model-release-notes","config":"4/4 reliability, aime 2024, default medium chatgpt thinking time"},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=7.5516, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3-pro, org=openai, run: high, system_type=cot + synthesis"},{"benchmark":"gpqa_diamond","score_raw":84.55,"score_norm":83.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-pro","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.845454545454545 is a 0-1 fraction, x100 at harvest"}],"coverage":0.3174,"subset_bias":{"offset":2.2,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":20,"completion_usd_per_m":80,"cache_read_usd_per_m":null,"context_length":200000,"source":"openrouter","source_model_id":"openai/o3-pro","as_of":"2026-10-06","blended_usd_per_m":35,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":21.9,"score_norm":21.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3-pro","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.29,"score_norm":33.29,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":90,"score_norm":87.5,"config":"4/4 reliability, aime 2024, default medium chatgpt thinking time","trust":1,"source_type":"vendor_claim","source_url":"https://help.openai.com/en/articles/9624314-model-release-notes","measured_date":"2025-06-10","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":94,"score_norm":92.5,"config":"2024, author compilation of publicly reported scores","trust":0,"source_type":"community","source_url":"https://patmcguinness.substack.com/p/o3-pro-the-ai-that-thinks-too-much","measured_date":"2025-06-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":94,"score_norm":92.5,"config":"aime 2024, no tools, openai-reported launch comparison chart","trust":0,"source_type":"community","source_url":"https://patmcguinness.substack.com/p/o3-pro-the-ai-that-thinks-too-much","measured_date":"2025-06-10","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":59.33,"score_norm":null,"config":"cost_per_task_usd=4.16, eval=arc-agi-1 semi-private (v1_semi_private), model_group=o3-pro, org=openai, run: high, system_type=cot + synthesis","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"config":"cost_per_task_usd=7.5516, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3-pro, org=openai, run: high, system_type=cot + synthesis","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.9,"score_norm":0,"config":"arc prize official leaderboard row dated 2025-06-10, cost/task $7.55. sibling effort rows on the same page: medium 1.9%, low 2.1%. [raw: arc-agi-2], o3-pro (high) — high reasoning effort, system type \"cot + synthesis\"","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"codeforces_elo","score_raw":2748,"score_norm":100,"config":"author compilation of publicly reported scores","trust":0,"source_type":"community","source_url":"https://patmcguinness.substack.com/p/o3-pro-the-ai-that-thinks-too-much","measured_date":"2025-06-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84,"score_norm":83.1,"config":"no tools, openai-reported at o3-pro launch","trust":2,"source_type":"aggregator","source_url":"https://evals.report/models/openai-o3-pro","measured_date":"2025-06-10","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.55,"score_norm":83.87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.845454545454545 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76,"score_norm":71.83,"config":"4/4 reliability, default medium chatgpt thinking time","trust":1,"source_type":"vendor_claim","source_url":"https://help.openai.com/en/articles/9624314-model-release-notes","measured_date":"2025-06-10","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o3-pro","publisher":"OpenRouter (listed 2025-06-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o4-mini","display_name":"OpenAI o4 Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier-o-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o4-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-04-16","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":["gpt-o3-mini"],"superseded_by":null,"aliases":["o4-mini","o4-mini-2025-04-16"],"intelligence":{"index":47.75,"inputs":[{"benchmark":"aime","score_raw":83.67,"score_norm":79.59,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.178, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":6.11,"score_norm":1.23,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.856, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o4-mini, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1390.8,"score_norm":49.55,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":74.5,"score_norm":69.72,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.19, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":14.28,"score_norm":18.06,"trust":3,"source_url":"https://labs.scale.com/leaderboard/humanitys_last_exam","config":"14.28 +/- 1.37. identical to openai's own reported 'o4-mini (no tools)' = 14.28. same leaderboard lists o4-mini (high) = 18.08 +/- 1.51. [raw: humanity's last exam], no tools, o4-mini (medium) (april 2025), text-only hle"},{"benchmark":"mmlu_pro","score_raw":80.56,"score_norm":67.6,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.386, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":55.56,"score_norm":47.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/o4-mini","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.555555555555556 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":18.5,"score_norm":15.88,"trust":3,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/1.0","config":"18.5% +/- 2.7, codex cli 20.0%. [raw: terminal-bench], dated 2025-05-15. other agents on the same board with o4-mini: goose 27.5%, terminal-bench 1.0, terminus 1 agent (stanford reference harness)"}],"coverage":0.7222,"subset_bias":{"offset":-1.42,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.1,"completion_usd_per_m":4.4,"cache_read_usd_per_m":0.275,"context_length":200000,"source":"openrouter","source_model_id":"openai/o4-mini","as_of":"2026-10-06","blended_usd_per_m":1.9250000000000003,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":16.7,"score_norm":16.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26.05,"score_norm":26.05,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":60,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.6 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":83.67,"score_norm":79.59,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.178, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":90.67,"score_norm":88.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.906666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92.7,"score_norm":90.88,"config":"aime 2025, o4-mini (no tools). openai separately reports 99.5% pass@1 with a python interpreter - not used here. aime 2024 (no tools) = 93.4. [raw: aime 2025]","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/20250424210419/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":58.67,"score_norm":null,"config":"cost_per_task_usd=0.4058, eval=arc-agi-1 semi-private (v1_semi_private), model_group=o4-mini, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":6.11,"score_norm":1.23,"config":"cost_per_task_usd=0.856, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o4-mini, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":2.3,"score_norm":0,"config":"arc-agi-2 semi-private eval, o4-mini-medium. o4-mini-low = 1.6%. arc prize excluded the high-effort run (only 11/120 tasks answered) from leaderboard reporting. [raw: arc-agi-2]","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/blog/analyzing-o3-with-arc-agi","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1390.8,"score_norm":49.55,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1389.9,"score_norm":49.15,"config":"45538 votes, arena entry o4-mini-2025-04-16, leaderboard text-overall-style_control, rank 156, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1200.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1201.5,"score_norm":null,"config":"41968 votes, arena entry o4-mini-2025-04-16, leaderboard vision-overall-style_control, rank 73, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.00571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":869,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":982,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":999,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1016,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":990,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":992,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o4-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.5,"score_norm":69.72,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.19, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.38,"score_norm":75.18,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.783838383838384 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.4,"score_norm":79.44,"config":"o4-mini (no tools), openai-published chart value. corroborated by artificial analysis field lab_claimed_gpqa=0.814. (aa's own independent run measured 78.4% for o4-mini high.) [raw: gpqa diamond]","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/20250424210419/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.28,"score_norm":18.06,"config":"14.28 +/- 1.37. identical to openai's own reported 'o4-mini (no tools)' = 14.28. same leaderboard lists o4-mini (high) = 18.08 +/- 1.51. [raw: humanity's last exam], no tools, o4-mini (medium) (april 2025), text-only hle","trust":3,"source_type":"leaderboard","source_url":"https://labs.scale.com/leaderboard/humanitys_last_exam","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":16.54,"score_norm":21.38,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.165430954587581 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":68.71,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.687074829931973 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":85.93,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.859259259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.56,"score_norm":67.6,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.386, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.2,"score_norm":72,"config":"artificial analysis independent eval of o4-mini (high), field mmlu_pro=0.832. archived snapshot: aa has since dropped mmlu-pro from the live o4-mini record. [raw: mmlu-pro]","trust":2,"source_type":"aggregator","source_url":"https://web.archive.org/web/20250801000000/https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.25,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.692485549132948 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.53,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.465277777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":55.56,"score_norm":47.41,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.555555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":49.2,"score_norm":38.93,"config":"o4-mini-high. openai published no cross-domain average, retail bar on the same chart is stacked/ambiguous so it was not read. artificial analysis separately measures tau [raw: tau-bench (airline)], tau-bench airline domain only","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/20250424210419/https://openai.com/index/introducing-o3-and-o4-mini/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":18.5,"score_norm":15.88,"config":"18.5% +/- 2.7, codex cli 20.0%. [raw: terminal-bench], dated 2025-05-15. other agents on the same board with o4-mini: goose 27.5%, terminal-bench 1.0, terminus 1 agent (stanford reference harness)","trust":3,"source_type":"leaderboard","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/1.0","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o4-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o4-mini","publisher":"OpenRouter (listed 2025-04-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o4-mini-high","display_name":"OpenAI o4 Mini High","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier-o-mini-high","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o4-mini-high","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-04-16","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["o4-mini-high","o4-mini-high-2025-04-16"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.1,"completion_usd_per_m":4.4,"cache_read_usd_per_m":0.275,"context_length":200000,"source":"openrouter","source_model_id":"openai/o4-mini-high","as_of":"2026-10-06","blended_usd_per_m":1.9250000000000003,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aider_polyglot","score_raw":72,"score_norm":null,"config":"diff edit format","trust":3,"source_type":"leaderboard","source_url":"https://aider.chat/docs/leaderboards/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o4-mini-high","publisher":"OpenRouter (listed 2025-04-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-oss-1-safeguard-20b","display_name":"gpt-oss-safeguard-20b","family":{"id":"gpt-oss","vendor":"OpenAI"},"generation":"1","size_class":"safeguard-20b","params_total_b":21.51,"params_active_b":3.6,"params_exact":21511953984,"params_source":"hf_safetensors","vram_est":{"bf16_gb":43,"int8_gb":22.8,"int4_gb":12.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"openai/gpt-oss-safeguard-20b","openrouter_id":"openai/gpt-oss-safeguard-20b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-10-29","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-oss-safeguard-20b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.075,"completion_usd_per_m":0.3,"cache_read_usd_per_m":0.0375,"context_length":131072,"source":"openrouter","source_model_id":"openai/gpt-oss-safeguard-20b","as_of":"2026-10-06","blended_usd_per_m":0.13124999999999998,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":43.02,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":45.376,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":22.801,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":24.146,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":12.906,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":14.111,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":13.761,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":14.965,"method":"vendor","source_url":"https://huggingface.co/openai/gpt-oss-safeguard-20b/tree/main"},{"quant":"int8","weights_gb":22.801,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":24.146,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":92.01,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":168.89,"bw_limited_tok_s":101.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":41.14,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.83,"bw_limited_tok_s":42.9,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":358.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":930.56,"bw_limited_tok_s":558.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"bbq_ambiguous","score_raw":91,"score_norm":null,"config":"accuracy on ambiguous questions, reported as 0.91","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbq_disambiguated","score_raw":93,"score_norm":null,"config":"accuracy on disambiguated questions, reported as 0.93","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multi_policy_accuracy","score_raw":43.6,"score_norm":null,"config":"openai internal moderation evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"openai_internal_multi_policy_moderation_accuracy","score_raw":43.6,"score_norm":null,"config":"table 1, internal moderation evaluation","trust":3,"source_type":"paper","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"openai_moderation_2022","score_raw":82.9,"score_norm":null,"config":"f1 score","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"openai_moderation_2022_f1","score_raw":82.9,"score_norm":null,"config":"table 2, f1 score","trust":3,"source_type":"paper","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"personqa","score_raw":14.2,"score_norm":null,"config":"accuracy, reported as 0.142","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":7.2,"score_norm":null,"config":"accuracy, reported as 0.072","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"toxicchat","score_raw":79.9,"score_norm":null,"config":"f1 score","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"toxicchat_f1","score_raw":79.9,"score_norm":null,"config":"table 2, f1 score, short hand-written policy prompt","trust":3,"source_type":"paper","source_url":"https://cdn.openai.com/pdf/08b7dee4-8bc6-4955-a219-7793fb69090c/Technical_report__Research_Preview_of_gpt_oss_safeguard.pdf","measured_date":"2025-10-29","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-safeguard-20b","publisher":"OpenRouter (listed 2025-10-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-oss-120-b-120b","display_name":"gpt-oss-120b","family":{"id":"gpt-oss","vendor":"OpenAI"},"generation":"120","size_class":"120b","params_total_b":116.83,"params_active_b":5.1,"params_exact":116829156672,"params_source":"hf_safetensors","vram_est":{"bf16_gb":234,"int8_gb":124,"int4_gb":70.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.41,"expected_int8_gb":175,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"openai/gpt-oss-120b","openrouter_id":"openai/gpt-oss-120b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-08-05","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-oss-1-120b","gpt-oss-120B-120b","gpt-oss-120b","gpt-oss-120b-120b"],"intelligence":{"index":50.16,"inputs":[{"benchmark":"aime","score_raw":92.6,"score_norm":90.75,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.225, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1351.6,"score_norm":32.08,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.253, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":19.6,"score_norm":25.88,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.196014828544949 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":79.17,"score_norm":65.28,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.395, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":33.6,"score_norm":26.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.114, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":65.79,"score_norm":61.05,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.657894736842105 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":18.7,"score_norm":16.12,"trust":3,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"18.7% +/- 2.7, submitted 2025-11-01. same leaderboard also lists mini-swe-agent + gpt-oss-120b at 14.2% +/- 2.3. [raw: terminal-bench 2.0], terminal-bench 2.0, terminus 2 agent scaffold"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.037,"completion_usd_per_m":0.17,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"openai/gpt-oss-120b","as_of":"2026-10-06","blended_usd_per_m":0.07025,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":233.66,"kv_cache_gb_per_8k":0.3067,"min_serve_gb":245.65,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":123.84,"kv_cache_gb_per_8k":0.3067,"min_serve_gb":130.339,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":70.098,"kv_cache_gb_per_8k":0.3067,"min_serve_gb":73.91,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":65.249,"kv_cache_gb_per_8k":0.3067,"min_serve_gb":68.818,"method":"vendor","source_url":"https://huggingface.co/openai/gpt-oss-120b/tree/main"},{"quant":"int8","weights_gb":123.84,"kv_cache_gb_per_8k":0.3067,"min_serve_gb":130.339,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":57.11,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":5.1,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":107.06,"bw_limited_tok_s":60.57,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":440.79,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":5.1,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":1313.73,"bw_limited_tok_s":788.24,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":3.7,"score_norm":3.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":13.43,"score_norm":13.43,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":30.4,"score_norm":30.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.6,"score_norm":11.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.13,"score_norm":24.13,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":51,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.51 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":92.6,"score_norm":90.75,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.225, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":93.44,"score_norm":91.8,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.934416666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92.5,"score_norm":90.63,"config":"aime 2025, no tools, openai gpt-oss model card table 3 (low 50.4 / medium 80.0 / high 92.5). with-tools variant is 97.9. [raw: aime 2025 (no tools)], reasoning effort = high","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2508.10925v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":3.1,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0309734513274336 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1351.6,"score_norm":32.08,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1352.2,"score_norm":32.35,"config":"30761 votes, arena entry gpt-oss-120b, leaderboard text-overall-style_control, rank 196, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":916,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":969,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":995,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1005,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":932,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":974,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":25.54,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.25544613548194567 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":15.05,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.15047000000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.253, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.1,"score_norm":77.61,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.9,"score_norm":78.73,"config":"reasoning: high, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":67.1,"score_norm":59.3,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":68.1,"score_norm":60.7,"config":"reasoning: low, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":73.1,"score_norm":67.75,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.5,"score_norm":68.31,"config":"reasoning: medium, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":78.18,"score_norm":74.9,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.781818181818182 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.1,"score_norm":77.61,"config":"no tools, openai gpt-oss model card table 3 (low 67.1 / medium 73.1 / high 80.1). with-tools variant is 80.9. [raw: gpqa diamond (no tools)], reasoning effort = high","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2508.10925v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":13.89,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.138949196699957 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":14.9,"score_norm":18.97,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":19,"score_norm":25,"config":"reasoning: high, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":5.2,"score_norm":4.71,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":9.1,"score_norm":10.44,"config":"reasoning: low, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":8.6,"score_norm":9.71,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.3,"score_norm":13.68,"config":"reasoning: medium, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":19.6,"score_norm":25.88,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.196014828544949 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.9,"score_norm":18.97,"config":"humanity's last exam, no tools, openai gpt-oss model card table 3 (low 5.2 / medium 8.6 / high 14.9). with-tools variant is 19.0. [raw: hle (no tools)], reasoning effort = high","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2508.10925v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":68.98,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.689795918367347 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":5.65,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=high, it-bench sre, reasoning=true, run: high, source value 0.0564971751412429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":87.83,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.878306878306878 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":1.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0111111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.17,"score_norm":65.28,"config":"edition=mmlu pro v1, runner=platform, stderr=0.395, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":38.89,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.388888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":16.2,"score_norm":18.67,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":33.6,"score_norm":26.22,"config":"edition=swe-bench verified v1, runner=external, stderr=2.114, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":62.4,"score_norm":58.22,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":47.9,"score_norm":42.11,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":52.6,"score_norm":47.33,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":62.4,"score_norm":58.22,"config":"openai gpt-oss model card table 3 (low 47.9 / medium 52.6 / high 62.4). model card does not state the instance-subset size. [raw: swe-bench verified], reasoning effort = high","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2508.10925v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":65.79,"score_norm":61.05,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.657894736842105 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":67.8,"score_norm":63.73,"config":"reasoning effort = high (low 49.4 / medium 62.0 / high 67.8). no average is published, tau-bench retail domain only, the airline domain is reported separately at 49.2 (high). [raw: tau-bench retail]","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2508.10925v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":12.78,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.127835051546392 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":18.7,"score_norm":16.12,"config":"18.7% +/- 2.7, submitted 2025-11-01. same leaderboard also lists mini-swe-agent + gpt-oss-120b at 14.2% +/- 2.3. [raw: terminal-bench 2.0], terminal-bench 2.0, terminus 2 agent scaffold","trust":3,"source_type":"leaderboard","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":23.48,"score_norm":21.74,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.234848484848485 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":26.22,"score_norm":24.96,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.262172284644195 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-120b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-120b","publisher":"OpenRouter (listed 2025-08-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-oss-20-b-20b","display_name":"gpt-oss-20b","family":{"id":"gpt-oss","vendor":"OpenAI"},"generation":"20","size_class":"20b","params_total_b":20.91,"params_active_b":3.6,"params_exact":20914757184,"params_source":"hf_safetensors","vram_est":{"bf16_gb":41.8,"int8_gb":22.2,"int4_gb":12.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":7.13,"expected_int8_gb":158,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"openai/gpt-oss-20b","openrouter_id":"openai/gpt-oss-20b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-08-05","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["gpt-oss-1-20b","gpt-oss-20b"],"intelligence":{"index":46.73,"inputs":[{"benchmark":"aime","score_raw":86.04,"score_norm":82.55,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.133, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1317.6,"score_norm":16.92,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":68.94,"score_norm":61.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.465, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":10.98,"score_norm":13.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.10982391102873 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":71.64,"score_norm":52.73,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.447, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":60.7,"score_norm":56.33,"trust":2,"source_url":"https://arxiv.org/abs/2508.10925","config":"reasoning: high"},{"benchmark":"tau_bench","score_raw":60.23,"score_norm":53.64,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.60233918128655 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":13.86,"score_norm":10.42,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.138576779026217 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.018,"completion_usd_per_m":0.09,"cache_read_usd_per_m":0.009,"context_length":131072,"source":"openrouter","source_model_id":"openai/gpt-oss-20b","as_of":"2026-10-06","blended_usd_per_m":0.036,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":41.82,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":44.116,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":22.165,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":23.478,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":12.546,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":13.75,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":13.761,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":14.965,"method":"vendor","source_url":"https://huggingface.co/openai/gpt-oss-20b/tree/main"},{"quant":"int8","weights_gb":22.165,"kv_cache_gb_per_8k":0.2045,"min_serve_gb":23.478,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":92.01,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":168.89,"bw_limited_tok_s":101.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":41.14,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.83,"bw_limited_tok_s":42.9,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":358.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.6,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":930.56,"bw_limited_tok_s":558.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.2,"score_norm":1.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":3.1,"score_norm":3.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":20.7,"score_norm":20.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10,"score_norm":10,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.23,"score_norm":15.23,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":33.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.333333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":86.04,"score_norm":82.55,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.133, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":89.33,"score_norm":86.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.893333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":0.74,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.00737463126843658 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1317.6,"score_norm":16.92,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1317.6,"score_norm":16.92,"config":"10669 votes, arena entry gpt-oss-20b, leaderboard text-overall-style_control, rank 247, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":940,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":859,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":3.33,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.03331 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":68.94,"score_norm":61.89,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.465, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.5,"score_norm":65.49,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.2,"score_norm":69.3,"config":"reasoning: high, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":56.8,"score_norm":44.79,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":58,"score_norm":46.48,"config":"reasoning: low, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":66,"score_norm":57.75,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.1,"score_norm":59.3,"config":"reasoning: medium, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":68.79,"score_norm":61.68,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.687878787878788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.9,"score_norm":13.09,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.3,"score_norm":22.5,"config":"reasoning: high, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":4.2,"score_norm":3.24,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.3,"score_norm":6.32,"config":"reasoning: low, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":7,"score_norm":7.35,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.8,"score_norm":10,"config":"reasoning: medium, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":10.98,"score_norm":13.21,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.10982391102873 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":65.1,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.651020408163265 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":77.67,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.776719576719577 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":0.56,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.00555555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":71.64,"score_norm":52.73,"config":"edition=mmlu pro v1, runner=platform, stderr=0.447, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":34.38,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.34375 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":60.7,"score_norm":56.33,"config":"reasoning: high","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":37.4,"score_norm":30.44,"config":"reasoning: low","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":53.2,"score_norm":48,"config":"reasoning: medium","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2508.10925","measured_date":"2025-08-05","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":60.23,"score_norm":53.64,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.60233918128655 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":7.01,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0701030927835052 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":10.61,"score_norm":6.6,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.106060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":13.86,"score_norm":10.42,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.138576779026217 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-oss-20b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-oss-20b","publisher":"OpenRouter (listed 2025-08-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"granite-3.3-2b-instruct","display_name":"Granite-3.3-2B-Instruct","family":{"id":"granite","vendor":"IBM"},"generation":"3.3","size_class":"2b","params_total_b":2.53,"params_active_b":2.53,"params_exact":2533539840,"params_source":"hf_safetensors","vram_est":{"bf16_gb":5.1,"int8_gb":2.7,"int4_gb":1.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"ibm-granite/granite-3.3-2b-instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-04-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":5.06,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":6.731,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":5.067,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":6.738,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct/tree/main"},{"quant":"fp8","weights_gb":2.682,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":4.353,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.518,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":3.189,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":2.682,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":4.353,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":126.02,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":2.53,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":240.32,"bw_limited_tok_s":144.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":60.81,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":2.53,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":107.91,"bw_limited_tok_s":64.74,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":442.73,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":2.53,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1324.11,"bw_limited_tok_s":794.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":3.28,"score_norm":0,"config":"aime24, granite-3.3-2b-instruct column, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"alpacaeval_2_0","score_raw":43.45,"score_norm":null,"config":"thinking=true","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_hard","score_raw":28.86,"score_norm":null,"config":"thinking=true","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bigbenchhard","score_raw":63.91,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":44.33,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm8k","score_raw":72.48,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":80.51,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":65.8,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":58.09,"score_norm":null,"config":"granite-3.3-2b-instruct column, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":55.88,"score_norm":null,"config":"granite-3.3-2b-instruct column, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":58.97,"score_norm":null,"config":"olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-3.3-2b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/ibm-granite/granite-3.3-2b-base","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"granite-3.3-8b-instruct","display_name":"Granite-3.3-8B-Instruct","family":{"id":"granite","vendor":"IBM"},"generation":"3.3","size_class":"8b","params_total_b":8.17,"params_active_b":8.17,"params_exact":8170864640,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.3,"int8_gb":8.7,"int4_gb":4.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.54,"expected_int8_gb":4.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-3.3-8b-instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-04-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["granite-3.2-8b-instruct"],"superseded_by":null,"aliases":[],"intelligence":{"index":5.24,"inputs":[{"benchmark":"aime","score_raw":6.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0666666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":33.84,"score_norm":12.45,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: non-reasoning, source value 0.338383838383838 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: non-reasoning, source value 0.0417052826691381 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.105263157894737 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16.34,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":18.682,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.342,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":18.684,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-3.3-8b-instruct/tree/main"},{"quant":"fp8","weights_gb":8.66,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.002,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.902,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":7.244,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.66,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.002,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":42.74,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.17,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":74.42,"bw_limited_tok_s":44.65,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.65,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.17,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":33.41,"bw_limited_tok_s":20.05,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":197.45,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.17,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":410.04,"bw_limited_tok_s":246.02,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":1.31,"score_norm":1.31,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":3.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0.0333333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":6.67,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0666666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":33.84,"score_norm":12.45,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: non-reasoning, source value 0.338383838383838 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: non-reasoning, source value 0.0417052826691381 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":22.45,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, run: non-reasoning, source value 0.224489795918367 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":12.7,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, run: non-reasoning, source value 0.126984126984127 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":10.07,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, scicode, source value 0.100694444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":10.53,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.105263157894737 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-3-3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/ibm-granite/granite-3.3-8b-instruct","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"granite-4.0-1b","display_name":"Granite-4.0-1B","family":{"id":"granite","vendor":"IBM"},"generation":"4.0","size_class":"1b","params_total_b":1.63,"params_active_b":1.63,"params_exact":1631750144,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3.3,"int8_gb":1.7,"int4_gb":1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.46,"expected_int8_gb":2.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-4.0-1b","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-10-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":2.36,"inputs":[{"benchmark":"aime","score_raw":6.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":28.08,"score_norm":4.34,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","config":"artificial analysis harness"},{"benchmark":"hle","score_raw":4.82,"score_norm":4.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","config":"artificial analysis harness, no tools"},{"benchmark":"mmlu_pro","score_raw":34.02,"score_norm":0,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","config":"1b dense column, 5-shot cot, olmes"},{"benchmark":"tau_bench","score_raw":22.81,"score_norm":3.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","config":"artificial analysis harness, tau2"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","config":"artificial analysis harness"}],"coverage":0.5635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":3.26,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":4.931,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":3.264,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":4.935,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b/tree/main"},{"quant":"fp8","weights_gb":1.728,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":3.399,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.978,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":2.649,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1.728,"kv_cache_gb_per_8k":0.6711,"min_serve_gb":3.399,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":182.88,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":1.63,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":373.01,"bw_limited_tok_s":223.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":91.31,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":1.63,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":167.48,"bw_limited_tok_s":100.49,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":552.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":1.63,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2055.21,"bw_limited_tok_s":1233.13,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":6,"score_norm":null,"config":"artificial analysis harness, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"agieval","score_raw":49.22,"score_norm":null,"config":"0-shot cot, 1b dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":6.33,"score_norm":0,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bbh","score_raw":60.37,"score_norm":null,"config":"1b dense column, 3-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":54.82,"score_norm":null,"config":"1b dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bigcodebench","score_raw":30.18,"score_norm":null,"config":"1b dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":28.08,"score_norm":4.34,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.91,"score_norm":6.92,"config":"0-shot cot, 1b dense column, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":76.35,"score_norm":null,"config":"1b dense column, 8-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":4.82,"score_norm":4.15,"config":"artificial analysis harness, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":74,"score_norm":null,"config":"1b dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":20.54,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":77.38,"score_norm":null,"config":"1b dense column, average of instruct-strict and prompt-strict","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":4.66,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":65,"score_norm":null,"config":"1b dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"minerva_math","score_raw":45.28,"score_norm":null,"config":"0-shot cot, 1b dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":59.39,"score_norm":null,"config":"1b dense column, 5-shot, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":34.02,"score_norm":0,"config":"1b dense column, 5-shot cot, olmes","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":45,"score_norm":null,"config":"1b dense column, 5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-1b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":22.81,"score_norm":3.75,"config":"artificial analysis harness, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-nano-1b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/unsloth/granite-4.0-1b-base","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"granite-4.0-350m","display_name":"Granite-4.0-350M","family":{"id":"granite","vendor":"IBM"},"generation":"4.0","size_class":"350m","params_total_b":0.35,"params_active_b":0.35,"params_exact":352379904,"params_source":"hf_safetensors","vram_est":{"bf16_gb":0.7,"int8_gb":0.4,"int4_gb":0.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":6.02,"expected_int8_gb":2.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-4.0-350m","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32000,"release_date":"2025-10-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":1.45,"inputs":[{"benchmark":"aime","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=granite-4-0-350m","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":26.1,"score_norm":1.55,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=granite-4-0-350m","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":5.5,"score_norm":5.15,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=granite-4-0-350m","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"mmlu_pro","score_raw":12.4,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=granite-4-0-350m","config":"artificial analysis harness, mmlu-pro"},{"benchmark":"tau_bench","score_raw":13.2,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=granite-4-0-350m","config":"artificial analysis harness, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=granite-4-0-350m","config":"artificial analysis harness, terminal-bench hard"}],"coverage":0.5635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":0.7,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":1.935,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":0.705,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":1.94,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m/tree/main"},{"quant":"fp8","weights_gb":0.371,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":1.606,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.21,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":1.445,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.371,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":1.606,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":510.35,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.35,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1737.14,"bw_limited_tok_s":1042.29,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":318.8,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.35,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":780,"bw_limited_tok_s":468,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":851.69,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.35,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9571.43,"bw_limited_tok_s":5742.86,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"agieval","score_raw":26.22,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":0,"score_norm":0,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"attaq","score_raw":82.53,"score_norm":null,"config":"350m dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":33.07,"score_norm":null,"config":"3-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":39.32,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bigcodebench","score_raw":11.14,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"cruxeval_o","score_raw":23.75,"score_norm":null,"config":"350m dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepmind_math","score_raw":8.45,"score_norm":null,"config":"0-shot cot, 350m dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":26.1,"score_norm":1.55,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":24.11,"score_norm":0,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":30.71,"score_norm":null,"config":"8-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm_symbolic","score_raw":26.76,"score_norm":null,"config":"350m dense column, 8-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":5.5,"score_norm":5.15,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":37,"score_norm":null,"config":"350m dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":39,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":55.4,"score_norm":null,"config":"average strict","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"include","score_raw":27.74,"score_norm":null,"config":"350m dense column, 5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":38,"score_norm":null,"config":"350m dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":48,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":14.72,"score_norm":null,"config":"8-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"minerva_math","score_raw":13.04,"score_norm":null,"config":"0-shot cot, 350m dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":35.01,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":12.4,"score_norm":0,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":12.13,"score_norm":0,"config":"5-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":28.23,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multiple","score_raw":15.99,"score_norm":null,"config":"350m dense column, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"salad_bench","score_raw":97.12,"score_norm":null,"config":"350m dense column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":13.2,"score_norm":0,"config":"artificial analysis harness, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"artificial analysis harness, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=granite-4-0-350m","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"granite-4.0-h-micro","display_name":"Granite 4.0 Micro","family":{"id":"granite","vendor":"IBM"},"generation":"4.0","size_class":"frontier-micro","params_total_b":3.19,"params_active_b":3.19,"params_exact":3191396096,"params_source":"hf_safetensors","vram_est":{"bf16_gb":6.4,"int8_gb":3.4,"int4_gb":1.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.42,"expected_int8_gb":4.8,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-4.0-h-micro","openrouter_id":"ibm-granite/granite-4.0-h-micro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131000,"release_date":"2025-10-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":5.45,"inputs":[{"benchmark":"aime","score_raw":6,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.06 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":33.64,"score_norm":12.17,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.336363636363636 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5,"score_norm":4.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0500463392029657 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":12.57,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.125730994152047 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.017,"completion_usd_per_m":0.112,"cache_read_usd_per_m":null,"context_length":131000,"source":"openrouter","source_model_id":"ibm-granite/granite-4.0-h-micro","as_of":"2026-10-06","blended_usd_per_m":0.04075,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":6.38,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":7.447,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":6.383,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":7.45,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.0-h-micro/tree/main"},{"quant":"fp8","weights_gb":3.381,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":4.448,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.914,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":2.981,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":3.381,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":4.448,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":102.62,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":190.6,"bw_limited_tok_s":114.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.84,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":85.58,"bw_limited_tok_s":51.35,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":386.54,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1050.16,"bw_limited_tok_s":630.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.1,"score_norm":5.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.0-h-micro","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":1.95,"score_norm":1.95,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":5.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.0566666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":6,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.06 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":33.64,"score_norm":12.17,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.336363636363636 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5,"score_norm":4.41,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0500463392029657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":24.76,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.247619047619048 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":17.99,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.17989417989418 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":11.92,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.119212962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":12.57,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.125730994152047 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-micro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.0-h-micro","publisher":"OpenRouter (listed 2025-10-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"granite-4.0-h-small-32b-instruct","display_name":"Granite-4.0-H-Small","family":{"id":"granite","vendor":"IBM"},"generation":"4.0","size_class":"32b","params_total_b":32.21,"params_active_b":9,"params_exact":32207337984,"params_source":"hf_safetensors","vram_est":{"bf16_gb":64.4,"int8_gb":34.1,"int4_gb":19.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.93,"expected_int8_gb":31.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-4.0-h-small","openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-10-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["granite-4.0-h-small"],"intelligence":{"index":9.02,"inputs":[{"benchmark":"aime","score_raw":13.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.136666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":41.62,"score_norm":23.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.416161616161616 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.8,"score_norm":2.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0379981464318814 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":17.25,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.172514619883041 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":64.42,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":67.775,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":64.415,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":67.77,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.0-h-small/tree/main"},{"quant":"fp8","weights_gb":34.143,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":35.984,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.326,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":20.46,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.143,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":35.984,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":74.99,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":9,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":135.11,"bw_limited_tok_s":81.07,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":16.87,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":9,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":30.33,"bw_limited_tok_s":17.16,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":182.56,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":9,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":372.22,"bw_limited_tok_s":223.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":4.93,"score_norm":4.93,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":11,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.11 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":13.67,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.136666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.62,"score_norm":23.41,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.416161616161616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.8,"score_norm":2.65,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0379981464318814 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":31.5,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.314965986394558 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":25.08,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.250793650793651 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":20.95,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.209490740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":17.25,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.172514619883041 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-0-h-small","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models","publisher":"IBM"},{"url":"https://www.ibm.com/granite/docs/models/granite","publisher":"IBM"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"granite-4.2-8b","display_name":"Granite 4.2 8B","family":{"id":"granite","vendor":"IBM"},"generation":"4.2","size_class":"8b","params_total_b":8.79,"params_active_b":8.79,"params_exact":8791592960,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.6,"int8_gb":9.3,"int4_gb":5.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.23,"expected_int8_gb":30.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"ibm-granite/granite-4.2-8b","openrouter_id":"ibm-granite/granite-4.2-8b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2026-08-31","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["granite-4.1-8b"],"superseded_by":null,"aliases":[],"intelligence":{"index":41.07,"inputs":[{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","config":"25, 8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. math-reasoning benchmark, card-wide settings: temperature=1.0, default full thinking/cot mode (aime-style tasks are the canonical thinking-mode showcase for this release). harness: nemo evaluator sdk-based framework. no shot-count stated. value already 0-100 scale as published., top_p=0.95"},{"benchmark":"arena_elo","score_raw":1288.1,"score_norm":3.78,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":63.1,"score_norm":53.66,"trust":2,"source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","config":"different harness/grader. no explicit eval-run date given., distinct from ibm's card's plain 'gpqa' figure of 64.14 — do not merge, sourced from artificial analysis per page footer ('metrics sourced from artificial analysis'). value already 0-100 scale as published (63.1%). this is aa's own gpqa diamond run"},{"benchmark":"hle","score_raw":9.7,"score_norm":11.32,"trust":2,"source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","config":"full name: humanity's last exam. sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given."},{"benchmark":"mmlu_pro","score_raw":74.04,"score_norm":56.73,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95"},{"benchmark":"swe_bench_verified","score_raw":47.67,"score_norm":41.86,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide inference settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95"},{"benchmark":"terminal_bench","score_raw":20.56,"score_norm":18.31,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. named edition: terminal-bench 2.1. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95"}],"coverage":0.7539,"subset_bias":{"offset":1.13,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.06,"completion_usd_per_m":0.25,"cache_read_usd_per_m":0.015,"context_length":131072,"source":"openrouter","source_model_id":"ibm-granite/granite-4.2-8b","as_of":"2026-10-06","blended_usd_per_m":0.1075,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":17.58,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":19.922,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.583,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":19.925,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b/tree/main"},{"quant":"fp8","weights_gb":9.317,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.659,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.274,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":7.616,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.317,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.659,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.85,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":69.17,"bw_limited_tok_s":41.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":31.06,"bw_limited_tok_s":18.63,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":186.11,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":381.11,"bw_limited_tok_s":228.67,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.3,"score_norm":1.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":9.2,"score_norm":9.2,"config":"composite index score (not a raw 0-100 percentage) on artificial analysis's own scale, sourced from artificial analysis per page footer. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":22.4,"score_norm":22.4,"config":"composite index score (not a raw 0-100 percentage) on artificial analysis's own scale, sourced from artificial analysis per page footer. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":22.4,"score_norm":22.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.1,"score_norm":11.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.6,"score_norm":19.6,"config":"composite index score (not a raw 0-100 percentage) on artificial analysis's own scale, only percentile rank ('better than 19% of models compared')., page footer states 'metrics sourced from artificial analysis' (artificialanalysis.ai/models/granite-4-2-8b). openrouter benchmarks panel gives no explicit eval-run date","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20,"score_norm":20,"config":"a composite index (not a raw 0-100 percentage benchmark) aggregating 9 evaluations per aa's own page: gdpval-aa v2, aa-lcr. page displays this as a rounded integer (20), aa-omniscience, critpt, do not treat as duplicate/conflicting since they are different precision levels of aa's own number. page states model release date (2026-08-25) but does not give a separate 'evaluated on' date for the index run itself, gpqa diamond, hence null measured_date., humanity's last exam, openrouter's aggregator restatement of the same underlying aa measurement shows an unrounded 19.6 — recorded as a separate claim below, scicode, terminal-bench v2.1, version: artificial analysis intelligence index v4.1.1, τ3-banking","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":43.3,"score_norm":null,"config":"artificial analysis long context reasoning eval. sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":11.2,"score_norm":null,"config":"sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":68,"score_norm":null,"config":"higher is better despite containing 'hallucination' in the name — this is the rate of avoiding hallucination (not a hallucination-rate metric), so do not flag as lower-is-better. sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":86.67,"score_norm":83.34,"config":"25, 8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. math-reasoning benchmark, card-wide settings: temperature=1.0, default full thinking/cot mode (aime-style tasks are the canonical thinking-mode showcase for this release). harness: nemo evaluator sdk-based framework. no shot-count stated. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1288.1,"score_norm":3.78,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":65.19,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. edition: arena-hard-v2 (llm-judged win-rate style benchmark). harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bfcl_v4","score_raw":52.39,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. edition explicitly stated by ibm as bfcl v4 (not v3). harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide settings: temperature=1.0, default full thinking/cot mode, tool-calling benchmark (model card notes tool calling is reasoning-augmented by default). value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"birdbench","score_raw":41.07,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.3,"score_norm":null,"config":"research-level physics reasoning eval. sourced from artificial analysis per page footer. value already 0-100 scale as published (very low score is plausible for an 8b model on this hard eval). no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":63.1,"score_norm":53.66,"config":"different harness/grader. no explicit eval-run date given., distinct from ibm's card's plain 'gpqa' figure of 64.14 — do not merge, sourced from artificial analysis per page footer ('metrics sourced from artificial analysis'). value already 0-100 scale as published (63.1%). this is aa's own gpqa diamond run","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":64.14,"score_norm":55.13,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. ibm labels this simply 'gpqa' (not explicitly 'diamond'), card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., do not assume it is the same run/config as third-party gpqa diamond scores of this model. harness: nemo evaluator sdk-based framework. no shot-count stated, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":9.7,"score_norm":11.32,"config":"full name: humanity's last exam. sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"hmmt_feb25","score_raw":78.33,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. edition: hmmt february 2025 set. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifbench_prompt","score_raw":79.33,"score_norm":null,"config":"'(prompt)' sub-metric of ifbench as explicitly labeled by ibm (prompt-level accuracy, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., distinct from any instruction-level variant ibm may also track internally). 8b dense column. harness: nemo evaluator sdk-based framework. no shot-count stated, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":73.24,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. edition explicitly stated as v6. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":74.04,"score_norm":56.73,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"mmlu_prox_lite_ibm","score_raw":61.06,"score_norm":null,"config":"card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., explicitly labeled '(ibm)' in the card — not the public mmlu-prox benchmark as released by its original authors, ibm's own internal variant, top_p=0.95, treat as an ibm in-house eval. 8b dense column. harness: nemo evaluator sdk-based framework. no shot-count stated","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"profbench","score_raw":41.2,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler_128k","score_raw":71.41,"score_norm":null,"config":"as explicitly labeled — matches the model's native 128k context claim. 8b dense column. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, context-length edition: 128k tokens, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ruler_64k","score_raw":80.99,"score_norm":null,"config":"as explicitly labeled. 8b dense column. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, context-length edition: 64k tokens, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":30.4,"score_norm":null,"config":"artificial analysis's own scicode run — distinct from ibm's card scicode figure of 36.09 (different harness/grader, do not merge). sourced from artificial analysis per page footer. value already 0-100 scale as published. no explicit eval-run date given.","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/ibm-granite/granite-4.2-8b","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":36.09,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no shot-count stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":30.78,"score_norm":null,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: 'evaluation framework based on nemo evaluator sdk' (card's own words). no shot-count/cot stated per-benchmark, card mandates temperature=1.0, model defaults to full thinking/cot mode unless run with enable_thinking=false. value already 0-100 scale as published, no conversion applied., top_p=0.95 across all tasks","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":19.11,"score_norm":23.52,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide inference settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":47.67,"score_norm":41.86,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide inference settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau3_bench_avg","score_raw":58.06,"score_norm":null,"config":"'avg' variant of τ3-bench as labeled by ibm. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, 8b dense column of ibm's 3b/8b/30b granite-4.2 eval table, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":20.56,"score_norm":18.31,"config":"8b dense column of ibm's 3b/8b/30b granite-4.2 eval table. named edition: terminal-bench 2.1. harness: nemo evaluator sdk-based framework. no per-benchmark shot-count/cot stated, card-wide settings: temperature=1.0, default full thinking/cot mode. value already 0-100 scale as published., top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.2-8b","measured_date":"2026-08-25","observed_at":"2026-09-02","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.2-8b","publisher":"OpenRouter (listed 2026-08-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-0.1-build","display_name":"Grok Build 0.1","family":{"id":"grok","vendor":"xAI"},"generation":"0.1","size_class":"frontier-build","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-build-0.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":256000,"release_date":"2026-05-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["grok-build-0.1"],"intelligence":{"index":69.49,"inputs":[{"benchmark":"gpqa_diamond","score_raw":89.49,"score_norm":90.83,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.894949494949495 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":38.28,"score_norm":53.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.382761816496756 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":67.78,"score_norm":61.73,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"terminal_bench","score_raw":52.06,"score_norm":55.36,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.52059925093633 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.4365,"subset_bias":{"offset":-3.52,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":2,"cache_read_usd_per_m":0.2,"context_length":256000,"source":"openrouter","source_model_id":"x-ai/grok-build-0.1","as_of":"2026-10-06","blended_usd_per_m":1.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":28.9,"score_norm":28.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-build-0.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":28.86,"score_norm":28.86,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":51.5,"score_norm":51.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-build-0.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":27.2,"score_norm":27.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-build-0.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.65,"score_norm":40.65,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.7 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":6.37,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":9.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0914285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":35.75,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.35754499999999995 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.49,"score_norm":90.83,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.894949494949495 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":38.28,"score_norm":53.35,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.382761816496756 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":67.78,"score_norm":61.73,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":76.53,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.765317919075144 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":50.23,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.502314814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":13.4,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.134020618556701 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.06,"score_norm":55.36,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.52059925093633 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-build-0-1-06-16","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-build-0.1","publisher":"OpenRouter (listed 2026-05-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-4.7","display_name":"Grok 4.7","family":{"id":"grok","vendor":"xAI"},"generation":"4.7","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.7","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":500000,"release_date":"2026-09-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["grok-4.20","grok-4.20-multi-agent","grok-4.3","grok-4.5","grok-4.6"],"superseded_by":null,"aliases":[],"intelligence":{"index":73.39,"inputs":[{"benchmark":"arena_elo","score_raw":1442.5,"score_norm":72.59,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"hle","score_raw":43.14,"score_norm":60.5,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-7","config":"effort=xhigh, reasoning=on, run: xhigh, tools=off"},{"benchmark":"livebench","score_raw":77.4,"score_norm":83.11,"trust":3,"source_url":"https://livebench.ai/","config":"effort=xhigh, livebench-2026-06-25 release, run: xhigh"},{"benchmark":"terminal_bench","score_raw":73.408,"score_norm":80.48,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"effort=xhigh, v2.1"}],"coverage":0.357,"subset_bias":{"offset":-5.7,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.5,"context_length":500000,"source":"openrouter","source_model_id":"x-ai/grok-4.7","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_briefcase","score_raw":1657.2,"score_norm":null,"config":"effort=xhigh, elo, reasoning=on, run: xhigh, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_briefcase","score_raw":1657,"score_norm":null,"config":"effort=xhigh, elo, run: xhigh, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_coding_agent_index","score_raw":56.27,"score_norm":null,"config":"agent=grok build, effort=xhigh, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":46.45,"score_norm":46.45,"config":"effort=xhigh, index v4.3.2, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":46.4,"score_norm":46.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":77,"score_norm":null,"config":"effort=high, reasoning=on, run: high, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":47.82,"score_norm":null,"config":"effort=high, reasoning=on, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":32.03,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":70.7,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/x-ai/grok-4.7","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1442.5,"score_norm":72.59,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1439.1,"score_norm":71.08,"config":"effort=xhigh, run: xhigh, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-25","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1637.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1629.3,"score_norm":null,"config":"code/webdev overall, effort=xhigh, raw (no style control), run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-09-25","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":59.15,"score_norm":null,"config":"effort=xhigh, harness=grok cli, overall, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://matharena.ai/models/xai_grok_47","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":65.56,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"biomysterybench","score_raw":69.259,"score_norm":null,"config":"effort=xhigh, harness=terminus 2","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/biomysterybench","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"biosecbench_function","score_raw":43.3,"score_norm":null,"config":"agent=grok build, effort=xhigh, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"biosecbench_surveillance","score_raw":48,"score_norm":null,"config":"agent=grok build, effort=xhigh, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"bixbench","score_raw":88.4,"score_norm":null,"config":"effort=high, run: high, tools=on, zero-shot mcq","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"brokenarxiv","score_raw":42.63,"score_norm":null,"config":"effort=xhigh, harness=grok cli, overall, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://matharena.ai/models/xai_grok_47","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cadgenbench","score_raw":44.4,"score_norm":null,"config":"agent=grok build, effort=high, generation split, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cathedralbench","score_raw":29,"score_norm":null,"config":"effort=xhigh, hard subset, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":44.816,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/code-migration","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":18,"score_norm":null,"config":"effort=high, reasoning=on, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":46.3,"score_norm":null,"config":"effort=xhigh, run: xhigh, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":46.3,"score_norm":null,"config":"effort=xhigh, run: xhigh, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cursorbench","score_raw":46.3,"score_norm":null,"config":"effort=xhigh, run: high, v4.0","trust":1,"source_type":"leaderboard","source_url":"https://cursor.com/cursorbench","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cve_bench","score_raw":37.7,"score_norm":null,"config":"agent=grok build, effort=high, reward, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":80.3,"score_norm":null,"config":"agent=grok build, effort=high, mean reproduced, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":72.57,"score_norm":null,"config":"agent=grok build, effort=xhigh, run: xhigh, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":71,"score_norm":null,"config":"effort=high, run: xhigh, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":71,"score_norm":null,"config":"effort=high, harness=mini-swe-agent, run: high, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1262,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1288,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1241,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"eebench","score_raw":64,"score_norm":null,"config":"harness=grok cli, v1 core","trust":1,"source_type":"leaderboard","source_url":"https://eebench.org/","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"eebench","score_raw":64,"score_norm":null,"config":"effort=xhigh, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"eebench","score_raw":66,"score_norm":null,"config":"effort=xhigh, harness=grok build, run: xhigh, v1 core","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"emb","score_raw":66.991,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/emb","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":52.251,"score_norm":null,"config":"effort=xhigh, v2","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/fabv2","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":29,"score_norm":null,"config":"effort=xhigh, harness=proximus, mean@5, run: xhigh, v2","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":29.5,"score_norm":null,"config":"harness=proximus, mean@5, v2","trust":1,"source_type":"leaderboard","source_url":"https://www.frontierswe.com/","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"furniture_assembly","score_raw":20.83,"score_norm":null,"config":"effort=xhigh, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://epoch.ai/models/grok-4-7","measured_date":"2026-09-24","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdp_pdf","score_raw":23.2,"score_norm":null,"config":"all-pass, effort=high, reasoning=on, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1695.2,"score_norm":null,"config":"effort=xhigh, elo, reasoning=on, run: xhigh, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1695,"score_norm":null,"config":"effort=xhigh, elo, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":59.76,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"harvey_lab_vals","score_raw":12.5,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/hlab","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"harvey_lab_vals","score_raw":19.6,"score_norm":null,"config":"effort=xhigh, harvey final score, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"harvey_lab_vals","score_raw":19.6,"score_norm":null,"config":"effort=xhigh, harvey final score, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":56.7,"score_norm":null,"config":"effort=xhigh, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":56.7,"score_norm":null,"config":"effort=xhigh, run: xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":43.14,"score_norm":60.5,"config":"effort=xhigh, reasoning=on, run: xhigh, tools=off","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle_diamond","score_raw":23.4,"score_norm":null,"config":null,"trust":3,"source_type":"leaderboard","source_url":"https://labs.scale.com/leaderboard/hle-diamond","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"ioi","score_raw":57.722,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/ioi","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":42.08,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"lab_bench_practical","score_raw":76.8,"score_norm":null,"config":"effort=high, mcq, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"latchbio_capabilities","score_raw":44.5,"score_norm":null,"config":"effort=xhigh, overall, run: xhigh, v1.0","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"legal_bench","score_raw":84.389,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/legal_bench","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"legal_research_bench","score_raw":47.115,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/legal_research","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":77.4,"score_norm":83.11,"config":"effort=xhigh, livebench-2026-06-25 release, run: xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"medcode","score_raw":49.553,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/medcode","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"medscribe","score_raw":89.377,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/medscribe","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":15,"score_norm":null,"config":"effort=xhigh, reasoning=on, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"mysterymechanism","score_raw":25.225,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mysterymechanism","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":0.5,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/programbench","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"proofbench","score_raw":26,"score_norm":null,"config":"effort=xhigh, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/proof_bench","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"protocolqa_open_ended","score_raw":70.4,"score_norm":null,"config":"effort=high, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"public_benefits_bench","score_raw":65.629,"score_norm":null,"config":"effort=xhigh, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/public-benefits-bench","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"sage","score_raw":40.79,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/sage","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":57.75,"score_norm":null,"config":"effort=high, reasoning=on, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7-high","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_atlas_qna","score_raw":62.9,"score_norm":null,"config":"agent=grok build, effort=xhigh, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/agents/coding-agents","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"swe_marathon","score_raw":46,"score_norm":null,"config":"effort=high, run: high, v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"tax_agent_bench","score_raw":65.596,"score_norm":null,"config":"effort=xhigh","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/tax_agent_bench","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":73.408,"score_norm":80.48,"config":"effort=xhigh, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.58,"score_norm":38.33,"config":"agent=grok build, effort=xhigh, v4.0","trust":3,"source_type":"leaderboard","source_url":"https://www.tbench.ai/leaderboard","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":25.76,"score_norm":24.42,"config":"effort=xhigh, reasoning=on, run: xhigh, v4.0","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.6,"score_norm":38.35,"config":"effort=xhigh, run: xhigh, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-7","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38,"score_norm":38.82,"config":"agent=grok build, effort=xhigh, run: xhigh, v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":11.429,"score_norm":null,"config":"effort=xhigh, v0.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-science","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":60.215,"score_norm":null,"config":"effort=xhigh, v2","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/vals_index","measured_date":"2026-09-23","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vct","score_raw":63,"score_norm":null,"config":"effort=high, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench","score_raw":86.175,"score_norm":null,"config":"effort=xhigh, harness=openhands, v1.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/vibe-code","measured_date":"2026-09-22","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"wmdp_bio","score_raw":88.1,"score_norm":null,"config":"effort=high, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"wmdp_chem","score_raw":84.9,"score_norm":null,"config":"effort=high, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"wmdp_cyber","score_raw":88.1,"score_norm":null,"config":"effort=high, run: high","trust":1,"source_type":"vendor_claim","source_url":"https://media.x.ai/v1/website/4p7card-5eccc980.pdf","measured_date":"2026-09-21","observed_at":"2026-09-27","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.7","publisher":"OpenRouter (listed 2026-09-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hermes-4-405b","display_name":"Hermes 4 405B","family":{"id":"hermes","vendor":"nousresearch"},"generation":"4","size_class":"405b","params_total_b":405.85,"params_active_b":405.85,"params_exact":405853388800,"params_source":"hf_safetensors","vram_est":{"bf16_gb":812,"int8_gb":430,"int4_gb":244,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.09,"expected_int8_gb":38.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"NousResearch/Hermes-4-405B","openrouter_id":"nousresearch/hermes-4-405b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-08-26","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":["hermes-3-llama-3.1-405b"],"superseded_by":null,"aliases":[],"intelligence":{"index":49.64,"inputs":[{"benchmark":"aime","score_raw":78.1,"score_norm":72.63,"trust":1,"source_url":"https://arxiv.org/abs/2508.18255","config":"aime'25, pass@1 (64 samples), reasoning mode"},{"benchmark":"gpqa_diamond","score_raw":70.6,"score_norm":64.23,"trust":1,"source_url":"https://arxiv.org/abs/2508.18255","config":"lighteval, pass@1 (8 samples), reasoning mode"},{"benchmark":"hle","score_raw":10.89,"score_norm":13.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-405b-reasoning","config":"aa intelligence index eval, hermes 4 405b (reasoning)"},{"benchmark":"mmlu_pro","score_raw":80.6,"score_norm":67.67,"trust":1,"source_url":"https://arxiv.org/abs/2508.18255","config":"lighteval, reasoning mode"},{"benchmark":"tau_bench","score_raw":22.22,"score_norm":2.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-405b-reasoning","config":"aa eval, hermes 4 405b (reasoning), tau2-bench telecom"}],"coverage":0.5159,"subset_bias":{"offset":2.69,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":3,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"nousresearch/hermes-4-405b","as_of":"2026-10-06","blended_usd_per_m":1.5,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":811.7,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":856.513,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":811.707,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":856.52,"method":"vendor","source_url":"https://huggingface.co/NousResearch/Hermes-4-405B/tree/main"},{"quant":"fp8","weights_gb":430.201,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":455.939,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":243.51,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":259.913,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":430.201,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":455.939,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.5,"score_norm":7.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-4-405b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":78.1,"score_norm":72.63,"config":"aime'25, pass@1 (64 samples), reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":93.7,"score_norm":null,"config":"arena-hard v1, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":86.3,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":83.5,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":70.6,"score_norm":64.23,"config":"lighteval, pass@1 (8 samples), reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.89,"score_norm":13.07,"config":"aa intelligence index eval, hermes 4 405b (reasoning)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-405b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":81.5,"score_norm":null,"config":"loose, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":61.4,"score_norm":null,"config":"lcbv6 aug2024+, pass@1 (16 samples), reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":96.2,"score_norm":null,"config":"pass@1 (4 samples), reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":87.2,"score_norm":null,"config":"lighteval, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.6,"score_norm":67.67,"config":"lighteval, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":66.1,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":25.8,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2508.18255","measured_date":"2025-08-25","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":22.22,"score_norm":2.96,"config":"aa eval, hermes 4 405b (reasoning), tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-405b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-4-405b","publisher":"OpenRouter (listed 2025-08-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hermes-4-70b","display_name":"hermes-4-70b","family":{"id":"hermes","vendor":"nousresearch"},"generation":"4","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.43,"expected_int8_gb":31.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"NousResearch/Hermes-4-70B","openrouter_id":"nousresearch/hermes-4-70b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-08-26","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["hermes-3-llama-3.1-70b"],"superseded_by":null,"aliases":[],"intelligence":{"index":42.96,"inputs":[{"benchmark":"aime","score_raw":68.67,"score_norm":60.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","config":"aime 2025, artificial analysis harness, reasoning mode"},{"benchmark":"gpqa_diamond","score_raw":69.9,"score_norm":63.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","config":"artificial analysis harness, reasoning mode"},{"benchmark":"hle","score_raw":8.76,"score_norm":9.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","config":"artificial analysis harness, no tools, reasoning mode"},{"benchmark":"mmlu_pro","score_raw":80.7,"score_norm":67.83,"trust":1,"source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","config":"hermes 4 70b, reasoning on (r column)"},{"benchmark":"tau_bench","score_raw":22.51,"score_norm":3.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","config":"artificial analysis harness, reasoning mode, tau2"},{"benchmark":"terminal_bench","score_raw":4.55,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","config":"artificial analysis harness, reasoning mode"}],"coverage":0.5635,"subset_bias":{"offset":0.93,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.839,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.847,"method":"vendor","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B/tree/main"},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":47.131,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.87,"bw_limited_tok_s":2.32,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":53.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70.55,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":94.97,"bw_limited_tok_s":56.98,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":9.67,"score_norm":null,"config":"artificial analysis harness, reasoning mode, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":11.33,"score_norm":0,"config":"aime 2025, artificial analysis harness, non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":68.67,"score_norm":60.84,"config":"aime 2025, artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":9.5,"score_norm":0,"config":"aime'24, non-reasoning (n column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":67.4,"score_norm":59.25,"config":"aime'25, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":73.5,"score_norm":66.88,"config":"aime'24, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":90.1,"score_norm":null,"config":"reasoning on (r column), v1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":87.8,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":85,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49.09,"score_norm":33.93,"config":"artificial analysis harness, non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":69.9,"score_norm":63.24,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":33.3,"score_norm":11.69,"config":"hermes 4 70b, non-reasoning (n column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.1,"score_norm":57.89,"config":"hermes 4 70b, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.57,"score_norm":2.31,"config":"artificial analysis harness, no tools, non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.76,"score_norm":9.94,"config":"artificial analysis harness, no tools, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":31.29,"score_norm":null,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":78.7,"score_norm":null,"config":"loose, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":50.5,"score_norm":null,"config":"lcbv6 aug2024+, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":95.6,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":88.4,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":54.9,"score_norm":24.83,"config":"hermes 4 70b, non-reasoning (n column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.7,"score_norm":67.83,"config":"hermes 4 70b, reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":70.4,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":17.9,"score_norm":null,"config":"reasoning on (r column)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/NousResearch/Hermes-4-70B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":22.51,"score_norm":3.35,"config":"artificial analysis harness, reasoning mode, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.55,"score_norm":0,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-4-llama-3-1-70b-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-4-70b","publisher":"OpenRouter (listed 2025-08-26)"}],"last_verified":"2026-09-09","last_enriched_at":null},{"id":"hunyuan-1-a13b-instruct","display_name":"Hunyuan A13B Instruct","family":{"id":"hunyuan","vendor":"Tencent"},"generation":"1","size_class":"a13b","params_total_b":80.39,"params_active_b":13,"params_exact":80393183232,"params_source":"hf_safetensors","vram_est":{"bf16_gb":161,"int8_gb":85.2,"int4_gb":48.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.47,"expected_int8_gb":296,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"tencent/Hunyuan-A13B-Instruct","openrouter_id":"tencent/hunyuan-a13b-instruct","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07-08","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["hunyuan-a13b-instruct"],"intelligence":{"index":67.22,"inputs":[{"benchmark":"aime","score_raw":87.3,"score_norm":84.13,"trust":1,"source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","config":"aime 2024, default slow-thinking reasoning"},{"benchmark":"gpqa_diamond","score_raw":71.2,"score_norm":65.07,"trust":1,"source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","config":"default slow-thinking reasoning"},{"benchmark":"tau_bench","score_raw":54.7,"score_norm":46.27,"trust":1,"source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","config":"default slow-thinking reasoning, tau-bench (v1)"}],"coverage":0.2858,"subset_bias":{"offset":15.46,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.14,"completion_usd_per_m":0.57,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"tencent/hunyuan-a13b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.2475,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":160.78,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":169.893,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":160.804,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":169.918,"method":"vendor","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct/tree/main"},{"quant":"fp8","weights_gb":85.213,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":90.547,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":48.234,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":51.719,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":85.213,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":90.547,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.74,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":21,"bw_limited_tok_s":11.88,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":236.19,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":13,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":515.38,"bw_limited_tok_s":309.23,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":76.8,"score_norm":71,"config":"aime 2025, default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87.3,"score_norm":84.13,"config":"aime 2024, default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"artifactsbench","score_raw":43,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":89.1,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":78.3,"score_norm":null,"config":"default slow-thinking reasoning, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"c3_bench","score_raw":63.5,"score_norm":null,"config":"default slow-thinking reasoning, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"complexfuncbench","score_raw":61.2,"score_norm":null,"config":"default slow-thinking reasoning, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":91.1,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"fullstackbench","score_raw":67.8,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.2,"score_norm":65.07,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":84.7,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":63.9,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":94.3,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"olympiadbench","score_raw":82.7,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"sysbench","score_raw":76.1,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":54.7,"score_norm":46.27,"config":"default slow-thinking reasoning, tau-bench (v1)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"zebralogic","score_raw":84.7,"score_norm":null,"config":"default slow-thinking reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hunyuan-A13B-Instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#tencent/hunyuan-a13b-instruct","publisher":"OpenRouter (listed 2025-07-08)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hy-3","display_name":"Hy3","family":{"id":"hy","vendor":"tencent"},"generation":"3","size_class":"frontier","params_total_b":298.79,"params_active_b":null,"params_exact":298786155776,"params_source":"hf_safetensors","vram_est":{"bf16_gb":598,"int8_gb":317,"int4_gb":179,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.19,"expected_int8_gb":378,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"tencent/Hy3","openrouter_id":"tencent/hy3","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-07-06","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["hy3","hy3-preview"],"intelligence":{"index":74.12,"inputs":[{"benchmark":"arena_elo","score_raw":1456,"score_norm":78.61,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.7,"score_norm":91.13,"trust":2,"source_url":"https://artificialanalysis.ai/models/hy3","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.896969696969697 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":33.5,"score_norm":46.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/hy3","config":"hle as run in aa intelligence index v4.1.1 (text-only, model slug 'hy3', no tools), page data field hle = 0.334569045412419 [raw: humanity's last exam (artificial analysis independent run)]"},{"benchmark":"swe_bench_verified","score_raw":78,"score_norm":75.56,"trust":1,"source_url":"https://huggingface.co/tencent/Hy3","config":"resolved; official hy3 model card"},{"benchmark":"terminal_bench","score_raw":64.42,"score_norm":69.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/hy3","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.644194756554307 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5555,"subset_bias":{"offset":-1.13,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.18,"completion_usd_per_m":0.6,"cache_read_usd_per_m":0.06,"context_length":262144,"source":"openrouter","source_model_id":"tencent/hy3-preview","as_of":"2026-10-06","blended_usd_per_m":0.28500000000000003,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":597.58,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":630.143,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":597.578,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":630.141,"method":"vendor","source_url":"https://huggingface.co/tencent/Hy3/tree/main"},{"quant":"fp8","weights_gb":316.717,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":335.237,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":179.274,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":190.922,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":316.717,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":335.237,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":24.1,"score_norm":24.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":31.42,"score_norm":31.42,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":58.8,"score_norm":58.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.3,"score_norm":25.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":42.21,"score_norm":42.21,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.746666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":25.6,"score_norm":null,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1456,"score_norm":78.61,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1456.7,"score_norm":78.92,"config":"5877 votes, arena entry hy3-tencent-cloud-text, leaderboard text-overall-style_control, rank 56, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1507.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1518.1,"score_norm":null,"config":"2680 votes, arena entry hy3-tencent-cloud-code, code/overall, leaderboard webdev-overall-raw, rank 28, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":4.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0485714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":28,"score_norm":null,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1191,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1183,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1134,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1189,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#tencent/hy3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":35.72,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.3571849999999999 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.7,"score_norm":91.13,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.896969696969697 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.4,"score_norm":92.11,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.46,"score_norm":46.26,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.334569045412419 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.5,"score_norm":46.32,"config":"hle as run in aa intelligence index v4.1.1 (text-only, model slug 'hy3', no tools), page data field hle = 0.334569045412419 [raw: humanity's last exam (artificial analysis independent run)]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":53.2,"score_norm":75.29,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37,"score_norm":51.47,"config":"highest reasoning effort, humanity's last exam, hy3-preview=30.0) [raw: hle (no tools, no tools, official hy3 model-card benchmark appendix (hy3 column, text-only, text-only)]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"long_horizon_terminal_bench","score_raw":28.8,"score_norm":null,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":47.57,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.475694444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":75.8,"score_norm":null,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":57.9,"score_norm":88.17,"config":"official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78,"score_norm":75.56,"config":"resolved; official hy3 model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":22.89,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.228865979381443 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":64.42,"score_norm":69.91,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.644194756554307 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":64.4,"score_norm":69.88,"config":"299b total / 21b active), aa independent run for model slug 'hy3' (tencent, page data field terminalbenchv21 = 0.644194756554307 [raw: terminal-bench v2.1 (artificial analysis independent run)], released 2026-07-06, terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hy3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":71.7,"score_norm":78.47,"config":"16 cores/32gb, 4h agent timeout, max 500 episodes, not hy3-preview=58.0), official hy3 model-card benchmark appendix (hy3 column, reasoning effort [raw: terminal-bench 2.1], scaffold terminus-2, terminal-bench 2.1, xml parser","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"wildclawbench","score_raw":53.6,"score_norm":null,"config":"overall","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy3","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#tencent/hy3","publisher":"OpenRouter (listed 2026-07-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hy-4","display_name":"Hy4 preview","family":{"id":"hy","vendor":"tencent"},"generation":"4","size_class":"frontier","params_total_b":779.96,"params_active_b":null,"params_exact":779960992733,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1560,"int8_gb":827,"int4_gb":468,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.67,"expected_int8_gb":557,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"tencent/Hy4-preview","openrouter_id":"tencent/hy4-preview","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-28","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["hy4-preview"],"intelligence":{"index":82.9,"inputs":[{"benchmark":"gpqa_diamond","score_raw":92.3,"score_norm":94.79,"trust":1,"source_url":"https://huggingface.co/tencent/Hy4-preview","config":"highest available reasoning setting, tencent benchmark appendix"},{"benchmark":"hle","score_raw":43.4,"score_norm":60.88,"trust":1,"source_url":"https://huggingface.co/tencent/Hy4-preview","config":"no tools, text-only"},{"benchmark":"terminal_bench","score_raw":85.4,"score_norm":94.59,"trust":1,"source_url":"https://huggingface.co/tencent/Hy4-preview","config":"12h timeout, 500 turns, claude code harness, v2.1"}],"coverage":0.3174,"subset_bias":{"offset":-0.05,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.834,"completion_usd_per_m":2.501,"cache_read_usd_per_m":0.042,"context_length":1048576,"source":"openrouter","source_model_id":"tencent/hy4-preview","as_of":"2026-10-06","blended_usd_per_m":1.25075,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1559.92,"kv_cache_gb_per_8k":null,"min_serve_gb":1641.916,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1559.984,"kv_cache_gb_per_8k":null,"min_serve_gb":1641.983,"method":"vendor","source_url":"https://huggingface.co/tencent/Hy4-preview/tree/main"},{"quant":"fp8","weights_gb":826.758,"kv_cache_gb_per_8k":null,"min_serve_gb":872.096,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":467.976,"kv_cache_gb_per_8k":null,"min_serve_gb":495.375,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":826.758,"kv_cache_gb_per_8k":null,"min_serve_gb":872.096,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"apex_agents","score_raw":37.1,"score_norm":null,"config":"250 steps, official react toolbelt harness, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1633.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arxivmath","score_raw":66.6,"score_norm":null,"config":"highest available reasoning setting","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":32.1,"score_norm":null,"config":"v1.0.6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":16.9,"score_norm":null,"config":"official","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":64.3,"score_norm":null,"config":"8 cpu / 16gb sandbox, mini-swe-agent","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.3,"score_norm":94.79,"config":"highest available reasoning setting, tencent benchmark appendix","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":55.4,"score_norm":78.53,"config":"text-only, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":43.4,"score_norm":60.88,"config":"no tools, text-only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"matharena_apex_2025","score_raw":74.2,"score_norm":null,"config":"highest available reasoning setting","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":83.7,"score_norm":null,"config":"500-task set, public, scale april 2026 methodology","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"nl2repo_bench","score_raw":58.9,"score_norm":null,"config":"1000-turn budget, claude code","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"superchem","score_raw":66.4,"score_norm":null,"config":"highest available reasoning setting","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":82.9,"score_norm":null,"config":"swe-agent scaffold","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":65.7,"score_norm":100,"config":"swe-agent scaffold","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_marathon","score_raw":31.9,"score_norm":null,"config":"2x agent timeout, official settings","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.4,"score_norm":94.59,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.4,"score_norm":94.59,"config":"12h timeout, 500 turns, claude code harness, v2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":74.1,"score_norm":null,"config":"108-task verified set, 3-run pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"widesearch","score_raw":83.9,"score_norm":null,"config":"internal harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/tencent/Hy4-preview","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#tencent/hy4-preview","publisher":"OpenRouter (listed 2026-08-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"inkling-1","display_name":"Inkling","family":{"id":"inkling","vendor":"thinkingmachines"},"generation":"1","size_class":"frontier","params_total_b":952.38,"params_active_b":null,"params_exact":952377623626,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1905,"int8_gb":1010,"int4_gb":571,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.44,"expected_int8_gb":447,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"thinkingmachines/Inkling","openrouter_id":"thinkingmachines/inkling","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":524288,"release_date":"2026-07-17","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["inkling"],"intelligence":{"index":78.25,"inputs":[{"benchmark":"aime","score_raw":97.1,"score_norm":96.37,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling","config":"aime 2026"},{"benchmark":"arena_elo","score_raw":1441.4,"score_norm":72.1,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":87.2,"score_norm":87.61,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling","config":null},{"benchmark":"hle","score_raw":46,"score_norm":64.71,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling","config":null},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling","config":null},{"benchmark":"terminal_bench","score_raw":63.8,"score_norm":69.18,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling","config":"best harness, v2.1"}],"coverage":0.6349,"subset_bias":{"offset":-0.24,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.95,"completion_usd_per_m":4.05,"cache_read_usd_per_m":0.16,"context_length":524288,"source":"openrouter","source_model_id":"thinkingmachines/inkling","as_of":"2026-10-06","blended_usd_per_m":1.7249999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1904.76,"kv_cache_gb_per_8k":2.2146,"min_serve_gb":2002.213,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1904.755,"kv_cache_gb_per_8k":2.2146,"min_serve_gb":2002.207,"method":"vendor","source_url":"https://huggingface.co/thinkingmachines/Inkling/tree/main"},{"quant":"fp8","weights_gb":1009.523,"kv_cache_gb_per_8k":2.2146,"min_serve_gb":1062.214,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":571.428,"kv_cache_gb_per_8k":2.2146,"min_serve_gb":602.214,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1009.523,"kv_cache_gb_per_8k":2.2146,"min_serve_gb":1062.214,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":22.5,"score_norm":22.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":52.1,"score_norm":52.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25,"score_norm":25,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":97.1,"score_norm":96.37,"config":"aime 2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.4,"score_norm":72.1,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1412.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":77.1,"score_norm":null,"config":"with context","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"charxiv_rq","score_raw":78.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1108,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1209,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1173,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1113,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1205,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1228,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1233,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":88.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.2,"score_norm":87.61,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":46,"score_norm":64.71,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":46,"score_norm":64.71,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":29.7,"score_norm":40.74,"config":"no tools, text only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":79.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":74.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmau","score_raw":77.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.5,"score_norm":null,"config":"standard 10","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":43.9,"score_norm":null,"config":"verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":54.3,"score_norm":82.17,"config":"public split","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":23.7,"score_norm":null,"config":"tau3/banking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.8,"score_norm":69.18,"config":"2.1, best harness. effort=0.99","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.8,"score_norm":69.18,"config":"best harness, v2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"voicebench","score_raw":91.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling","publisher":"OpenRouter (listed 2026-07-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"inkling-1-small","display_name":"Inkling Small","family":{"id":"inkling","vendor":"thinkingmachines"},"generation":"1","size_class":"frontier-small","params_total_b":265.96,"params_active_b":null,"params_exact":265956439090,"params_source":"hf_safetensors","vram_est":{"bf16_gb":532,"int8_gb":282,"int4_gb":160,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.14,"expected_int8_gb":321,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"thinkingmachines/Inkling-Small","openrouter_id":"thinkingmachines/inkling-small","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":524288,"release_date":"2026-07-30","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["inkling-small"],"intelligence":{"index":69.65,"inputs":[{"benchmark":"aime","score_raw":95.5,"score_norm":94.38,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","config":"aime 2026"},{"benchmark":"arc_agi_2","score_raw":40.1,"score_norm":39,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","config":"reasoning (abstract)"},{"benchmark":"arena_elo","score_raw":1405.2,"score_norm":55.97,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.49,"score_norm":90.83,"trust":2,"source_url":"https://artificialanalysis.ai/models/inkling-small","config":"artificial analysis harness"},{"benchmark":"hle","score_raw":33.32,"score_norm":46.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/inkling-small","config":"artificial analysis harness, no tools"},{"benchmark":"swe_bench_verified","score_raw":80.2,"score_norm":78,"trust":1,"source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","config":"bash-only harness"},{"benchmark":"terminal_bench","score_raw":55.06,"score_norm":58.89,"trust":2,"source_url":"https://artificialanalysis.ai/models/inkling-small","config":"artificial analysis harness, v2.1"}],"coverage":0.7142,"subset_bias":{"offset":-2.06,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.45,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.1,"context_length":524288,"source":"openrouter","source_model_id":"thinkingmachines/inkling-small","as_of":"2026-10-06","blended_usd_per_m":0.6375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":531.92,"kv_cache_gb_per_8k":1.4093,"min_serve_gb":559.925,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":531.913,"kv_cache_gb_per_8k":1.4093,"min_serve_gb":559.918,"method":"vendor","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small/tree/main"},{"quant":"fp8","weights_gb":281.918,"kv_cache_gb_per_8k":1.4093,"min_serve_gb":297.423,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":159.576,"kv_cache_gb_per_8k":1.4093,"min_serve_gb":168.964,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":281.918,"kv_cache_gb_per_8k":1.4093,"min_serve_gb":297.423,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":23.5,"score_norm":23.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling-small","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":52.9,"score_norm":52.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.7,"score_norm":25.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling-small","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75.67,"score_norm":null,"config":"artificial analysis harness, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.5,"score_norm":94.38,"config":"aime 2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":84,"score_norm":null,"config":"reasoning (abstract)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":40.1,"score_norm":39,"config":"reasoning (abstract)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1405.2,"score_norm":55.97,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1412.2,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1206.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1408.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":77.4,"score_norm":null,"config":"with context management","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8.29,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8.3,"score_norm":null,"config":"reasoning (general)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":86.7,"score_norm":null,"config":"chat","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.49,"score_norm":90.83,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.5,"score_norm":90.85,"config":"reasoning (general)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.32,"score_norm":46.06,"config":"artificial analysis harness, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":47.8,"score_norm":67.35,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":31.6,"score_norm":43.53,"config":"no tools, text only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt_feb_2026","score_raw":90.2,"score_norm":null,"config":"reasoning (general)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":82.2,"score_norm":null,"config":"chat","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":79.6,"score_norm":null,"config":"public","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74,"score_norm":null,"config":"standard 10, vision","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":48.7,"score_norm":null,"config":"agentic coding","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":20.6,"score_norm":null,"config":"factuality, verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":55.9,"score_norm":84.83,"config":"public","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.2,"score_norm":78,"config":"bash-only harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":18.76,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":15.5,"score_norm":null,"config":"tau 3 banking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.06,"score_norm":58.89,"config":"artificial analysis harness, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/inkling-small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":64.7,"score_norm":70.24,"config":"best harness (internal coding harness), v2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":54.4,"score_norm":null,"config":"agentic (general)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/thinkingmachines/Inkling-Small","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#thinkingmachines/inkling-small","publisher":"OpenRouter (listed 2026-07-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"jamba-1.5-mini-52b","display_name":"AI21-Jamba-Mini-1.5","family":{"id":"jamba","vendor":"AI21 Labs"},"generation":"1.5","size_class":"52b","params_total_b":51.57,"params_active_b":12,"params_exact":51570323328,"params_source":"hf_safetensors","vram_est":{"bf16_gb":103,"int8_gb":54.7,"int4_gb":30.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"ai21labs/AI21-Jamba-Mini-1.5","openrouter_id":null,"is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2024-08-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ai21-jamba-mini-1.5"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":103.141,"kv_cache_gb_per_8k":null,"min_serve_gb":108.778,"method":"vendor","source_url":"https://huggingface.co/ai21labs/AI21-Jamba-Mini-1.5/tree/main"},{"quant":"fp16","weights_gb":103.14,"kv_cache_gb_per_8k":null,"min_serve_gb":108.777,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":54.664,"kv_cache_gb_per_8k":null,"min_serve_gb":57.877,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":30.942,"kv_cache_gb_per_8k":null,"min_serve_gb":32.969,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":54.664,"kv_cache_gb_per_8k":null,"min_serve_gb":57.877,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":12.71,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.75,"bw_limited_tok_s":12.87,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":143.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":279.17,"bw_limited_tok_s":167.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/ai21labs/AI21-Jamba-Mini-1.5/blob/main/README.md","publisher":"Hugging Face"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"kat-coder-2-pro","display_name":"kat-coder-pro-v2","family":{"id":"kat-coder","vendor":"kwaipilot"},"generation":"2","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"kwaipilot/kat-coder-pro-v2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-03-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["kat-coder-pro-v2"],"intelligence":{"index":65.42,"inputs":[{"benchmark":"gpqa_diamond","score_raw":85.45,"score_norm":85.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","config":"artificial analysis independent run"},{"benchmark":"hle","score_raw":16.13,"score_norm":20.78,"trust":2,"source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","config":"artificial analysis independent run, no tools"},{"benchmark":"tau_bench","score_raw":89.47,"score_norm":92.63,"trust":2,"source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","config":"artificial analysis independent run, tau2, with tools"},{"benchmark":"terminal_bench","score_raw":70.04,"score_norm":76.52,"trust":2,"source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","config":"2.1, artificial analysis independent run"}],"coverage":0.3651,"subset_bias":{"offset":-1.46,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":59.5,"score_norm":59.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#kwaipilot/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"artificial analysis aa-lcr v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.45,"score_norm":85.14,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":16.13,"score_norm":20.78,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":66.67,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":89.47,"score_norm":92.63,"config":"artificial analysis independent run, tau2, with tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.04,"score_norm":76.52,"config":"2.1, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kat-coder-pro-v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#kwaipilot/kat-coder-pro-v2","publisher":"OpenRouter (listed 2026-03-27)"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"kat-coder-2.5-pro","display_name":"KAT-Coder-Pro V2.5","family":{"id":"kat-coder","vendor":"kwaipilot"},"generation":"2.5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"kwaipilot/kat-coder-pro-v2.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-07-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["kat-coder-pro-v2.5"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.74,"completion_usd_per_m":2.96,"cache_read_usd_per_m":0.15,"context_length":262144,"source":"openrouter","source_model_id":"kwaipilot/kat-coder-pro-v2.5","as_of":"2026-10-06","blended_usd_per_m":1.295,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"kat_claw_bench","score_raw":85.5,"score_norm":null,"config":"internal benchmark","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-06","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"kat_claw_bench","score_raw":85.5,"score_norm":null,"config":"kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"kat_code_bench","score_raw":53.1,"score_norm":null,"config":"internal benchmark","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-06","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"kat_code_bench","score_raw":53.1,"score_norm":null,"config":"kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"pinchbench","score_raw":94.9,"score_norm":null,"config":"average","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"pinchbench","score_raw":94.9,"score_norm":null,"config":"kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":50.3,"score_norm":null,"config":"default","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-06","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":50.3,"score_norm":null,"config":"kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":65.2,"score_norm":100,"config":"with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-06","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":65.2,"score_norm":100,"config":"kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.7,"score_norm":65.53,"config":"terminal-bench 2.1","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.05471","measured_date":"2026-07-06","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.7,"score_norm":65.53,"config":"2.1, kat-coder-v2.5 technical report, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2607.05471v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#kwaipilot/kat-coder-pro-v2.5","publisher":"OpenRouter (listed 2026-07-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2-0905","display_name":"Kimi K2 0905","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2","size_class":"frontier-k","params_total_b":1026.47,"params_active_b":32,"params_exact":1026470735448,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2053,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.14,"expected_int8_gb":154,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"moonshotai/Kimi-K2-Instruct-0905","openrouter_id":"moonshotai/kimi-k2-0905","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-09-09","release_status":"released","knowledge_cutoff":"2024-12-31","status":"current","supersedes":["kimi-k2"],"superseded_by":null,"aliases":["kimi-k2-0905-1t-instruct","kimi-k2-instruct-0905"],"intelligence":{"index":45.91,"inputs":[{"benchmark":"aime","score_raw":57.33,"score_norm":46.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.573333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":76.67,"score_norm":72.77,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.39,"score_norm":6.46,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0639481000926784 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":73.39,"score_norm":71.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.733918128654971 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":23.48,"score_norm":21.74,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.234848484848485 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-2.36,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":2.5,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"moonshotai/kimi-k2-0905","as_of":"2026-10-06","blended_usd_per_m":1.075,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2052.94,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2156.163,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1088.058,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.037,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1029.191,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1081.226,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905/tree/main"},{"quant":"int4","weights_gb":615.882,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.252,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1088.058,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.037,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":15.3,"score_norm":15.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-0905","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23.96,"score_norm":23.96,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":53.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.536666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":57.33,"score_norm":46.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.573333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1104,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-0905","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1114,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-0905","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.67,"score_norm":72.77,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.39,"score_norm":6.46,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0639481000926784 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.7,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.417006802721088 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":60.95,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.60952380952381 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":30.67,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.306712962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":73.39,"score_norm":71.19,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.733918128654971 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":23.48,"score_norm":21.74,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.234848484848485 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-0905","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-0905","publisher":"OpenRouter (listed 2025-09-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2-reasoning","display_name":"Kimi K2 Thinking","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2","size_class":"frontier-k","params_total_b":1026.41,"params_active_b":32,"params_exact":1026408240256,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2053,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.24,"expected_int8_gb":257,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"moonshotai/Kimi-K2-Thinking","openrouter_id":"moonshotai/kimi-k2-thinking","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-11-06","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["kimi-k2-1t-reasoning","kimi-k2-thinking"],"intelligence":{"index":62.84,"inputs":[{"benchmark":"aime","score_raw":85.42,"score_norm":81.77,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.327, task=overall, updated=2026-04-16"},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.183, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":23.82,"score_norm":32.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.238183503243744 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":81.07,"score_norm":68.45,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.396, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":60.2,"score_norm":55.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.191, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":92.98,"score_norm":97.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.929824561403509 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":35.7,"score_norm":36.12,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.7223,"subset_bias":{"offset":3.62,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.15,"context_length":262144,"source":"openrouter","source_model_id":"moonshotai/kimi-k2-thinking","as_of":"2026-10-06","blended_usd_per_m":1.075,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2052.82,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2156.037,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1087.995,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1142.97,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":615.846,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.214,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":594.233,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":624.52,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Thinking/tree/main"},{"quant":"int8","weights_gb":1087.995,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1142.97,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":21,"score_norm":21,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-thinking","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22,"score_norm":22,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.49,"score_norm":33.49,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.703333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":85.42,"score_norm":81.77,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.327, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":94.67,"score_norm":93.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.946666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":2.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0257142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1120,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-thinking","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.183, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.84,"score_norm":82.87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.838383838383838 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.5,"score_norm":83.8,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Thinking","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.82,"score_norm":32.09,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.238183503243744 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.9,"score_norm":32.21,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Thinking","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":44.9,"score_norm":63.09,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Thinking","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":68.1,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.680952380952381 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":85.29,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.852910052910053 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.07,"score_norm":68.45,"config":"edition=mmlu pro v1, runner=platform, stderr=0.396, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":42.36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.423611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":60.2,"score_norm":55.78,"config":"edition=swe-bench verified v1, runner=external, stderr=2.191, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":71.3,"score_norm":68.11,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Thinking","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.98,"score_norm":97.31,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.929824561403509 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":35.7,"score_norm":36.12,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2025-11-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2-thinking","publisher":"OpenRouter (listed 2025-11-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2.7-code","display_name":"Kimi K2.7 Code","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2.7","size_class":"frontier-k","params_total_b":1026.88,"params_active_b":32,"params_exact":1026879376368,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2054,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.34,"expected_int8_gb":371,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"moonshotai/Kimi-K2.7-Code","openrouter_id":"moonshotai/kimi-k2.7-code","is_moe":true,"variant_axis":"code","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-06-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":73.59,"inputs":[{"benchmark":"gpqa_diamond","score_raw":89.6,"score_norm":90.99,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.895959595959596 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":35.03,"score_norm":48.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.35032437442076 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":68.41,"score_norm":63.13,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.848, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":90.06,"score_norm":93.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.900584795321637 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":67.04,"score_norm":72.99,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=0.375, task=overall, updated=2026-08-19"}],"coverage":0.643,"subset_bias":{"offset":-1.32,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.6712,"completion_usd_per_m":3.35,"cache_read_usd_per_m":0.18,"context_length":262144,"source":"openrouter","source_model_id":"moonshotai/kimi-k2.7-code","as_of":"2026-10-06","blended_usd_per_m":1.3409,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2053.76,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2157.024,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":616.128,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.51,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":595.178,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":625.513,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2.7-Code/tree/main"},{"quant":"int8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":21,"score_norm":21,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":30.28,"score_norm":30.28,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":60.8,"score_norm":60.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.8,"score_norm":25.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":43.02,"score_norm":43.02,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.75 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1510.8,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1472.9,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1472.7,"score_norm":null,"config":"4680 votes, arena entry kimi-k2.7-code, code/overall, leaderboard webdev-overall-raw, rank 41, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":22.53,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.22528586634678724 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":10,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.1 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1256,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1130,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1152,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1210,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1259,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1232,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1174,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1233,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1147,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1201,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1140,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1154,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1175,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1268,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1169,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":40.23,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.40226797970754996 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":34.49,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.34494500000000006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.6,"score_norm":90.99,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.895959595959596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":85.02,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.850195397307859 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":35.03,"score_norm":48.57,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.35032437442076 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":63.13,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.631292517006803 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":68.41,"score_norm":63.13,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":16.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.161111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":47.45,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.474537037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"config":"edition=swe-bench verified v1, runner=external, stderr=1.848, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":90.06,"score_norm":93.41,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.900584795321637 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":20.21,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.202061855670103 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.04,"score_norm":72.99,"config":"edition=terminal-bench 2.1, runner=external, stderr=0.375, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.7,"score_norm":46.71,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.446969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.42,"score_norm":73.44,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.674157303370786 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-7-code","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.kimi.ai/docs/models","publisher":"Moonshot AI"},{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.7-code","publisher":"OpenRouter (listed 2026-06-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k3","display_name":"Kimi K3","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"3","size_class":"frontier-k","params_total_b":2779.93,"params_active_b":104,"params_exact":2779931837184,"params_source":"hf_safetensors","vram_est":{"bf16_gb":5560,"int8_gb":2947,"int4_gb":1668,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.2,"expected_int8_gb":604,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"moonshotai/Kimi-K3","openrouter_id":"moonshotai/kimi-k3","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-07-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["kimi-k2.5-0127","kimi-k2.6"],"superseded_by":null,"aliases":["kimi-k3-2.8t","kimi-k3-2.8t-vision"],"intelligence":{"index":84.38,"inputs":[{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"trust":3,"source_url":"https://arcprize.org/results/moonshot-kimi-k3","config":"cost_per_task_usd=1.5926940250000003, eval=arc-agi-2 semi-private (v2_semi_private), model_group=moonshot-kimi-k3, org=moonshot ai, run: max, setting=evaluation conducted with 'max' reasoning effort via baseten., system_type=cot"},{"benchmark":"arena_elo","score_raw":1489,"score_norm":93.32,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"arena.ai text leaderboard, kimi-k3-max (+/-6)"},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.309, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":46.9,"score_norm":66.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.468952734012975 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":79.19,"score_norm":87.09,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":87.97,"score_norm":79.95,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.321, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":93.4,"score_norm":92.67,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.111, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":80.9,"score_norm":89.29,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=0.649, task=overall, updated=2026-08-19"}],"coverage":0.873,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.95,"completion_usd_per_m":14,"cache_read_usd_per_m":0.31,"context_length":1048576,"source":"openrouter","source_model_id":"moonshotai/kimi-k3","as_of":"2026-10-06","blended_usd_per_m":4.2125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":5559.86,"kv_cache_gb_per_8k":0.2265,"min_serve_gb":5838.08,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":2946.726,"kv_cache_gb_per_8k":0.2265,"min_serve_gb":3094.289,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1667.958,"kv_cache_gb_per_8k":0.2265,"min_serve_gb":1751.582,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1560.936,"kv_cache_gb_per_8k":0.2265,"min_serve_gb":1639.209,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K3/tree/main"},{"quant":"int8","weights_gb":2946.726,"kv_cache_gb_per_8k":0.2265,"min_serve_gb":3094.289,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":50,"score_norm":50,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":54.26,"score_norm":54.26,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.2,"score_norm":76.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":43.6,"score_norm":43.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":59.7,"score_norm":59.7,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":82.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.826666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.7,"score_norm":null,"config":"reasoning_effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":19.7,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":38.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.3875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":41.3,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.412979351032448 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":94.5,"score_norm":null,"config":"cost_per_task_usd=0.7704815999999995, eval=arc-agi-1 semi-private (v1_semi_private), model_group=moonshot-kimi-k3, org=moonshot ai, run: max, setting=evaluation conducted with 'max' reasoning effort via baseten., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/moonshot-kimi-k3","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"cost_per_task_usd=1.5926940250000003, eval=arc-agi-2 semi-private (v2_semi_private), model_group=moonshot-kimi-k3, org=moonshot ai, run: max, setting=evaluation conducted with 'max' reasoning effort via baseten., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/moonshot-kimi-k3","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"arc prize leaderboard, moonshot-kimi-k3-max, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/media/data/evaluations.json","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1489,"score_norm":93.32,"config":"arena.ai text leaderboard, kimi-k3-max (+/-6)","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":52.71,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.5270588443066796 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":91.2,"score_norm":null,"config":"context-compaction at 300k tokens (90.4 with full 1m context, no management)","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":23.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.234285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":67.5,"score_norm":null,"config":"kimi code harness (67.3 with mini-swe-agent)","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1406,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1252,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1262,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1371,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1354,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1313,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1379,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1223,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1256,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1263,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1273,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1362,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1285,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1344,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":45.33,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.45329752312742466 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":81.2,"score_norm":null,"config":"kimi code harness","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":58.91,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.58912 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.309, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.54,"score_norm":96.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.935353535353535 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.5,"score_norm":96.48,"config":"reasoning_effort=max, temp=1.0, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.5,"score_norm":96.48,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":94.64,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.946446663771892 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":46.9,"score_norm":66.03,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.468952734012975 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.5,"score_norm":61.03,"config":"hle-full, no tools, reasoning_effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":56,"score_norm":79.41,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"it_bench_sre","score_raw":47.69,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, reasoning=true, run: max, source value 0.476930320150659 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":79.19,"score_norm":87.09,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":38.33,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.383333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.97,"score_norm":79.95,"config":"edition=mmlu pro v1, reasoning_effort=max, runner=platform, stderr=0.321, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.973,"score_norm":79.96,"config":"kimi-k3, reasoning effort max, temp 1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.52,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, mmmu-pro, reasoning=true, run: max, source value 0.805202312138728 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":81.6,"score_norm":null,"config":"no tools (83.4 with python tool), avg of 3 runs","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":84.8,"score_norm":null,"config":"reasoning_effort=max, verified","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":77.8,"score_norm":null,"config":"reasoning_effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":58.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, scicode, source value 0.586805555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":58.7,"score_norm":null,"config":"reasoning_effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":93.4,"score_norm":92.67,"config":"edition=swe-bench verified v1, runner=external, stderr=1.111, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":93.4,"score_norm":92.67,"config":"kimi-k3, max output 262144, temp 1, vals harness","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_marathon","score_raw":42,"score_norm":null,"config":"claude code harness, h20-calibrated tasks","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":45.98,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.45979381443299 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.9,"score_norm":89.29,"config":"edition=terminal-bench 2.1, runner=external, stderr=0.649, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.02,"score_norm":94.14,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.850187265917603 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.3,"score_norm":98,"config":"terminal-bench 2.1, kimi code harness, reasoning_effort=max","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/MoonshotAI/Kimi-K3/blob/main/README.md","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.kimi.ai/docs/models","publisher":"Moonshot AI"},{"url":"https://www.moonshot.ai/","publisher":"Moonshot AI"},{"url":"https://huggingface.co/moonshotai/Kimi-K3","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k3","publisher":"OpenRouter (listed 2026-07-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"l-3-lunaris-8b","display_name":"Sao10K Llama 3 8B Lunaris","family":{"id":"l","vendor":"sao10k"},"generation":"3","size_class":"8b","params_total_b":8.03,"params_active_b":8.03,"params_exact":8030261248,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.1,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Sao10K/L3-8B-Lunaris-v1","openrouter_id":"sao10k/l3-lunaris-8b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2024-08-13","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["l3-8b-lunaris-v1","l3-lunaris-8b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.04,"completion_usd_per_m":0.05,"cache_read_usd_per_m":null,"context_length":8192,"source":"openrouter","source_model_id":"sao10k/l3-lunaris-8b","as_of":"2026-10-06","blended_usd_per_m":0.042499999999999996,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":16.06,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":18.134,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.061,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":18.135,"method":"vendor","source_url":"https://huggingface.co/Sao10K/L3-8B-Lunaris-v1/tree/main"},{"quant":"fp8","weights_gb":8.512,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":10.586,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.818,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.892,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.512,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":10.586,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.72,"bw_limited_tok_s":45.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.99,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34,"bw_limited_tok_s":20.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":417.19,"bw_limited_tok_s":250.31,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sao10k/l3-lunaris-8b","publisher":"OpenRouter (listed 2024-08-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"l-3.1-euryale-70b","display_name":"Sao10K Llama 3.1 Euryale 70B v2.2","family":{"id":"l","vendor":"sao10k"},"generation":"3.1","size_class":"70b","params_total_b":70,"params_active_b":70,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":140,"int8_gb":74.2,"int4_gb":42,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Sao10K/L3.1-70B-Euryale-v2.2","openrouter_id":"sao10k/l3.1-euryale-70b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"cc-by-nc-4.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-08-28","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["l3.1-70b-euryale-v2.2","l3.1-euryale-70b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.85,"completion_usd_per_m":0.85,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"sao10k/l3.1-euryale-70b","as_of":"2026-10-06","blended_usd_per_m":0.85,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":140,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":149.684,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":74.2,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":80.594,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":46.784,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.2,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":80.594,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.9,"bw_limited_tok_s":2.34,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":54.31,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":95.71,"bw_limited_tok_s":57.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sao10k/l3.1-euryale-70b","publisher":"OpenRouter (listed 2024-08-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"l-3.3-euryale-70b-v2.3","display_name":"Sao10K Llama 3.3 Euryale 70B","family":{"id":"l","vendor":"sao10k"},"generation":"3.3","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Sao10K/L3.3-70B-Euryale-v2.3","openrouter_id":"sao10k/l3.3-euryale-70b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-12-18","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["l3.3-70b-euryale-v2.3","l3.3-euryale-70b","l3.3-euryale-70b-v2.3"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.65,"completion_usd_per_m":0.75,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"sao10k/l3.3-euryale-70b","as_of":"2026-10-06","blended_usd_per_m":0.675,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.839,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.847,"method":"vendor","source_url":"https://huggingface.co/Sao10K/L3.3-70B-Euryale-v2.3/tree/main"},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":47.131,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.87,"bw_limited_tok_s":2.32,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":53.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70.55,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":94.97,"bw_limited_tok_s":56.98,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sao10k/l3.3-euryale-70b","publisher":"OpenRouter (listed 2024-12-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"laguna-s-2.1","display_name":"Laguna S 2.1","family":{"id":"laguna","vendor":"poolside"},"generation":"2.1","size_class":"frontier-s","params_total_b":117.56,"params_active_b":null,"params_exact":117561977600,"params_source":"hf_safetensors","vram_est":{"bf16_gb":235,"int8_gb":125,"int4_gb":70.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"poolside/Laguna-S-2.1","openrouter_id":"poolside/laguna-s-2.1","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"openmdw-1.1","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-07-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.09,"completion_usd_per_m":0.18,"cache_read_usd_per_m":0.009,"context_length":1048576,"source":"openrouter","source_model_id":"poolside/laguna-s-2.1","as_of":"2026-10-06","blended_usd_per_m":0.1125,"endpoint_expires_at":"2026-10-31"},"memory":[{"quant":"fp16","weights_gb":235.12,"kv_cache_gb_per_8k":0.4782,"min_serve_gb":247.354,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":235.129,"kv_cache_gb_per_8k":0.4782,"min_serve_gb":247.364,"method":"vendor","source_url":"https://huggingface.co/poolside/Laguna-S-2.1/tree/main"},{"quant":"fp8","weights_gb":124.614,"kv_cache_gb_per_8k":0.4782,"min_serve_gb":131.323,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":70.536,"kv_cache_gb_per_8k":0.4782,"min_serve_gb":74.541,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":124.614,"kv_cache_gb_per_8k":0.4782,"min_serve_gb":131.323,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":117.56,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.64,"bw_limited_tok_s":2.63,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":33.06,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":117.56,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":56.99,"bw_limited_tok_s":34.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"deepswe","score_raw":40.4,"score_norm":null,"config":"1.1, max thinking, pass@1 avg of 3","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_atlas","score_raw":46.2,"score_norm":null,"config":"codebase qna, pass@1 avg of 3","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":78.5,"score_norm":null,"config":"pass@1 avg of 4","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":59.4,"score_norm":90.67,"config":"pass@1 avg of 4, public dataset","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":59.4,"score_norm":90.67,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-S-2.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.2,"score_norm":76.71,"config":"2.1, max thinking, pass@1 avg of 4","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.2,"score_norm":76.71,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-S-2.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":49.7,"score_norm":null,"config":"avg of 3 runs","trust":1,"source_type":"vendor_claim","source_url":"https://poolside.ai/blog/introducing-laguna-s-2-1","measured_date":"2026-07-21","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#poolside/laguna-s-2.1","publisher":"OpenRouter (listed 2026-07-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"laguna-xs-2.1","display_name":"Laguna XS 2.1","family":{"id":"laguna","vendor":"poolside"},"generation":"2.1","size_class":"frontier-xs","params_total_b":33.44,"params_active_b":null,"params_exact":33442617088,"params_source":"hf_safetensors","vram_est":{"bf16_gb":66.9,"int8_gb":35.4,"int4_gb":20.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"poolside/Laguna-XS-2.1","openrouter_id":"poolside/laguna-xs-2.1","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"openmdw-1.1","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-07-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.06,"completion_usd_per_m":0.12,"cache_read_usd_per_m":0.03,"context_length":262144,"source":"openrouter","source_model_id":"poolside/laguna-xs-2.1","as_of":"2026-10-06","blended_usd_per_m":0.075,"endpoint_expires_at":"2026-10-31"},"memory":[{"quant":"fp16","weights_gb":66.88,"kv_cache_gb_per_8k":0.3985,"min_serve_gb":70.622,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":66.889,"kv_cache_gb_per_8k":0.3985,"min_serve_gb":70.632,"method":"vendor","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1/tree/main"},{"quant":"fp8","weights_gb":35.446,"kv_cache_gb_per_8k":0.3985,"min_serve_gb":37.617,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":20.064,"kv_cache_gb_per_8k":0.3985,"min_serve_gb":21.466,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":35.446,"kv_cache_gb_per_8k":0.3985,"min_serve_gb":37.617,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.35,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":33.44,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":36.36,"bw_limited_tok_s":21.82,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.6,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":33.44,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.16,"bw_limited_tok_s":4.62,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":56.7,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":33.44,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":100.18,"bw_limited_tok_s":60.11,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"swe_bench_multilingual","score_raw":63.1,"score_norm":null,"config":"harbor framework, mean pass@1 over 4 attempts","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":47.6,"score_norm":71,"config":"mean pass@1 over 2 attempts, public dataset","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70.9,"score_norm":67.67,"config":"harbor framework + pool agent harness, mean pass@1 over 4 attempts, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.5,"score_norm":38.24,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.5,"score_norm":38.24,"config":"48gb ram / 32 cpu, mean pass@1 over 5 attempts, v2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/poolside/Laguna-XS-2.1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#poolside/laguna-xs-2.1","publisher":"OpenRouter (listed 2026-07-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"lfm-2.5-2.6b","display_name":"LFM2.5-2.6B","family":{"id":"lfm","vendor":"liquid"},"generation":"2.5","size_class":"2.6b","params_total_b":2.7,"params_active_b":2.7,"params_exact":2697198592,"params_source":"hf_safetensors","vram_est":{"bf16_gb":5.4,"int8_gb":2.9,"int4_gb":1.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":5.83,"expected_int8_gb":16.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"LiquidAI/LFM2.5-2.6B","openrouter_id":"liquid/lfm-2.5-2.6b:free","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2026-08-11","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["lfm2.5-2.6b"],"intelligence":{"index":23.83,"inputs":[{"benchmark":"gpqa_diamond","score_raw":55.76,"score_norm":43.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.557575757575758 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.21,"score_norm":6.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.06209453197405 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":4.49,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3174,"subset_bias":{"offset":-6.34,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":65536,"source":"openrouter","source_model_id":"liquid/lfm-2.5-2.6b:free","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":5.4,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":6.534,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":5.394,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":6.528,"method":"vendor","source_url":"https://huggingface.co/LiquidAI/LFM2.5-2.6B/tree/main"},{"quant":"fp8","weights_gb":2.862,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":3.996,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.62,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":2.754,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":2.862,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":3.996,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":119.03,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":2.7,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":225.19,"bw_limited_tok_s":135.11,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":57.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":2.7,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":101.11,"bw_limited_tok_s":60.67,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":426.75,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":2.7,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1240.74,"bw_limited_tok_s":744.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":2.41,"score_norm":2.41,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":7.7,"score_norm":7.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#liquid/lfm-2.5-2.6b:free","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.4,"score_norm":8.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#liquid/lfm-2.5-2.6b:free","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.99,"score_norm":10.99,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":5.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.0533333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":55.76,"score_norm":43.32,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.557575757575758 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.21,"score_norm":6.19,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.06209453197405 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":14.24,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.142361111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":7.22,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0721649484536082 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.49,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/lfm2-5-2-6b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#liquid/lfm-2.5-2.6b:free","publisher":"OpenRouter (listed 2026-08-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ling-2.6-1t","display_name":"ling-2.6-1t","family":{"id":"ling","vendor":"inclusionAI"},"generation":"2.6","size_class":"frontier","params_total_b":1000,"params_active_b":1000,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":2000,"int8_gb":1060,"int4_gb":600,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ling-2.6-1t","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-04-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":49.86,"inputs":[{"benchmark":"gpqa_diamond","score_raw":75.15,"score_norm":70.63,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.751515151515151 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":8.71,"score_norm":9.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0871177015755329 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":89.77,"score_norm":93.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.89766081871345 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-6.12,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":26.57,"score_norm":26.57,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":38,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.38 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.15,"score_norm":70.63,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.751515151515151 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.71,"score_norm":9.87,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0871177015755329 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":56.87,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.568707482993197 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":37.04,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.37037037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":89.77,"score_norm":93.03,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.89766081871345 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-2.6-1t","publisher":"OpenRouter (listed 2026-04-23)"}],"last_verified":"2026-08-24","last_enriched_at":null},{"id":"ling-3.0-flash","display_name":"Ling 3.0 Flash","family":{"id":"ling","vendor":"inclusionAI"},"generation":"3.0","size_class":"frontier-flash","params_total_b":127.49,"params_active_b":null,"params_exact":127486405600,"params_source":"hf_safetensors","vram_est":{"bf16_gb":255,"int8_gb":135,"int4_gb":76.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.13,"expected_int8_gb":288,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"inclusionAI/Ling-3.0-flash","openrouter_id":"inclusionai/ling-3.0-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-07-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["ling-2.6-flash"],"superseded_by":null,"aliases":[],"intelligence":{"index":66.39,"inputs":[{"benchmark":"aime","score_raw":93.2,"score_norm":91.5,"trust":1,"source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash","config":"2026"},{"benchmark":"gpqa_diamond","score_raw":84.85,"score_norm":84.3,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.197, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":23.68,"score_norm":31.88,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.236793327154773 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.01,"score_norm":70.02,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.375, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":65.2,"score_norm":61.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.132, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=2.456, task=overall, updated=2026-08-19"}],"coverage":0.6746,"subset_bias":{"offset":2.08,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.021,"completion_usd_per_m":0.063,"cache_read_usd_per_m":0.0042,"context_length":262144,"source":"openrouter","source_model_id":"inclusionai/ling-3.0-flash","as_of":"2026-10-06","blended_usd_per_m":0.0315,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":254.98,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":268.125,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":254.981,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":268.126,"method":"vendor","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash/tree/main"},{"quant":"fp8","weights_gb":135.139,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":142.292,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":76.494,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":80.715,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":135.139,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":142.292,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.42,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":127.49,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.28,"bw_limited_tok_s":2.42,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":19.3,"score_norm":19.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":29.31,"score_norm":29.31,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":50.6,"score_norm":50.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.1,"score_norm":20.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.82,"score_norm":37.82,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.67 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":93.2,"score_norm":91.5,"config":"2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0171428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":30.36,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.30359 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.85,"score_norm":84.3,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.197, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.45,"score_norm":85.14,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.854545454545454 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.68,"score_norm":31.88,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.236793327154773 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.7,"score_norm":30.44,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.01,"score_norm":70.02,"config":"edition=mmlu pro v1, runner=platform, stderr=0.375, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":41.09,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.41087962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":56.6,"score_norm":86,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":65.2,"score_norm":61.33,"config":"edition=swe-bench verified v1, runner=external, stderr=2.132, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":27.22,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.272164948453608 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"config":"edition=terminal-bench 2.1, runner=external, stderr=2.456, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.43,"score_norm":59.33,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.554307116104869 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash","publisher":"OpenRouter (listed 2026-07-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ling-3.0-flash-fin","display_name":"Ling 3.0 Flash Fin","family":{"id":"ling","vendor":"inclusionAI"},"generation":"3.0","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ling-3.0-flash-fin","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-08-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.042,"completion_usd_per_m":0.1232,"cache_read_usd_per_m":0.0084,"context_length":262144,"source":"openrouter","source_model_id":"inclusionai/ling-3.0-flash-fin","as_of":"2026-10-06","blended_usd_per_m":0.0623,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":27.9,"score_norm":27.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-fin","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_briefcase","score_raw":967,"score_norm":null,"config":"agentic business workflows, elo","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_briefcase_analytical_quality_elo","score_raw":866,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_briefcase_presentation_elo","score_raw":1095,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_briefcase_rubric_pass_rate","score_raw":23.5,"score_norm":null,"config":"rubric checks","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":55.6,"score_norm":55.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-fin","measured_date":null,"observed_at":"2026-09-17","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":55.6,"score_norm":55.6,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/inclusionai/ling-3-0-flash-fin","measured_date":"2026-09-11","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":55.6,"score_norm":55.6,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/inclusionai/ling-3.0-flash-fin/","measured_date":"2026-08-27","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23,"score_norm":23,"config":"independently measured by artificial analysis, reasoning version, v4.3","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.6,"score_norm":22.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-fin","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":29.2,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":29.2,"score_norm":null,"config":"vendor scorecard / launch chart","trust":0,"source_type":"aggregator","source_url":"https://llm-stats.com/models/compare/ling-3.0-flash-vs-ling-3.0-flash-fin","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"artificial_analysis_finance_accounting_index","score_raw":24,"score_norm":null,"config":"agentic work, combines business knowledge, long-context analysis and non-hallucination, reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"automationbench_aa","score_raw":7,"score_norm":null,"config":"agentic saas workflows with guardrails, percent","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"business_knowledge_accuracy","score_raw":17,"score_norm":null,"config":"artificial analysis finance & accounting related knowledge split","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"business_knowledge_hallucination","score_raw":33,"score_norm":null,"config":"artificial analysis, complementary non-hallucination rate 67%","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":1.82,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"cyberbench_v1_1","score_raw":58.33,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"emb","score_raw":36.83,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"finance_agent_v1_1","score_raw":69.2,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"finance_agent_v1_1","score_raw":69.2,"score_norm":null,"config":"vendor scorecard / launch chart","trust":0,"source_type":"aggregator","source_url":"https://llm-stats.com/models/compare/ling-3.0-flash-vs-ling-3.0-flash-fin","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":54.93,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":59.8,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":59.8,"score_norm":null,"config":"vendor scorecard / launch chart","trust":0,"source_type":"aggregator","source_url":"https://llm-stats.com/models/compare/ling-3.0-flash-vs-ling-3.0-flash-fin","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1171,"score_norm":null,"config":"agentic professional knowledge work, elo","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"harveys_legal_agent_benchmark","score_raw":0,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":23,"score_norm":30.88,"config":"artificial analysis, reasoning model","trust":1,"source_type":"aggregator","source_url":"https://opper.ai/inclusionai/ling-3-0-flash-fin","measured_date":"2026-09-11","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.6,"score_norm":30.29,"config":"artificial analysis independent eval, pass@1 text-only hle","trust":1,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/inclusionai/ling-3.0-flash-fin/","measured_date":"2026-08-27","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"lcr","score_raw":73.7,"score_norm":null,"config":"artificial analysis (aa-lcr), long-context reasoning","trust":1,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/inclusionai/ling-3.0-flash-fin/","measured_date":"2026-08-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"legal_bench","score_raw":68.74,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"legal_research_bench","score_raw":15.38,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"long_context_reasoning","score_raw":74,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/inclusionai/ling-3-0-flash-fin","measured_date":"2026-09-11","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"medcode","score_raw":29.3,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"medscribe","score_raw":75.59,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/inclusionai/ling-3-0-flash-fin","measured_date":"2026-09-11","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.4,"score_norm":null,"config":"artificial analysis","trust":1,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/inclusionai/ling-3.0-flash-fin/","measured_date":"2026-08-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"spreadsheetbench_2","score_raw":21.8,"score_norm":null,"config":"vendor launch chart","trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"spreadsheetbench_v1","score_raw":86.5,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"spreadsheetbench_v1","score_raw":86.5,"score_norm":null,"config":"vendor scorecard / launch chart","trust":0,"source_type":"aggregator","source_url":"https://llm-stats.com/models/compare/ling-3.0-flash-vs-ling-3.0-flash-fin","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":41,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/ling-3-0-flash-fin/","measured_date":"2026-09-16","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":41,"score_norm":null,"config":"banking domain, vendor scorecard / launch chart","trust":0,"source_type":"aggregator","source_url":"https://llm-stats.com/models/compare/ling-3.0-flash-vs-ling-3.0-flash-fin","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"tax_agent_bench","score_raw":45.7,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"config":"2.1, default provider ant group, max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"4.0, difficult terminal-use tasks","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/ant-group-releases-finance-focused-ling-3-0-flash-fin","measured_date":"2026-09-16","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"vibe_code_bench_v1_1","score_raw":3.4,"score_norm":null,"config":"max output tokens 131072, temperature 1, top_k 20, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/ant_ling-3.0-flash-fin","measured_date":"2026-10-02","observed_at":"2026-10-03","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-fin:free","publisher":"OpenRouter (listed 2026-08-27)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-03T06:04:04.440Z"},{"id":"ling-3.0-flash-sante","display_name":"Ling 3.0 Flash Sante","family":{"id":"ling","vendor":"inclusionAI"},"generation":"3.0","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ling-3.0-flash-sante:free","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-09-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"inclusionai/ling-3.0-flash-sante:free","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"afumed_drug","score_raw":89.59,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI/status/2095953761504285107","measured_date":"2026-09-04","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"afumed_drug","score_raw":89.59,"score_norm":null,"config":"flash-size comparison, vendor-reported internal benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"afumed_drug_internal","score_raw":89.6,"score_norm":null,"config":"ant announcement chart transcription, internal","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"afusafe_medsce","score_raw":78.63,"score_norm":null,"config":"vendor-reported internal safety test","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"afusafe_medsce","score_raw":78.6,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://lookonchain.com/feeds/71605","measured_date":"2026-09-05","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"afusafe_medsce_internal","score_raw":78.6,"score_norm":null,"config":"ant announcement chart transcription, internal","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":86.89,"score_norm":null,"config":"multi-agent setup, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"browsecomp_multi_agent","score_raw":86.9,"score_norm":null,"config":"ant announcement chart, multi-agent","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"browsecomp_single_agent","score_raw":73.9,"score_norm":null,"config":"ant announcement chart, single agent with context","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":86.8,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"diagnosisarena_mcq","score_raw":83.83,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI/status/2095953761504285107","measured_date":"2026-09-04","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"diagnosisarena_mcq","score_raw":83.83,"score_norm":null,"config":"flash-size comparison, no shot/cot details published, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"diagnosisarena_mcq","score_raw":83.8,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":45.73,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI/status/2095953761504285107","measured_date":"2026-09-04","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":45.73,"score_norm":null,"config":"no shot/cot details published, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"healthbench_professional","score_raw":45.7,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":53.24,"score_norm":75.35,"config":"tools on, vendor-reported, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":53.2,"score_norm":75.29,"config":"ant announcement chart transcription, tools on, with tools","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medethicalign","score_raw":82.06,"score_norm":null,"config":"no shot/cot details published, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medethicalign","score_raw":82.1,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://lookonchain.com/feeds/71605","measured_date":"2026-09-05","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"medethicalign","score_raw":82.1,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medmcqa","score_raw":81.6,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medqa_usmle","score_raw":93.5,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medxpertqa_text","score_raw":53.88,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI/status/2095953761504285107","measured_date":"2026-09-04","observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"medxpertqa_text","score_raw":53.88,"score_norm":null,"config":"flash-size comparison, no shot/cot details published, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://x.com/AntLingAGI?lang=en","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"medxpertqa_text","score_raw":53.9,"score_norm":null,"config":"ant announcement chart transcription, no shot/cot details published","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"redisqa","score_raw":91.3,"score_norm":null,"config":"ant announcement chart transcription","trust":0,"source_type":"community","source_url":"https://aiwiki.ai/wiki/ling_3_0_flash_sante","measured_date":"2026-09-04","observed_at":"2026-09-29","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-sante:free","publisher":"OpenRouter (listed 2026-09-04)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-05T06:07:27.898Z"},{"id":"ling-3.0-flash-vision","display_name":"Ling 3.0 Flash VL","family":{"id":"ling","vendor":"inclusionAI"},"generation":"3.0","size_class":"frontier-flash","params_total_b":124.85,"params_active_b":null,"params_exact":124848460496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":250,"int8_gb":132,"int4_gb":74.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"inclusionAI/Ling-3.0-flash-VL","openrouter_id":"inclusionai/ling-3.0-flash-vl","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image","video"],"segment":"open_large","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-09-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ling-3.0-flash-vl"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.021,"completion_usd_per_m":0.0616,"cache_read_usd_per_m":0.0042,"context_length":262144,"source":"openrouter","source_model_id":"inclusionai/ling-3.0-flash-vl","as_of":"2026-10-06","blended_usd_per_m":0.03115,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":249.7,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":262.581,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":249.705,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":262.587,"method":"vendor","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash-VL/tree/main"},{"quant":"fp8","weights_gb":132.341,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":139.354,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":74.91,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":79.052,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":132.341,"kv_cache_gb_per_8k":0.3964,"min_serve_gb":139.354,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.47,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":124.85,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.37,"bw_limited_tok_s":2.47,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":31.19,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":124.85,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":53.66,"bw_limited_tok_s":32.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":28.7,"score_norm":28.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":30,"score_norm":30,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl:free","measured_date":null,"observed_at":"2026-09-11","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":57,"score_norm":57,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl","measured_date":null,"observed_at":"2026-09-12","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":57,"score_norm":57,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl:free","measured_date":null,"observed_at":"2026-09-11","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25,"score_norm":25,"config":"reasoning version, v4.3","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-3-0-flash-vl","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.6,"score_norm":24.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.8,"score_norm":24.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl:free","measured_date":null,"observed_at":"2026-09-11","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":42,"score_norm":42,"config":"thinking mode enabled by default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/inclusionAI/Ling-3.0-flash-VL","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25,"score_norm":25,"config":null,"trust":0,"source_type":"community","source_url":"https://alphasignal.ai/news/ant-group-s-ling-3-0-flash-vl-adds-vision-to-a-124b-open-reasoning-model","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"aa_lcr","score_raw":78.3,"score_norm":null,"config":"artificial analysis, long context reasoning","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"aa_omniscience","score_raw":14,"score_norm":null,"config":"accuracy","trust":0,"source_type":"community","source_url":"https://alphasignal.ai/news/ant-group-s-ling-3-0-flash-vl-adds-vision-to-a-124b-open-reasoning-model","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":14.3,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":78,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"automationbench_aa","score_raw":16,"score_norm":null,"config":"guarded business-application workflows","trust":0,"source_type":"community","source_url":"https://alphasignal.ai/news/ant-group-s-ling-3-0-flash-vl-adds-vision-to-a-124b-open-reasoning-model","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.2,"score_norm":86.2,"config":"artificial analysis, thinking/reasoning","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22,"score_norm":29.41,"config":"artificial analysis, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22,"score_norm":29.41,"config":"artificial analysis","trust":1,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/inclusionai/ling-3.0-flash-vl/","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"ling_3_0_flash_vl_agentic_index","score_raw":28.7,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"ling_3_0_flash_vl_coding_index","score_raw":57,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"ling_3_0_flash_vl_intelligence_index","score_raw":24.6,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":44.2,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inclusionai/ling-3.0-flash-vl:free","measured_date":"2026-09-10","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"4.0, agentic coding & terminal use, artificial analysis","trust":0,"source_type":"community","source_url":"https://alphasignal.ai/news/ant-group-s-ling-3-0-flash-vl-adds-vision-to-a-124b-open-reasoning-model","measured_date":"2026-09-11","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.0-flash-vl:free","publisher":"OpenRouter (listed 2026-09-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ling-3.1-flash","display_name":"Ling 3.1 Flash","family":{"id":"ling","vendor":"inclusionAI"},"generation":"3.1","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ling-3.1-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-09-29","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"inclusionai/ling-3.1-flash","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":41.1,"score_norm":41.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-3.1-flash","measured_date":null,"observed_at":"2026-10-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://benchlm.ai/model-updates","publisher":"BenchLM"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.1-405b-instruct","display_name":"Llama-3.1-405B-Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"405b","params_total_b":405.85,"params_active_b":405.85,"params_exact":405853388800,"params_source":"hf_safetensors","vram_est":{"bf16_gb":812,"int8_gb":430,"int4_gb":244,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.04,"expected_int8_gb":16.3,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-3.1-405B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":"llama3.1","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-07-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-3.1-405b"],"intelligence":{"index":23.22,"inputs":[{"benchmark":"aime","score_raw":3,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","config":"2025, artificial analysis independent run, non-reasoning"},{"benchmark":"arena_elo","score_raw":1335.2,"score_norm":24.77,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"41375 votes, arena entry llama-3.1-405b-instruct-bf16, leaderboard text-overall-style_control, rank 223, run: llama-3.1-405b-instruct-bf16, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":51.52,"score_norm":37.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.515151515151515 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.98,"score_norm":2.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","config":"artificial analysis independent run, no tools"},{"benchmark":"mmlu_pro","score_raw":73.3,"score_norm":55.5,"trust":1,"source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","config":"cot, 5-shot, micro_avg/acc_char"},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":1.78,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":811.7,"kv_cache_gb_per_8k":null,"min_serve_gb":856.285,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":811.707,"kv_cache_gb_per_8k":null,"min_serve_gb":856.292,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct/tree/main"},{"quant":"fp8","weights_gb":430.201,"kv_cache_gb_per_8k":null,"min_serve_gb":455.711,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":243.51,"kv_cache_gb_per_8k":null,"min_serve_gb":259.686,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":430.201,"kv_cache_gb_per_8k":null,"min_serve_gb":455.711,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":8.32,"score_norm":8.32,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":24.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.246666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":3,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.03 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":3,"score_norm":0,"config":"2025, artificial analysis independent run, non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_challenge","score_raw":96.9,"score_norm":null,"config":"0-shot, acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1335.2,"score_norm":24.77,"config":"41375 votes, arena entry llama-3.1-405b-instruct-bf16, leaderboard text-overall-style_control, rank 223, run: llama-3.1-405b-instruct-bf16, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"bfcl","score_raw":88.5,"score_norm":null,"config":"0-shot, acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":51.52,"score_norm":37.35,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.515151515151515 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":50.7,"score_norm":36.2,"config":"0-shot, exact match (main set, not diamond)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":96.8,"score_norm":null,"config":"cot, 8-shot, em_maj1@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.98,"score_norm":2.91,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0398 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.98,"score_norm":2.91,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":89,"score_norm":null,"config":"0-shot, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":39.05,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.390476190476191 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":88.6,"score_norm":null,"config":"as published on model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":30.48,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.304761904761905 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"lmarena_text_arena_elo","score_raw":1335,"score_norm":null,"config":"+/-4, 375 votes, 41, llama-3.1-405b-instruct-bf16, rank 219, text arena overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"math","score_raw":73.8,"score_norm":null,"config":"cot, 0-shot, final_em","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":91.6,"score_norm":null,"config":"cot, 0-shot, em (multilingual)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":87.3,"score_norm":null,"config":"5-shot, macro_avg/acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":73.3,"score_norm":55.5,"config":"cot, 5-shot, micro_avg/acc_char","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"multipl_e_humaneval","score_raw":75.2,"score_norm":null,"config":"0-shot, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":29.86,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.298611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":11.18,"score_norm":10.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"config":"artificial analysis independent run, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"artificial analysis independent run, terminal-bench hard subset","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-405b","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.meta.com/blog/meta-llama-3-1/","publisher":"Meta"},{"url":"https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct","publisher":"Hugging Face"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"llama-3.1-8b","display_name":"Llama 3.1 8B","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"8b","params_total_b":8.03,"params_active_b":8.03,"params_exact":8030261248,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.1,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.46,"expected_int8_gb":3.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-3.1-8B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"llama3.1","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2024-07-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":4.22,"inputs":[{"benchmark":"aime","score_raw":4.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":25.86,"score_norm":1.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"artificial analysis harness"},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"artificial analysis harness, no tools"},{"benchmark":"mmlu_pro","score_raw":48.3,"score_norm":13.83,"trust":1,"source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","config":"5-shot cot macro_avg/acc, llama 3.1 8b instruct column"},{"benchmark":"tau_bench","score_raw":16.37,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"artificial analysis harness, tau2"},{"benchmark":"terminal_bench","score_raw":1.5,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"artificial analysis harness, v2.1"}],"coverage":0.5635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16.06,"kv_cache_gb_per_8k":null,"min_serve_gb":17.381,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.061,"kv_cache_gb_per_8k":null,"min_serve_gb":17.382,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/tree/main"},{"quant":"fp8","weights_gb":8.512,"kv_cache_gb_per_8k":null,"min_serve_gb":9.833,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.818,"kv_cache_gb_per_8k":null,"min_serve_gb":6.139,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.512,"kv_cache_gb_per_8k":null,"min_serve_gb":9.833,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.72,"bw_limited_tok_s":45.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.99,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34,"bw_limited_tok_s":20.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":417.19,"bw_limited_tok_s":250.31,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":18,"score_norm":null,"config":"artificial analysis harness, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":4.33,"score_norm":0,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_c","score_raw":83.4,"score_norm":null,"config":"0-shot acc","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl","score_raw":76.1,"score_norm":null,"config":"0-shot acc","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":25.86,"score_norm":1.21,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":30.4,"score_norm":7.61,"config":"0-shot em, llama 3.1 8b instruct column","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":84.5,"score_norm":null,"config":"8-shot cot em_maj1@1","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"config":"artificial analysis harness, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":72.6,"score_norm":null,"config":"0-shot pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":28.57,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":80.4,"score_norm":null,"config":"llama 3.1 8b instruct column","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":11.64,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":51.9,"score_norm":null,"config":"0-shot cot final_em","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":69.4,"score_norm":null,"config":"5-shot macro_avg/acc","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":48.3,"score_norm":13.83,"config":"5-shot cot macro_avg/acc, llama 3.1 8b instruct column","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":16.37,"score_norm":0,"config":"artificial analysis harness, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.5,"score_norm":0,"config":"artificial analysis harness, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.llama.com/llama3_1/","publisher":"Meta"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"llama-3.1-8b-instruct","display_name":"Llama 3.1 8B Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"8b","params_total_b":8.03,"params_active_b":8.03,"params_exact":8030261248,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.1,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.26,"expected_int8_gb":2.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Meta-Llama-3.1-8B-Instruct","openrouter_id":"meta-llama/llama-3.1-8b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"llama3.1","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-07-23","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["meta-llama-3.1-8b-instruct"],"intelligence":{"index":1.39,"inputs":[{"benchmark":"aime","score_raw":4.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0433333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1211.2,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":25.86,"score_norm":1.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.258585858585859 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0529 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":16.37,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.16374269005848 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":1.5,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0149812734082397 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5238,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.08,"cache_read_usd_per_m":0.025,"context_length":131072,"source":"openrouter","source_model_id":"meta-llama/llama-3.1-8b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.05750000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":16.06,"kv_cache_gb_per_8k":null,"min_serve_gb":17.381,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.061,"kv_cache_gb_per_8k":null,"min_serve_gb":17.382,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct/tree/main"},{"quant":"fp8","weights_gb":8.512,"kv_cache_gb_per_8k":null,"min_serve_gb":9.833,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.818,"kv_cache_gb_per_8k":null,"min_serve_gb":6.139,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.512,"kv_cache_gb_per_8k":null,"min_serve_gb":9.833,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":75.72,"bw_limited_tok_s":45.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.99,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34,"bw_limited_tok_s":20.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":417.19,"bw_limited_tok_s":250.31,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":5.4,"score_norm":5.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.1-8b-instruct","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.9,"score_norm":6.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.1-8b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.4,"score_norm":7.4,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":18,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.18 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":4.33,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0433333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1211.2,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1211.3,"score_norm":0,"config":"49605 votes, arena entry llama-3.1-8b-instruct, leaderboard text-overall-style_control, rank 321, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":25.86,"score_norm":1.21,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.258585858585859 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":30.4,"score_norm":7.61,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.29,"score_norm":4.84,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0529 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":28.57,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":11.64,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.116402116402116 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":13.19,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.131944444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":16.37,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.16374269005848 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.5,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0149812734082397 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.1-8b-instruct","publisher":"OpenRouter (listed 2024-07-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.1-nemotron-ultra-253b-v1","display_name":"Llama-3.1-Nemotron-Ultra-253B-v1","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"253b","params_total_b":253.4,"params_active_b":253.4,"params_exact":253401268224,"params_source":"hf_safetensors","vram_est":{"bf16_gb":507,"int8_gb":269,"int4_gb":152,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.15,"expected_int8_gb":39.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/Llama-3_1-Nemotron-Ultra-253B-v1","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-05-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["llama-3.1-nemotron-ultra-253b"],"superseded_by":null,"aliases":["llama-3-1-nemotron-ultra-253b-v1"],"intelligence":{"index":50.39,"inputs":[{"benchmark":"aime","score_raw":72.5,"score_norm":65.63,"trust":2,"source_url":"https://arxiv.org/pdf/2505.00949","config":"aime25, reasoning on"},{"benchmark":"arena_elo","score_raw":1347.6,"score_norm":30.3,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":73,"score_norm":67.61,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent"},{"benchmark":"hle","score_raw":7,"score_norm":7.35,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, no tools"},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro","config":"artificial analysis independent eval, reasoning on"}],"coverage":0.5475,"subset_bias":{"offset":3.59,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":506.8,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":534.288,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":506.803,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":534.291,"method":"vendor","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/tree/main"},{"quant":"fp8","weights_gb":268.604,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":284.182,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":152.04,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":161.79,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":268.604,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":284.182,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":72.5,"score_norm":65.63,"config":"aime25, reasoning on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1347.6,"score_norm":30.3,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1347,"score_norm":30.03,"config":"2549 votes, rank 206, text arena overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":87,"score_norm":null,"config":"reasoning on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v2_live","score_raw":74.1,"score_norm":null,"config":"reasoning on, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76,"score_norm":71.83,"config":"reasoning on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73,"score_norm":67.61,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7,"score_norm":7.35,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":88.9,"score_norm":null,"config":"reasoning on, strict-instruction","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"judgebench","score_raw":79.14,"score_norm":null,"config":"overall","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":66.3,"score_norm":null,"config":"2408-2502, reasoning on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math500","score_raw":97,"score_norm":null,"config":"reasoning on","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2505.00949","measured_date":"2025-05-02","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"config":"artificial analysis independent eval, reasoning on","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"llama-3.2-1b-instruct","display_name":"Llama 3.2 1B Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.2","size_class":"1b","params_total_b":1.24,"params_active_b":1.24,"params_exact":1235814400,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2.5,"int8_gb":1.3,"int4_gb":0.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.7,"expected_int8_gb":2.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-3.2-1B-Instruct","openrouter_id":"meta-llama/llama-3.2-1b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"llama3.2","is_open_weights":true,"self_hostable":true,"context_window_tok":60000,"release_date":"2024-09-25","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-3.2-1b"],"intelligence":{"index":1.08,"inputs":[{"benchmark":"aime","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1111.1,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":19.6,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.195959595959596 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.46,"score_norm":5.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0546 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.027,"completion_usd_per_m":0.201,"cache_read_usd_per_m":null,"context_length":60000,"source":"openrouter","source_model_id":"meta-llama/llama-3.2-1b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.07050000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2.48,"kv_cache_gb_per_8k":null,"min_serve_gb":3.53,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":2.472,"kv_cache_gb_per_8k":null,"min_serve_gb":3.522,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/tree/main"},{"quant":"fp8","weights_gb":1.314,"kv_cache_gb_per_8k":null,"min_serve_gb":2.364,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.744,"kv_cache_gb_per_8k":null,"min_serve_gb":1.794,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1.314,"kv_cache_gb_per_8k":null,"min_serve_gb":2.364,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":227.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":1.24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":490.32,"bw_limited_tok_s":294.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":116.68,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":1.24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":220.16,"bw_limited_tok_s":132.1,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":618.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":1.24,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2701.61,"bw_limited_tok_s":1620.97,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":4.8,"score_norm":4.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.2-1b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":1,"score_norm":1,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":6,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.06 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":0,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1111.1,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1110.9,"score_norm":0,"config":"8045 votes, arena entry llama-3.2-1b-instruct, leaderboard text-overall-style_control, rank 376, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":19.6,"score_norm":0,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.195959595959596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.46,"score_norm":5.09,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0546 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":22.79,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.227891156462585 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":1.9,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.019047619047619 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":1.74,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.0173611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-1b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.2-1b-instruct","publisher":"OpenRouter (listed 2024-09-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.2-3b-instruct","display_name":"Llama 3.2 3B Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.2","size_class":"3b","params_total_b":3.21,"params_active_b":3.21,"params_exact":3212749824,"params_source":"hf_safetensors","vram_est":{"bf16_gb":6.4,"int8_gb":3.4,"int4_gb":1.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.66,"expected_int8_gb":2.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-3.2-3B-Instruct","openrouter_id":"meta-llama/llama-3.2-3b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"llama3.2","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-09-25","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-3.2-3b"],"intelligence":{"index":1.5,"inputs":[{"benchmark":"aime","score_raw":3.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0333333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1166.7,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":25.45,"score_norm":0.63,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.254545454545455 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.35,"score_norm":4.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0535 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.210526315789474 is a 0-1 fraction, x100 at harvest"}],"coverage":0.4762,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.33,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"meta-llama/llama-3.2-3b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.12000000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":6.42,"kv_cache_gb_per_8k":null,"min_serve_gb":7.548,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":6.426,"kv_cache_gb_per_8k":null,"min_serve_gb":7.554,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct/tree/main"},{"quant":"fp8","weights_gb":3.403,"kv_cache_gb_per_8k":null,"min_serve_gb":4.531,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.926,"kv_cache_gb_per_8k":null,"min_serve_gb":3.054,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":3.403,"kv_cache_gb_per_8k":null,"min_serve_gb":4.531,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":102.05,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.21,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":189.41,"bw_limited_tok_s":113.64,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.55,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.21,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":85.05,"bw_limited_tok_s":51.03,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":385.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.21,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1043.61,"bw_limited_tok_s":626.17,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.7,"score_norm":5.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.2-3b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":3.88,"score_norm":3.88,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":2.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.0266666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":3.33,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0333333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1166.7,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1166.6,"score_norm":0,"config":"7936 votes, arena entry llama-3.2-3b-instruct, leaderboard text-overall-style_control, rank 347, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":25.45,"score_norm":0.63,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.254545454545455 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.35,"score_norm":4.93,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0535 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":26.19,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.261904761904762 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":8.25,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.0825396825396825 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":5.21,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.0520833333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.210526315789474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-2-instruct-3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.2-3b-instruct","publisher":"OpenRouter (listed 2024-09-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.3-70b-instruct","display_name":"Llama-3.3-70B-Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.3","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.16,"expected_int8_gb":12,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-3.3-70B-Instruct","openrouter_id":"meta-llama/llama-3.3-70b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"llama3.3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-12-06","release_status":"released","knowledge_cutoff":"2023-12-31","status":"current","supersedes":["llama-3.1-70b-instruct"],"superseded_by":null,"aliases":["llama-3.3-70b"],"intelligence":{"index":16.6,"inputs":[{"benchmark":"aime","score_raw":7.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0766666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1317.8,"score_norm":17.01,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":49.8,"score_norm":34.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: llama 3.3 instruct 70b, source value 0.497979797979798 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.56,"score_norm":2.29,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0356 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":68.9,"score_norm":48.17,"trust":1,"source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","config":"cot, 5-shot, macro_avg/acc"},{"benchmark":"swe_bench_verified","score_raw":5.4,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/pdf/2502.18449","config":"'baseline comparison on swe-bench verified'. row: llama-3.3-70b-instruct, 'repair performance (oracle)' = 5.4 (correct-format rate 12.2% [raw: swe-bench verified], arxiv:2502.18449) table 2, greedy decoding, swe-rl paper (meta"},{"benchmark":"tau_bench","score_raw":26.61,"score_norm":8.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.266081871345029 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.22,"completion_usd_per_m":0.5,"cache_read_usd_per_m":0.11,"context_length":131072,"source":"openrouter","source_model_id":"meta-llama/llama-3.3-70b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.29000000000000004,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":null,"min_serve_gb":150.977,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":null,"min_serve_gb":150.984,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/tree/main"},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":null,"min_serve_gb":81.344,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":null,"min_serve_gb":47.269,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":null,"min_serve_gb":81.344,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.87,"bw_limited_tok_s":2.32,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":53.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70.55,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":94.97,"bw_limited_tok_s":56.98,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":11.9,"score_norm":11.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.3-70b-instruct","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.7,"score_norm":7.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.3-70b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.26,"score_norm":9.26,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":15.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.156666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":7.67,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1317.8,"score_norm":17.01,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1318.4,"score_norm":17.28,"config":"54774 votes, arena entry llama-3.3-70b-instruct, leaderboard text-overall-style_control, rank 243, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1318,"score_norm":17.1,"config":"links to huggingface.co/meta-llama [raw: lmarena (chatbot arena) — text arena elo], official lmarena text leaderboard (lmarena.ai/leaderboard/text 301-redirects to arena.ai/leaderboard/text). row model key 'llama-3.3-70b-instruct' (meta · llama-3.3","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"bfcl_v2","score_raw":77.3,"score_norm":null,"config":"0-shot, overall_ast_summary/macro_avg/valid","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49.8,"score_norm":34.93,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.497979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49.8,"score_norm":34.93,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: llama 3.3 instruct 70b, source value 0.497979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":50.5,"score_norm":35.92,"config":"cot, 0-shot, acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.56,"score_norm":2.29,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0356 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":88.4,"score_norm":null,"config":"0-shot, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":47.07,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.470748299319728 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":92.1,"score_norm":null,"config":"steerability","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":0.56,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=false, source value 0.00564971751412429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":28.78,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.287830687830688 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math","score_raw":77,"score_norm":null,"config":"cot, 0-shot, sympy_intersection_score","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mbpp_evalplus","score_raw":87.6,"score_norm":null,"config":"base, 0-shot, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":91.1,"score_norm":null,"config":"0-shot, em","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":86,"score_norm":null,"config":"cot, 0-shot, macro_avg/acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":68.9,"score_norm":48.17,"config":"cot, 5-shot, macro_avg/acc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct","measured_date":"2024-12-06","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":26.04,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.260416666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":5.4,"score_norm":0,"config":"'baseline comparison on swe-bench verified'. row: llama-3.3-70b-instruct, 'repair performance (oracle)' = 5.4 (correct-format rate 12.2% [raw: swe-bench verified], arxiv:2502.18449) table 2, greedy decoding, swe-rl paper (meta","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2502.18449","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":26.61,"score_norm":8.81,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.266081871345029 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.87,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.048689138576779 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-3-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/meta-llama","publisher":"Hugging Face"},{"url":"https://www.interconnects.ai/p/llama-4","publisher":"Interconnects"},{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.3-70b-instruct","publisher":"OpenRouter (listed 2024-12-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.3-nemotron-super-49b-v1.5","display_name":"Llama-3.3-Nemotron-Super-49B-v1.5","family":{"id":"llama","vendor":"Meta"},"generation":"3.3","size_class":"49b","params_total_b":49,"params_active_b":49,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":98,"int8_gb":51.9,"int4_gb":29.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.68,"expected_int8_gb":35.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/Llama-3.3-Nemotron-Super-49B-v1.5","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"NVIDIA Open Model License","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07-25","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nemotron-llama-3.3-super-49b-v1.5"],"intelligence":{"index":46.15,"inputs":[{"benchmark":"aime","score_raw":76.67,"score_norm":70.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","config":"aime 2025, artificial analysis harness, reasoning mode"},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","config":"artificial analysis harness, reasoning mode"},{"benchmark":"hle","score_raw":7.32,"score_norm":7.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","config":"artificial analysis harness, no tools, reasoning mode"},{"benchmark":"mmlu_pro","score_raw":79.53,"score_norm":65.88,"trust":1,"source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","config":"cot, pass@1 avg over 1 run, reasoning on"},{"benchmark":"tau_bench","score_raw":28.07,"score_norm":10.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","config":"artificial analysis harness, reasoning mode, tau2"},{"benchmark":"terminal_bench","score_raw":5.3,"score_norm":0.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","config":"artificial analysis harness, reasoning mode"}],"coverage":0.5635,"subset_bias":{"offset":0.3,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":98,"kv_cache_gb_per_8k":null,"min_serve_gb":104.86,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":51.94,"kv_cache_gb_per_8k":null,"min_serve_gb":56.497,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":29.4,"kv_cache_gb_per_8k":null,"min_serve_gb":32.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":51.94,"kv_cache_gb_per_8k":null,"min_serve_gb":56.497,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":3.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":49,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":5.57,"bw_limited_tok_s":3.34,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":39.4,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":49,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":68.37,"bw_limited_tok_s":41.02,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":76.67,"score_norm":70.84,"config":"aime 2025, artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":82.71,"score_norm":78.39,"config":"aime 2025, pass@1 avg over 16 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87.5,"score_norm":84.38,"config":"aime 2024, pass@1 avg over 16 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arenahard","score_raw":92,"score_norm":null,"config":"pass@1 avg over 1 run, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":71.75,"score_norm":null,"config":"pass@1 avg over 2 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.97,"score_norm":66.15,"config":"nemo-skills, pass@1 avg over 4 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.32,"score_norm":7.82,"config":"artificial analysis harness, no tools, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.64,"score_norm":8.29,"config":"pass@1 avg over 1 run, reasoning on, text-only subset","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":37.01,"score_norm":null,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":88.61,"score_norm":null,"config":"reasoning on, strict:instruction","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":73.58,"score_norm":null,"config":"24.10-25.02, pass@1 avg over 4 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math500","score_raw":97.4,"score_norm":null,"config":"pass@1 avg over 4 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.53,"score_norm":65.88,"config":"cot, pass@1 avg over 1 run, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":28.07,"score_norm":10.76,"config":"artificial analysis harness, reasoning mode, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":5.3,"score_norm":0.35,"config":"artificial analysis harness, reasoning mode","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-nemotron-super-49b-v1-5-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/nvidia/nemotron-open-models-data","publisher":"Hugging Face / NVIDIA"},{"url":"https://docs.vultr.com/inference-cookbook/cuda/model-guides/nemotron-super-49b","publisher":"Vultr"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"llama-4-guard-12b","display_name":"Llama Guard 4 12B","family":{"id":"llama","vendor":"Meta"},"generation":"4","size_class":"guard-12b","params_total_b":12,"params_active_b":12,"params_exact":12001097216,"params_source":"hf_safetensors","vram_est":{"bf16_gb":24,"int8_gb":12.7,"int4_gb":7.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"meta-llama/Llama-Guard-4-12B","openrouter_id":"meta-llama/llama-guard-4-12b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-04-30","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-guard-4-12b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.18,"completion_usd_per_m":0.18,"cache_read_usd_per_m":null,"context_length":163840,"source":"openrouter","source_model_id":"meta-llama/llama-guard-4-12b","as_of":"2026-10-06","blended_usd_per_m":0.18,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":24,"kv_cache_gb_per_8k":null,"min_serve_gb":25.68,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":24.002,"kv_cache_gb_per_8k":null,"min_serve_gb":25.682,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-Guard-4-12B/tree/main"},{"quant":"fp8","weights_gb":12.72,"kv_cache_gb_per_8k":null,"min_serve_gb":14.2,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.2,"kv_cache_gb_per_8k":null,"min_serve_gb":8.68,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":12.72,"kv_cache_gb_per_8k":null,"min_serve_gb":14.2,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":29.5,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":50.67,"bw_limited_tok_s":30.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.75,"bw_limited_tok_s":13.65,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":143.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":279.17,"bw_limited_tok_s":167.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-guard-4-12b","publisher":"OpenRouter (listed 2025-04-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-4-maverick-17b-128e-instruct","display_name":"Llama-4-Maverick-17B-128E-Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"4","size_class":"17b","params_total_b":401.58,"params_active_b":17,"params_exact":401583781376,"params_source":"hf_safetensors","vram_est":{"bf16_gb":803,"int8_gb":426,"int4_gb":241,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.21,"expected_int8_gb":88.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-4-Maverick-17B-128E-Instruct","openrouter_id":"meta-llama/llama-4-maverick","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2025-04-05","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-4-maverick","llama-4-maverick-17b","llama-4-maverick-17b-128e","llama-4-maverick-17b-128e-instruct-fp8","llama-4-maverick-400b"],"intelligence":{"index":28.58,"inputs":[{"benchmark":"aime","score_raw":25.21,"score_norm":6.51,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"overall accuracy (95% ci 17.5-32.9), served via fireworks as llama4-maverick-instruct-basic, vals ai aime leaderboard"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0121, eval=arc-agi-2 semi-private (v2_semi_private), model_group=llama 4, org=meta, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1327,"score_norm":21.11,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":67.07,"score_norm":59.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-4-maverick","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.670707070707071 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":5.7,"score_norm":5.44,"trust":2,"source_url":"https://epoch.ai/models/llama-4-maverick","config":"best score across settings, epoch ai independent eval, no tools"},{"benchmark":"livebench","score_raw":54.38,"score_norm":31.96,"trust":3,"source_url":"https://livebench.ai/table_2025_04_02.csv","config":"global average = mean of 6 category averages computed from official table_2025_04_02.csv, livebench 2025-04-02 release"},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"trust":1,"source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","config":"macro_avg/acc; 0-shot, temperature=0, no majority voting"},{"benchmark":"tau_bench","score_raw":17.84,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-4-maverick","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.178362573099415 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":2.25,"score_norm":0,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2","config":"overall accuracy (2/89 tasks), terminus 2 harness, vals ai terminal-bench 2.0"}],"coverage":0.8412,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1875,"completion_usd_per_m":0.6525,"cache_read_usd_per_m":null,"context_length":1048576,"source":"openrouter","source_model_id":"meta-llama/llama-4-maverick","as_of":"2026-10-06","blended_usd_per_m":0.30374999999999996,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":803.16,"kv_cache_gb_per_8k":null,"min_serve_gb":843.998,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":803.168,"kv_cache_gb_per_8k":null,"min_serve_gb":844.006,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct/tree/main"},{"quant":"fp8","weights_gb":425.675,"kv_cache_gb_per_8k":null,"min_serve_gb":447.639,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":240.948,"kv_cache_gb_per_8k":null,"min_serve_gb":253.675,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":425.675,"kv_cache_gb_per_8k":null,"min_serve_gb":447.639,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.6,"score_norm":0.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.24,"score_norm":1.24,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":16.3,"score_norm":16.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10,"score_norm":10,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.48,"score_norm":14.48,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":50,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.5 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":25.21,"score_norm":6.51,"config":"overall accuracy (95% ci 17.5-32.9), served via fireworks as llama4-maverick-instruct-basic, vals ai aime leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":19.33,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.193333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":4.38,"score_norm":null,"config":"cost_per_task_usd=0.0078, eval=arc-agi-1 semi-private (v1_semi_private), model_group=llama 4, org=meta, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.0121, eval=arc-agi-2 semi-private (v2_semi_private), model_group=llama 4, org=meta, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"best score across settings, epoch ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1327,"score_norm":21.11,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1327.1,"score_norm":21.16,"config":"40038 votes, arena entry llama-4-maverick-17b-128e-instruct, leaderboard text-overall-style_control, rank 230, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417,"score_norm":61.23,"config":"llama-4-maverick-03-26-experimental chat variant (not the public release); meta-reported","trust":1,"source_type":"vendor_claim","source_url":"https://ai.meta.com/blog/llama-4-multimodal-intelligence/","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1145.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1146.6,"score_norm":null,"config":"7006 votes, arena entry llama-4-maverick-17b-128e-instruct, leaderboard vision-overall-style_control, rank 101, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":90,"score_norm":null,"config":"relaxed accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":916,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":885,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":889,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":850,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":906,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":877,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":94.4,"score_norm":null,"config":"test split, anls","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.07,"score_norm":59.25,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.670707070707071 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":69.8,"score_norm":63.1,"config":"accuracy; 0-shot, averaged over multiple generations","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.91,"score_norm":4.28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0491195551436515 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.7,"score_norm":5.44,"config":"best score across settings, epoch ai independent eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":42.99,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.429931972789116 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":54.38,"score_norm":31.96,"config":"global average = mean of 6 category averages computed from official table_2025_04_02.csv, livebench 2025-04-02 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2025_04_02.csv","measured_date":"2025-04-02","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":39.68,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.396825396825397 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":43.4,"score_norm":null,"config":"pass@1; window 10/01/2024-02/01/2025","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mathvista","score_raw":73.7,"score_norm":null,"config":"accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":2.22,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0222222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"config":"macro_avg/acc; 0-shot, temperature=0, no majority voting","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu","score_raw":73.4,"score_norm":null,"config":"accuracy, image reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":62.14,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.621387283236994 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"multilingual_mmlu","score_raw":84.6,"score_norm":null,"config":"average across languages","trust":1,"source_type":"vendor_claim","source_url":"https://developer.meta.com/ai/models/llama-4/","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":33.1,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.331018518518519 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":5.24,"score_norm":0.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":17.84,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.178362573099415 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":3.71,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0371134020618557 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.25,"score_norm":0,"config":"overall accuracy (2/89 tasks), terminus 2 harness, vals ai terminal-bench 2.0","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":7.87,"score_norm":3.38,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0786516853932584 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-maverick","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.meta.com/blog/llama-4-multimodal-intelligence/","publisher":"Meta"},{"url":"https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-maverick","publisher":"OpenRouter (listed 2025-04-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-4-scout-17b-16e-instruct","display_name":"Llama-4-Scout-17B-16E-Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"4","size_class":"17b","params_total_b":108.64,"params_active_b":17,"params_exact":108641793536,"params_source":"hf_safetensors","vram_est":{"bf16_gb":217,"int8_gb":115,"int4_gb":65.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.48,"expected_int8_gb":55.3,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-llama/Llama-4-Scout-17B-16E-Instruct","openrouter_id":"meta-llama/llama-4-scout","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1310720,"release_date":"2025-04-05","release_status":"released","knowledge_cutoff":"2024-08-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["llama-4-scout","llama-4-scout-109b","llama-4-scout-17b","llama-4-scout-17b-16e","llama-4-scout-17b-16e-instruct-fp8"],"intelligence":{"index":17.4,"inputs":[{"benchmark":"aime","score_raw":18.96,"score_norm":0,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.7, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0062, eval=arc-agi-2 semi-private (v2_semi_private), model_group=llama 4, org=meta, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1321.6,"score_norm":18.71,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":46.97,"score_norm":30.94,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.491, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":3.78,"score_norm":2.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-4-scout","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0378 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":69.63,"score_norm":49.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.445, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":15.5,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-4-scout","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.154970760233918 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-4-scout","config":"artificial analysis harness, intelligence index v4.1.1 component, page value terminalbenchv21=0.0374531835205993 [raw: terminal-bench v2.1], terminal-bench v2.1 (89-task dataset)"}],"coverage":0.7222,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.3,"cache_read_usd_per_m":null,"context_length":1310720,"source":"openrouter","source_model_id":"meta-llama/llama-4-scout","as_of":"2026-10-06","blended_usd_per_m":0.15000000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":217.28,"kv_cache_gb_per_8k":null,"min_serve_gb":228.824,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":217.284,"kv_cache_gb_per_8k":null,"min_serve_gb":228.828,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct/tree/main"},{"quant":"fp8","weights_gb":115.158,"kv_cache_gb_per_8k":null,"min_serve_gb":121.596,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":65.184,"kv_cache_gb_per_8k":null,"min_serve_gb":69.123,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":115.158,"kv_cache_gb_per_8k":null,"min_serve_gb":121.596,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":17,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":16.06,"bw_limited_tok_s":9.09,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":191.25,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":17,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":394.12,"bw_limited_tok_s":236.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.5,"score_norm":0.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.1,"score_norm":1.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":8.2,"score_norm":8.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.1,"score_norm":8.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.26,"score_norm":10.26,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":30.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.303333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":18.96,"score_norm":0,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.7, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":14,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.14 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":14,"score_norm":0,"config":"aime 2025, artificial analysis independent run, no tools, non-reasoning model (isreasoning=false), page value aime25=0.14 [raw: aime 2025]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":0.5,"score_norm":null,"config":"cost_per_task_usd=0.0041, eval=arc-agi-1 semi-private (v1_semi_private), model_group=llama 4, org=meta, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.0062, eval=arc-agi-2 semi-private (v2_semi_private), model_group=llama 4, org=meta, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1321.6,"score_norm":18.71,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1322.5,"score_norm":19.11,"config":"30378 votes, arena entry llama-4-scout-17b-16e-instruct, leaderboard text-overall-style_control, rank 237, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1322,"score_norm":18.88,"config":"displayed score 1322 ±5 (raw rating 1322.4888144600643 [raw: lmarena (chatbot arena) text arena score], lmarena text leaderboard row keyed modelkey=\"llama-4-scout-17b-16e-instruct\" linking to huggingface.co/meta-llama/llama-4-scout-17b-16e-instruct","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1127.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1127.8,"score_norm":null,"config":"6492 votes, arena entry llama-4-scout-17b-16e-instruct, leaderboard vision-overall-style_control, rank 109, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":88.8,"score_norm":null,"config":"0-shot relaxed_accuracy, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":795,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":903,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":786,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":774,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":757,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":94.4,"score_norm":null,"config":"test, 0-shot anls, instruction-tuned","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.97,"score_norm":30.94,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.491, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":58.69,"score_norm":47.45,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.586868686868687 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.2,"score_norm":45.35,"config":"0-shot accuracy, bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.78,"score_norm":2.62,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0378 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.78,"score_norm":2.62,"config":"1 repeat, 158 questions), artificial analysis independent run, hle text-only subset (2, intelligence index v4.1.1, no tools, page value hle=0.0378 [raw: humanity's last exam], pass@1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":39.52,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.395238095238095 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":29.95,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.299470899470899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":32.8,"score_norm":null,"config":"0-shot pass@1, window 10/01/2024-02/01/2025","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mathvista","score_raw":70.7,"score_norm":null,"config":"0-shot accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":90.6,"score_norm":null,"config":"0-shot, average/em","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":79.6,"score_norm":null,"config":"pre-trained model, 5-shot, macro_avg/acc_char","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.63,"score_norm":49.38,"config":"edition=mmlu pro v1, runner=platform, stderr=0.445, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":74.3,"score_norm":57.17,"config":"instruction-tuned, 0-shot, macro_avg/acc, bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu","score_raw":69.4,"score_norm":null,"config":"0-shot accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":52.95,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.529479768786127 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":52.2,"score_norm":null,"config":"0-shot, average of standard and vision","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","measured_date":"2025-04-05","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":17.01,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.170138888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":15.5,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.154970760233918 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":3.3,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0329896907216495 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"config":"artificial analysis harness, intelligence index v4.1.1 component, page value terminalbenchv21=0.0374531835205993 [raw: terminal-bench v2.1], terminal-bench v2.1 (89-task dataset)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-4-scout","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/llama4-release","publisher":"Hugging Face"},{"url":"https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-4-scout","publisher":"OpenRouter (listed 2025-04-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"longcat-2.0","display_name":"LongCat 2.0","family":{"id":"longcat","vendor":"meituan"},"generation":"2.0","size_class":"frontier","params_total_b":1775.56,"params_active_b":null,"params_exact":1775560491136,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3551,"int8_gb":1882,"int4_gb":1065,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.13,"expected_int8_gb":247,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meituan-longcat/LongCat-2.0","openrouter_id":"meituan/longcat-2.0","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048756,"release_date":"2026-07-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":61.6,"inputs":[{"benchmark":"gpqa_diamond","score_raw":77.98,"score_norm":74.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/longcat-2-0","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.77979797979798 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":33.69,"score_norm":46.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/longcat-2-0","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.336886005560704 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/longcat-2-0","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.50187265917603 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3174,"subset_bias":{"offset":-4.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.006,"context_length":1048756,"source":"openrouter","source_model_id":"meituan/longcat-2.0","as_of":"2026-10-06","blended_usd_per_m":0.5249999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":3551.12,"kv_cache_gb_per_8k":0.3586,"min_serve_gb":3729.035,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":3551.711,"kv_cache_gb_per_8k":0.3586,"min_serve_gb":3729.655,"method":"vendor","source_url":"https://huggingface.co/meituan-longcat/LongCat-2.0/tree/main"},{"quant":"fp8","weights_gb":1882.094,"kv_cache_gb_per_8k":0.3586,"min_serve_gb":1976.557,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1065.336,"kv_cache_gb_per_8k":0.3586,"min_serve_gb":1118.961,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1882.094,"kv_cache_gb_per_8k":0.3586,"min_serve_gb":1976.557,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":14,"score_norm":14,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meituan/longcat-2.0","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.96,"score_norm":21.96,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":45.3,"score_norm":45.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meituan/longcat-2.0","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.1,"score_norm":19.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meituan/longcat-2.0","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.97,"score_norm":33.97,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":62.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.626666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0257142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":26.52,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.265245 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.98,"score_norm":74.62,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.77979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.69,"score_norm":46.6,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.336886005560704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":35.42,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.354166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":13.2,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.131958762886598 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.19,"score_norm":53.16,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.50187265917603 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/longcat-2-0","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.8,"score_norm":77.41,"config":"2.1, measured in-house under longcat's own unified harness, no further scaffold/temperature/context details given.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meituan-longcat/LongCat-2.0","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meituan/longcat-2.0","publisher":"OpenRouter (listed 2026-07-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"lyria-3-clip","display_name":"lyria-3-clip-preview","family":{"id":"lyria","vendor":"google"},"generation":"3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/lyria-3-clip-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-03-30","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["lyria-3-clip-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":1048576,"source":"openrouter","source_model_id":"google/lyria-3-clip-preview","as_of":"2026-09-26","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/lyria-3-clip-preview","publisher":"OpenRouter (listed 2026-03-30)"}],"last_verified":"2026-09-26","last_enriched_at":null},{"id":"lyria-3-pro","display_name":"lyria-3-pro-preview","family":{"id":"lyria","vendor":"google"},"generation":"3","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/lyria-3-pro-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-03-30","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["lyria-3-pro-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":1048576,"source":"openrouter","source_model_id":"google/lyria-3-pro-preview","as_of":"2026-09-26","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/lyria-3-pro-preview","publisher":"OpenRouter (listed 2026-03-30)"}],"last_verified":"2026-09-26","last_enriched_at":null},{"id":"mercury-2","display_name":"Mercury 2","family":{"id":"mercury","vendor":"inception"},"generation":"2","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inception/mercury-2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2026-03-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":46.86,"inputs":[{"benchmark":"arena_elo","score_raw":1343.3,"score_norm":28.38,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":76.97,"score_norm":73.2,"trust":2,"source_url":"https://artificialanalysis.ai/models/mercury-2","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.76969696969697 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":17.15,"score_norm":22.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/mercury-2","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.171455050973123 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":70.76,"score_norm":67.68,"trust":2,"source_url":"https://artificialanalysis.ai/models/mercury-2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.707602339181287 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":27.34,"score_norm":26.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/mercury-2","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.273408239700375 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-6.43,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":0.75,"cache_read_usd_per_m":0.025,"context_length":128000,"source":"openrouter","source_model_id":"inception/mercury-2","as_of":"2026-10-06","blended_usd_per_m":0.375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.6,"score_norm":1.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":9.46,"score_norm":9.46,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":31.1,"score_norm":31.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.8,"score_norm":13.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.9,"score_norm":21.9,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":40.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, source value 0.406666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1343.3,"score_norm":28.38,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1346.7,"score_norm":29.89,"config":"3122 votes, arena entry mercury-2, leaderboard text-overall-style_control, rank 206, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1170,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-30","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1166.2,"score_norm":null,"config":"908 votes, arena entry mercury-2, code/overall, leaderboard webdev-overall-raw, rank 115, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.85,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, source value 0.00848979591836735 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":994,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":999,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1023,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1015,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":999,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":996,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1000,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1037,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":9.89,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, source value 0.09886500000000001 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.97,"score_norm":73.2,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.76969696969697 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.15,"score_norm":22.28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.171455050973123 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":69.8,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, source value 0.697959183673469 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":38.66,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, scicode, source value 0.386574074074074 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":70.76,"score_norm":67.68,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.707602339181287 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.48,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0948453608247423 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":26.52,"score_norm":25.32,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.265151515151515 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":27.34,"score_norm":26.28,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.273408239700375 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inception/mercury-2","publisher":"OpenRouter (listed 2026-03-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mercury-2.5","display_name":"Mercury 2.5","family":{"id":"mercury","vendor":"inception"},"generation":"2.5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inception/mercury-2.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":260000,"release_date":"2026-08-31","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mercury-2.5-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.04,"completion_usd_per_m":0.15,"cache_read_usd_per_m":0.004,"context_length":260000,"source":"openrouter","source_model_id":"inception/mercury-2.5","as_of":"2026-10-06","blended_usd_per_m":0.0675,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.3,"score_norm":12.3,"config":"artificial analysis intelligence index, openrouter republish","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12,"score_norm":12,"config":"intelligence index v4.3.2, reasoning version","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mercury-2-5","measured_date":"2026-09","observed_at":"2026-10-04","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12.3,"score_norm":12.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inception/mercury-2.5","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71.7,"score_norm":null,"config":"artificial analysis long context reasoning","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":22.7,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":19.7,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":4.52,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"cyberbench_v1_1","score_raw":45.48,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":34,"score_norm":null,"config":"ten-tool-call budget, vendor launch chart","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"emb","score_raw":8.79,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":18.89,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79,"score_norm":76.06,"config":"config not stated, vendor launch chart","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"harveys_legal_agent_benchmark","score_raw":0,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":11.8,"score_norm":14.41,"config":"artificial analysis, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77,"score_norm":null,"config":"config not stated, vendor launch chart","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"ioi","score_raw":2.44,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"legal_bench","score_raw":83.06,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"legal_research_bench","score_raw":4.33,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"medcode","score_raw":31.33,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"medscribe","score_raw":55.09,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"proofbench","score_raw":3,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/mercury-2-5","measured_date":null,"observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"proofbench_v1_1","score_raw":3,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"public_benefits_bench_v1_1","score_raw":45.26,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":38.5,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/inception/mercury-2.5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":38,"score_norm":null,"config":"config not stated, vendor launch chart","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"skillsbench","score_raw":18.06,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"tau3_bench_telecom","score_raw":96,"score_norm":null,"config":"config otherwise not stated, telecom domain, vendor launch chart","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"tax_agent_bench","score_raw":12.41,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"4.0, vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":true,"excluded":null},{"benchmark":"terminal_bench_2_1_vals","score_raw":34.1,"score_norm":null,"config":"terminal-bench 2.1, vals ai run","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/mercury-2-5","measured_date":"2026-09-08","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":0,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":8.86,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench_v1_1","score_raw":3.68,"score_norm":null,"config":"vals ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/inception_mercury-2.5","measured_date":"2026-10-03","observed_at":"2026-10-04","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inception/mercury-2.5-preview","publisher":"OpenRouter (listed 2026-08-31)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-04T06:04:33.895Z"},{"id":"mimo-2.5","display_name":"MiMo-V2.5","family":{"id":"mimo","vendor":"xiaomi"},"generation":"2.5","size_class":"frontier","params_total_b":310.78,"params_active_b":null,"params_exact":310775040000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":622,"int8_gb":329,"int4_gb":186,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.93,"expected_int8_gb":307,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"XiaomiMiMo/MiMo-V2.5","openrouter_id":"xiaomi/mimo-v2.5","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","audio","image","video"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1050000,"release_date":"2026-04-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mimo-v2.5"],"intelligence":{"index":68.33,"inputs":[{"benchmark":"arena_elo","score_raw":1433.6,"score_norm":68.63,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":81.57,"score_norm":79.68,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"overall, vals ai independent run"},{"benchmark":"hle","score_raw":27.2,"score_norm":37.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","config":"artificial analysis independent run, no tools"},{"benchmark":"mmlu_pro","score_raw":82.93,"score_norm":71.55,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"overall, vals ai independent run"},{"benchmark":"tau_bench","score_raw":90.64,"score_norm":94.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","config":"artificial analysis independent run, tau2, with tools"},{"benchmark":"terminal_bench","score_raw":63.67,"score_norm":69.02,"trust":2,"source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","config":"2.1, artificial analysis independent run"}],"coverage":0.5635,"subset_bias":{"offset":2.37,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.14,"completion_usd_per_m":0.28,"cache_read_usd_per_m":0.0028,"context_length":1050000,"source":"openrouter","source_model_id":"xiaomi/mimo-v2.5","as_of":"2026-10-06","blended_usd_per_m":0.17500000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":621.56,"kv_cache_gb_per_8k":1.208,"min_serve_gb":653.846,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":329.427,"kv_cache_gb_per_8k":1.208,"min_serve_gb":347.106,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":315.04,"kv_cache_gb_per_8k":1.208,"min_serve_gb":332,"method":"vendor","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5/tree/main"},{"quant":"int4","weights_gb":186.468,"kv_cache_gb_per_8k":1.208,"min_serve_gb":196.999,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":329.427,"kv_cache_gb_per_8k":1.208,"min_serve_gb":347.106,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":15.8,"score_norm":15.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":56.8,"score_norm":56.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.2,"score_norm":25.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"artificial analysis aa-lcr v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1433.6,"score_norm":68.63,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1233.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1437.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1230,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1150,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1257,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1257,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1262,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.57,"score_norm":79.68,"config":"overall, vals ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.95,"score_norm":84.44,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.2,"score_norm":37.06,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.14,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.93,"score_norm":71.55,"config":"overall, vals ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.43,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":56.1,"score_norm":85.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":90.64,"score_norm":94.19,"config":"artificial analysis independent run, tau2, with tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.67,"score_norm":69.02,"config":"2.1, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-5-0424","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":65.8,"score_norm":71.53,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5","publisher":"OpenRouter (listed 2026-04-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mimo-2.5-pro","display_name":"MiMo-V2.5-Pro","family":{"id":"mimo","vendor":"xiaomi"},"generation":"2.5","size_class":"frontier-pro","params_total_b":1023.18,"params_active_b":null,"params_exact":1023182841600,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2046,"int8_gb":1085,"int4_gb":614,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.3,"expected_int8_gb":323,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"XiaomiMiMo/MiMo-V2.5-Pro","openrouter_id":"xiaomi/mimo-v2.5-pro","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1050000,"release_date":"2026-04-22","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mimo-v2.5-pro"],"intelligence":{"index":69.83,"inputs":[{"benchmark":"arena_elo","score_raw":1467.5,"score_norm":83.74,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":86.6,"score_norm":86.76,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=mimo-v2-5-pro","config":"artificial analysis harness, gpqa diamond, reasoning"},{"benchmark":"hle","score_raw":35.7,"score_norm":49.56,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=mimo-v2-5-pro","config":"artificial analysis harness, humanity's last exam, reasoning"},{"benchmark":"mmlu_pro","score_raw":68.5,"score_norm":47.5,"trust":1,"source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","config":null},{"benchmark":"swe_bench_verified","score_raw":78.9,"score_norm":76.56,"trust":1,"source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","config":null},{"benchmark":"terminal_bench","score_raw":65.2,"score_norm":70.82,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1?models=mimo-v2-5-pro","config":"artificial analysis harness, reasoning, terminal-bench v2.1"}],"coverage":0.6746,"subset_bias":{"offset":0.7,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.435,"completion_usd_per_m":0.87,"cache_read_usd_per_m":0.0036,"context_length":1050000,"source":"openrouter","source_model_id":"xiaomi/mimo-v2.5-pro","as_of":"2026-10-06","blended_usd_per_m":0.54375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2046.36,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":2152.201,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1084.571,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":1142.323,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1033.39,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":1088.583,"method":"vendor","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/tree/main"},{"quant":"int4","weights_gb":613.908,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":648.127,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1084.571,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":1142.323,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":21.3,"score_norm":21.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":60.2,"score_norm":60.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26,"score_norm":26,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1467.5,"score_norm":83.74,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1477.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1260,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1163,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1266,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1252,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1274,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.6,"score_norm":86.76,"config":"artificial analysis harness, gpqa diamond, reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=mimo-v2-5-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.7,"score_norm":58.73,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":35.7,"score_norm":49.56,"config":"artificial analysis harness, humanity's last exam, reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=mimo-v2-5-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48,"score_norm":67.65,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"mmlu_pro","score_raw":68.5,"score_norm":47.5,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":57.2,"score_norm":87,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.9,"score_norm":76.56,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":65.2,"score_norm":70.82,"config":"artificial analysis harness, reasoning, terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-v2-1?models=mimo-v2-5-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":68.4,"score_norm":74.59,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.5-pro","publisher":"OpenRouter (listed 2026-04-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mimo-2.6-flash","display_name":"MiMo-V2.6-Flash","family":{"id":"mimo","vendor":"xiaomi"},"generation":"2.6","size_class":"frontier-flash","params_total_b":310.76,"params_active_b":null,"params_exact":310756322688,"params_source":"hf_safetensors","vram_est":{"bf16_gb":622,"int8_gb":329,"int4_gb":186,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"XiaomiMiMo/MiMo-V2.6-Flash-RL","openrouter_id":"xiaomi/mimo-v2.6-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1050000,"release_date":"2026-09-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mimo-v2.6-flash","mimo-v2.6-flash-rl"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.14,"completion_usd_per_m":0.28,"cache_read_usd_per_m":0.0028,"context_length":1050000,"source":"openrouter","source_model_id":"xiaomi/mimo-v2.6-flash","as_of":"2026-10-06","blended_usd_per_m":0.17500000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":621.52,"kv_cache_gb_per_8k":1.208,"min_serve_gb":653.804,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":329.406,"kv_cache_gb_per_8k":1.208,"min_serve_gb":347.084,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":186.456,"kv_cache_gb_per_8k":1.208,"min_serve_gb":196.987,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":172.933,"kv_cache_gb_per_8k":1.208,"min_serve_gb":182.788,"method":"vendor","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/tree/main"},{"quant":"int8","weights_gb":329.406,"kv_cache_gb_per_8k":1.208,"min_serve_gb":347.084,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":38,"score_norm":38,"config":"intelligence index v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-6-flash","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.9,"score_norm":37.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-flash","measured_date":null,"observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"agents_last_exam","score_raw":27.6,"score_norm":null,"config":"thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1451.8,"score_norm":76.74,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1247.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":52.3,"score_norm":null,"config":"1.0.6, thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":95.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":67.9,"score_norm":null,"config":"1.1, thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"exploitbench","score_raw":25.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"exploitgym","score_raw":6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":61.2,"score_norm":null,"config":"thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_code_bench","score_raw":61.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_cyber_bench","score_raw":77.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_visualcoding","score_raw":71.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":80.8,"score_norm":null,"config":"thinking, tools, verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":26,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"sec_bench_pro","score_raw":47.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.6,"score_norm":97.18,"config":"2.1, thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.8,"score_norm":28,"config":"4.0, thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.6,"score_norm":97.18,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":73.6,"score_norm":null,"config":"thinking, tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-flash","publisher":"OpenRouter (listed 2026-09-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mimo-2.6-pro","display_name":"MiMo-V2.6-Pro","family":{"id":"mimo","vendor":"xiaomi"},"generation":"2.6","size_class":"frontier-pro","params_total_b":1024.22,"params_active_b":null,"params_exact":1024216603392,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2048,"int8_gb":1086,"int4_gb":615,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"XiaomiMiMo/MiMo-V2.6-Pro-RL","openrouter_id":"xiaomi/mimo-v2.6-pro","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1050000,"release_date":"2026-09-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mimo-v2.6-pro","mimo-v2.6-pro-rl"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.435,"completion_usd_per_m":0.87,"cache_read_usd_per_m":0.0036,"context_length":1050000,"source":"openrouter","source_model_id":"xiaomi/mimo-v2.6-pro","as_of":"2026-10-06","blended_usd_per_m":0.54375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2048.44,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":2154.385,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1085.673,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":1143.48,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":614.532,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":648.782,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":566.048,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":597.874,"method":"vendor","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/tree/main"},{"quant":"int8","weights_gb":1085.673,"kv_cache_gb_per_8k":3.5232,"min_serve_gb":1143.48,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":46,"score_norm":46,"config":"reasoning version, v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mimo-v2-6-pro","measured_date":"2026-09","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":46.3,"score_norm":46.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":46.32,"score_norm":46.32,"config":"september 2026, v4.3","trust":1,"source_type":"vendor_claim","source_url":"https://mimo.xiaomi.com/mimo-v2-6","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"agents_last_exam","score_raw":31.6,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1480.1,"score_norm":89.35,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1245.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1618.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":53.1,"score_norm":null,"config":"1.0.6, vendor-run agent eval","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"code_migration","score_raw":43.01,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"cyberbench_v1_1","score_raw":72.86,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":94,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":71.9,"score_norm":null,"config":"1.1, vendor-run agent eval","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1347,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1323,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1340,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1325,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1345,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1283,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1317,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"emb","score_raw":62.86,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"exploitbench","score_raw":47.9,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"exploitgym","score_raw":17.8,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":57.34,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1673,"score_norm":null,"config":"2.1, elo","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"harveys_legal_agent_benchmark","score_raw":10.83,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"ioi","score_raw":39.33,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":62,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"legal_research_bench","score_raw":47.12,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"medcode","score_raw":44.97,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"medscribe","score_raw":88.31,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_code_bench","score_raw":63.2,"score_norm":null,"config":"in-house","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_cyber_bench","score_raw":80.2,"score_norm":null,"config":"in-house","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_aa_lcr","score_raw":86.3,"score_norm":null,"config":"artificial analysis, long context reasoning","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_aa_omniscience_accuracy","score_raw":34.8,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_aa_omniscience_non_hallucination_rate","score_raw":59.4,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_critpt","score_raw":26.6,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_gdpval_aa","score_raw":58.9,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_hle","score_raw":49.4,"score_norm":null,"config":"artificial analysis, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_intelligence_index","score_raw":46.3,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_v2_6_pro_scicode","score_raw":60.9,"score_norm":null,"config":"artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/xiaomi/mimo-v2.6-pro","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mimo_visualcoding","score_raw":72.3,"score_norm":null,"config":"in-house","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"mysterymechanism","score_raw":15.31,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":82,"score_norm":null,"config":"thinking, tools, vendor-run, verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":0.5,"score_norm":null,"config":"1 of 200 fully resolved, reasoning on, temperature 1.0, vals independent, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":26.5,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"proofbench_v1_1","score_raw":70,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"public_benefits_bench_v1_1","score_raw":68.94,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"sage","score_raw":45.05,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"sec_bench_pro","score_raw":66.3,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"sre_bench","score_raw":3.05,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"tax_agent_bench","score_raw":64.94,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"config":"2.1, reasoning on, temperature 1.0, vals independent, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.31,"score_norm":30.95,"config":"4.0, reasoning on, temperature 1.0, vals independent, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":89.9,"score_norm":99.88,"config":"2.1, thinking, tools, vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.9,"score_norm":35.18,"config":"4.0, thinking, tools, vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":89.9,"score_norm":99.88,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench_science","score_raw":2.86,"score_norm":null,"config":"reasoning on, temperature 1.0, vals independent, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":76.9,"score_norm":null,"config":"vendor-run","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL","measured_date":"2026-09-21","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vals_index","score_raw":55.2,"score_norm":null,"config":"max output 128000, official xiaomi api, reasoning enabled, temperature 1.0, top_p 0.95","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"vibe_code_bench_v1_1","score_raw":85.22,"score_norm":null,"config":"reasoning on, temperature 1.0, vals, xiaomi api","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/xiaomi_mimo-v2.6-pro","measured_date":"2026-09-22","observed_at":"2026-10-01","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro","publisher":"OpenRouter (listed 2026-09-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mimo-2.6-pro-ultraspeed","display_name":"MiMo-V2.6-Pro-UltraSpeed","family":{"id":"mimo","vendor":"xiaomi"},"generation":"2.6","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"xiaomi/mimo-v2.6-pro-ultraspeed","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mimo-v2.6-pro-ultraspeed"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":4.35,"completion_usd_per_m":8.7,"cache_read_usd_per_m":0.036,"context_length":1048576,"source":"openrouter","source_model_id":"xiaomi/mimo-v2.6-pro-ultraspeed","as_of":"2026-10-06","blended_usd_per_m":5.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#xiaomi/mimo-v2.6-pro-ultraspeed","publisher":"OpenRouter (listed 2026-09-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-01","display_name":"MiniMax-01","family":{"id":"minimax","vendor":"MiniMax"},"generation":"01","size_class":"frontier","params_total_b":456.09,"params_active_b":null,"params_exact":456089655296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":912,"int8_gb":483,"int4_gb":274,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"MiniMaxAI/MiniMax-Text-01","openrouter_id":"minimax/minimax-01","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":1000192,"release_date":"2025-01-15","release_status":"released","knowledge_cutoff":"2024-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["minimax-text-01"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":1.1,"cache_read_usd_per_m":null,"context_length":1000192,"source":"openrouter","source_model_id":"minimax/minimax-01","as_of":"2026-10-06","blended_usd_per_m":0.42500000000000004,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":912.18,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":958.125,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":914.67,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":960.739,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01/tree/main"},{"quant":"fp8","weights_gb":483.455,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":507.963,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":273.654,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":287.672,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":483.455,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":507.963,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_hard","score_raw":89.1,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":87.8,"score_norm":null,"config":"0-shot cot, f1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":54.4,"score_norm":41.41,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":94.8,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":86.9,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":89.1,"score_norm":null,"config":"avg","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"longbench_v2","score_raw":56.5,"score_norm":null,"config":"overall, with cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":77.4,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":71.7,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":88.5,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":75.7,"score_norm":59.5,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"simpleqa","score_raw":23.7,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-Text-01","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-01","publisher":"OpenRouter (listed 2025-01-15)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m1-reasoning","display_name":"MiniMax-M1","family":{"id":"minimax","vendor":"MiniMax"},"generation":"1","size_class":"frontier-m","params_total_b":456.09,"params_active_b":45.9,"params_exact":456089655296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":912,"int8_gb":483,"int4_gb":274,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.29,"expected_int8_gb":139,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"MiniMaxAI/MiniMax-M1-80k","openrouter_id":"minimax/minimax-m1","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":1000000,"release_date":"2025-06","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["minimax-m1","minimax-m1-40k-hf","minimax-m1-456b","minimax-m1-80k"],"intelligence":{"index":42.35,"inputs":[{"benchmark":"aime","score_raw":61,"score_norm":51.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.61 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1363.9,"score_norm":37.56,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":69.7,"score_norm":62.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.696969696969697 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":8.9,"score_norm":10.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0889712696941613 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":81.1,"score_norm":68.5,"trust":2,"source_url":"https://arxiv.org/html/2506.13585v1","config":"minimax-m1-80k, table 2 core benchmarks, temperature=1.0, top_p=0.95 [raw: mmlu-pro]"},{"benchmark":"tau_bench","score_raw":34.21,"score_norm":18.95,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.342105263157895 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":1.96,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.55,"completion_usd_per_m":2.2,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"minimax/minimax-m1","as_of":"2026-10-06","blended_usd_per_m":0.9625000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":912.18,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":958.125,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":912.212,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":958.158,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M1-80k/tree/main"},{"quant":"fp8","weights_gb":483.455,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":507.963,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":273.654,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":287.672,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":483.455,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":507.963,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.7,"score_norm":11.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":17.86,"score_norm":17.86,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":56.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.566666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":61,"score_norm":51.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.61 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":76.9,"score_norm":71.13,"config":"aime 2025, avg pass rate over 32 samples, no tools, temperature=1.0, top_p=0.95 [raw: aime 2025]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2506.13585v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1363.9,"score_norm":37.56,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1363.2,"score_norm":37.25,"config":"35247 votes, arena entry minimax-m1, leaderboard text-overall-style_control, rank 185, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":69.7,"score_norm":62.96,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.696969696969697 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":70,"score_norm":63.38,"config":"minimax-m1-80k, no tools [raw: gpqa diamond], table 2 core benchmarks, temperature=1.0, top_p=0.95","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2506.13585v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.9,"score_norm":10.15,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0889712696941613 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.4,"score_norm":9.41,"config":"no external tools [raw: hle (humanity's last exam)], text-only hle subset","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2506.13585v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.77,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.417687074829932 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":71.11,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.711111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.1,"score_norm":68.5,"config":"minimax-m1-80k, table 2 core benchmarks, temperature=1.0, top_p=0.95 [raw: mmlu-pro]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2506.13585v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":37.38,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.373842592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":34.21,"score_norm":18.95,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.342105263157895 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":63.5,"score_norm":58,"config":"40-step interaction cap [raw: tau-bench (retail)], gpt-4.1 as user model, no custom tools, retail domain only (no published average)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2506.13585v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m1-80k","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/MiniMax-AI/MiniMax-M1","publisher":"MiniMax"},{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m1","publisher":"OpenRouter (listed 2025-06-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m2.7","display_name":"MiniMax-M2.7","family":{"id":"minimax","vendor":"MiniMax"},"generation":"2.7","size_class":"frontier-m","params_total_b":228.69,"params_active_b":null,"params_exact":228689764864,"params_source":"hf_safetensors","vram_est":{"bf16_gb":457,"int8_gb":242,"int4_gb":137,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.33,"expected_int8_gb":323,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"MiniMaxAI/MiniMax-M2.7","openrouter_id":"minimax/minimax-m2.7","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2026-03-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["minimax-m2","minimax-m2-her","minimax-m2.1","minimax-m2.5"],"superseded_by":null,"aliases":[],"intelligence":{"index":69.82,"inputs":[{"benchmark":"aime","score_raw":91.04,"score_norm":88.8,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.734, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1415,"score_norm":60.34,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":86.62,"score_norm":86.79,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.953, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":29.61,"score_norm":40.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-7","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.29610750695088 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":80.43,"score_norm":67.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.39, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":73.8,"score_norm":70.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.968, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":84.8,"score_norm":86.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-7","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.847953216374269 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":48.69,"score_norm":51.4,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=3.066, task=overall, updated=2026-08-19"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.21,"completion_usd_per_m":0.84,"cache_read_usd_per_m":0.042,"context_length":204800,"source":"openrouter","source_model_id":"minimax/minimax-m2.7","as_of":"2026-10-06","blended_usd_per_m":0.3675,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":457.38,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":482.329,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":230.134,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":243.721,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7/tree/main"},{"quant":"int4","weights_gb":137.214,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":146.155,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":15.3,"score_norm":15.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":25.92,"score_norm":25.92,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":52.6,"score_norm":52.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.8,"score_norm":22.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.87,"score_norm":38.87,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.753333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.77,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.04,"score_norm":88.8,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.734, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":94.2,"score_norm":92.75,"config":"aime 2026","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2605.26494v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":11.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.1125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":10.62,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.106194690265487 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1415,"score_norm":60.34,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1416.3,"score_norm":60.92,"config":"62076 votes, arena entry deep-octo, leaderboard text-overall-style_control, rank 121, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1397,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1398.1,"score_norm":null,"config":"14198 votes, arena entry deep-octo, code/overall, leaderboard webdev-overall-raw, rank 67, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":77.8,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2605.26494v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.00571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1209,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1148,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1239,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1223,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1136,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1216,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1249,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1495,"score_norm":null,"config":"elo","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":32.99,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.329895 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.62,"score_norm":86.79,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.953, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.8,"score_norm":91.27,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2605.26494v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.37,"score_norm":87.85,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.873737373737374 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28,"score_norm":38.24,"config":"aa index v4.0 protocol, no-tool subset, pass@1","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2605.26494","measured_date":"2026-07-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":29.61,"score_norm":40.6,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.29610750695088 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":29.6,"score_norm":40.59,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.71,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.757142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":26.46,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, source value 0.264595103578154 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":3.33,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0333333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mle_bench_lite","score_raw":66.6,"score_norm":null,"config":"medal rate, 22 ml competitions","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mm_claw","score_raw":62.7,"score_norm":null,"config":"end-to-end","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.43,"score_norm":67.38,"config":"edition=mmlu pro v1, runner=platform, stderr=0.39, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.8,"score_norm":69.67,"config":"artificial analysis index v4.0 protocol, no tools, single sample (pass@1), thinking enabled","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2605.26494","measured_date":"2026-07-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"multi_swe_bench","score_raw":52.7,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2605.26494v1","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"nl2repo","score_raw":39.8,"score_norm":null,"config":"m2.5 = 26.6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.99,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.469907407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"skillsbench","score_raw":34.9,"score_norm":null,"config":"skillsbench v1.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":76.5,"score_norm":null,"config":"% resolved","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":56.2,"score_norm":85.33,"config":"reported as 56.22; m2.5 = 55.4","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.8,"score_norm":70.89,"config":"edition=swe-bench verified v1, runner=external, stderr=1.968, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.8,"score_norm":70.89,"config":"max output 196608, temperature 1.0, top_p 0.95, vals.ai independent evaluation","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":84.8,"score_norm":86.4,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.847953216374269 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":84.8,"score_norm":86.4,"config":"artificial analysis intelligence index, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.9,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0989690721649484 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":48.69,"score_norm":51.4,"config":"edition=terminal-bench 2.1, runner=external, stderr=3.066, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":57,"score_norm":61.18,"config":"terminal-bench 2.0","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2605.26494v1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":39.39,"score_norm":40.46,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.393939393939394 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.43,"score_norm":59.33,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.554307116104869 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":57,"score_norm":61.18,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"toolathlon","score_raw":46.3,"score_norm":null,"config":"accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"vibe_pro","score_raw":55.6,"score_norm":null,"config":"end-to-end project delivery (web/android/ios)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"wildclawbench","score_raw":33.8,"score_norm":null,"config":"overall","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.minimax.io/docs/api-reference/api-overview","publisher":"MiniMax"},{"url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.7","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.7","publisher":"OpenRouter (listed 2026-03-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m3-reasoning","display_name":"MiniMax-M3","family":{"id":"minimax","vendor":"MiniMax"},"generation":"3","size_class":"frontier-m","params_total_b":427.04,"params_active_b":null,"params_exact":427040140160,"params_source":"hf_safetensors","vram_est":{"bf16_gb":854,"int8_gb":453,"int4_gb":256,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.8,"expected_int8_gb":362,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"MiniMaxAI/Minimax-M3","openrouter_id":"minimax/minimax-m3","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text","image","video"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-06-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["minimax-m3"],"intelligence":{"index":72.98,"inputs":[{"benchmark":"arena_elo","score_raw":1440.1,"score_norm":71.52,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":92.68,"score_norm":95.32,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.443, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":38.97,"score_norm":54.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.389712696941613 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":67.26,"score_norm":60.58,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":84.22,"score_norm":73.7,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.364, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.938, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":88.89,"score_norm":91.85,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m3","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.888888888888889 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=0.749, task=overall, updated=2026-08-19"}],"coverage":0.8413,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.06,"context_length":1048576,"source":"openrouter","source_model_id":"minimax/minimax-m3","as_of":"2026-10-06","blended_usd_per_m":0.5249999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":854.08,"kv_cache_gb_per_8k":1.0066,"min_serve_gb":897.791,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":854.176,"kv_cache_gb_per_8k":1.0066,"min_serve_gb":897.891,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/Minimax-M3/tree/main"},{"quant":"fp8","weights_gb":452.662,"kv_cache_gb_per_8k":1.0066,"min_serve_gb":476.302,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":256.224,"kv_cache_gb_per_8k":1.0066,"min_serve_gb":270.042,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":452.662,"kv_cache_gb_per_8k":1.0066,"min_serve_gb":476.302,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":29.5,"score_norm":29.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":36.12,"score_norm":36.12,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":58.6,"score_norm":58.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.2,"score_norm":29.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45.4,"score_norm":45.4,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":80.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.803333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":1.35,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":10,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.1 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1440.1,"score_norm":71.52,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1442.7,"score_norm":72.68,"config":"41023 votes, arena entry minimax-m3, leaderboard text-overall-style_control, rank 76, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1434.8,"score_norm":null,"config":"6263 votes, arena entry minimax-m3, document/overall, leaderboard document-overall-raw, rank 27, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1434.8,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1472.7,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1236.6,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1236.8,"score_norm":null,"config":"12732 votes, arena entry minimax-m3, leaderboard vision-overall-style_control, rank 51, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1482,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1487.6,"score_norm":null,"config":"10721 votes, arena entry minimax-m3, code/overall, leaderboard webdev-overall-raw, rank 37, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":15.9,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.1589777434670109 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0371428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1224,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1158,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1171,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1254,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1244,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1196,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1225,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1167,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1204,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1162,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1253,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1192,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1264,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":32.11,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.3210981796478663 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":44.35,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.44346500000000005 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.68,"score_norm":95.32,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.443, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.929292929292929 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93,"score_norm":95.77,"config":"article states \"gpqa diamond +6 (87% to 93%)\" vs minimax-m2.7, artificial analysis intelligence index eval suite, i.e. m3 = 93% [raw: gpqa diamond], no tools, reasoning enabled","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/minimax-m3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":88.41,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.884064264003474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":38.97,"score_norm":54.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.389712696941613 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37,"score_norm":51.47,"config":"article states \"hle +9 points (28% to 37%)\" vs minimax-m2.7, artificial analysis eval, i.e. m3 = 37% [raw: humanity's last exam (hle)], reasoning enabled, text-only / no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/minimax-m3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":82.86,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.828571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"kernelbench_hard","score_raw":28.8,"score_norm":null,"config":"official minimax m3 report","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":67.26,"score_norm":60.58,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mcp_atlas","score_raw":74.2,"score_norm":null,"config":"official minimax m3 report","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":17.22,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.172222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.22,"score_norm":73.7,"config":"edition=mmlu pro v1, runner=platform, stderr=0.364, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.55,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.785549132947977 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":70.06,"score_norm":null,"config":"task completion rate at max steps=200 (68.70 at 100 steps), verified","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":45.37,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.453703703703704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":59,"score_norm":90,"config":"official minimax m3 report","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":59,"score_norm":90,"config":"evaluated on internal infrastructure using claude code as the scaffolding. testing logic is aligned with the official evaluation.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"config":"edition=swe-bench verified v1, runner=external, stderr=1.938, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.5,"score_norm":78.33,"config":"between deepseek-v4-pro 80.6 and kimi-k2.6 80.2), i.e. submitted/self-reported run rat [raw: swe-bench verified], official swe-bench verified leaderboard row \"minimaxai/minimax-m3\" (80.5, score carries an asterisk on the board","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.5,"score_norm":78.33,"config":"evaluated on internal infrastructure using claude code as the scaffolding. each test was run 4 times and the average was taken.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_fficiency","score_raw":34.8,"score_norm":null,"config":"official minimax m3 report","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":88.89,"score_norm":91.85,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.888888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76,"score_norm":74.67,"config":"auto-routing endpoint (parasail endpoint = 74.6%), endpoint-quality eval, not a canonical tau-bench domain-average publication [raw: tau-bench], openrouter \"exacto benchmarks\" tau-bench","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/minimax/minimax-m3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":15.26,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.152577319587629 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"config":"edition=terminal-bench 2.1, runner=external, stderr=0.749, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":42.42,"score_norm":44.02,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.424242424242424 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":65.17,"score_norm":70.79,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.651685393258427 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":66,"score_norm":71.76,"config":"terminus 2 harness","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"video_mme","score_raw":84.6,"score_norm":null,"config":"512 frames","trust":1,"source_type":"vendor_claim","source_url":"https://www.minimax.io/blog/minimax-m3","measured_date":"2026-06-01","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.minimax.io/docs/api-reference/api-overview","publisher":"MiniMax"},{"url":"https://www.minimax.io/blog/minimax-m3","publisher":"MiniMax"},{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m3","publisher":"OpenRouter (listed 2026-05-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m3.1-flash","display_name":"MiniMax M3.1 Flash Preview","family":{"id":"minimax","vendor":"MiniMax"},"generation":"3.1","size_class":"frontier-m-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-27","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["minimax-m3.1-flash-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://llm-releases.com/changelog","publisher":"LLM Releases"},{"url":"https://www.llmreference.com/changelog/2026-09","publisher":"LLM Reference"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"ministral-1-8b","display_name":"ministral-8b","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"1","size_class":"frontier","params_total_b":8,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":16,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/ministral-8b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-10-17","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-8b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b","publisher":"OpenRouter (listed 2024-10-17)"}],"last_verified":"2026-08-25","last_enriched_at":null},{"id":"ministral-3-14b","display_name":"Ministral 3 14B","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"3","size_class":"14b","params_total_b":14,"params_active_b":14,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":28,"int8_gb":14.8,"int4_gb":8.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.24,"expected_int8_gb":18.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":26.42,"inputs":[{"benchmark":"aime","score_raw":30,"score_norm":12.5,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=ministral-3-14b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":57.2,"score_norm":45.35,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=ministral-3-14b","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":4.6,"score_norm":3.82,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=ministral-3-14b","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"mmlu_pro","score_raw":69.3,"score_norm":48.83,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=ministral-3-14b","config":"artificial analysis harness, mmlu-pro"},{"benchmark":"tau_bench","score_raw":27.2,"score_norm":9.6,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=ministral-3-14b","config":"artificial analysis harness, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":4.5,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=ministral-3-14b","config":"artificial analysis harness, terminal-bench hard"}],"coverage":0.5635,"subset_bias":{"offset":1.2,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":28,"kv_cache_gb_per_8k":null,"min_serve_gb":29.96,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":14.84,"kv_cache_gb_per_8k":null,"min_serve_gb":16.4,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.4,"kv_cache_gb_per_8k":null,"min_serve_gb":9.96,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":14.84,"kv_cache_gb_per_8k":null,"min_serve_gb":16.4,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":25.4,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":43.43,"bw_limited_tok_s":26.06,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.56,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":19.5,"bw_limited_tok_s":11.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":125.55,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":239.29,"bw_limited_tok_s":143.57,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":30,"score_norm":12.5,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":85,"score_norm":81.25,"config":"14b reasoning variant, aime 2025","trust":1,"source_type":"vendor_claim","source_url":"https://mistral.ai/news/mistral-3","measured_date":"2025-12-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.2,"score_norm":45.35,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.6,"score_norm":3.82,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.3,"score_norm":48.83,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":27.2,"score_norm":9.6,"config":"artificial analysis harness, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.5,"score_norm":0,"config":"artificial analysis harness, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/mistral-3/","publisher":"Mistral AI"},{"url":"https://arxiv.org/html/2601.08584","publisher":"arXiv"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"ministral-3-3b-2512-reasoning","display_name":"Ministral-3-3B-Reasoning-2512","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"3","size_class":"3b","params_total_b":4.25,"params_active_b":4.25,"params_exact":4251743232,"params_source":"hf_safetensors","vram_est":{"bf16_gb":8.5,"int8_gb":4.5,"int4_gb":2.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Ministral-3-3B-Reasoning-2512","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-3-3b-reasoning-2512"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":8.5,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":10.372,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":4.505,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":6.377,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.55,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":4.422,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.505,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":6.377,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":79.05,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":4.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":143.06,"bw_limited_tok_s":85.84,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":37.11,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":4.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":64.24,"bw_limited_tok_s":38.54,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":321.09,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":4.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":788.24,"bw_limited_tok_s":472.94,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":77.5,"score_norm":71.88,"config":"2024, ministral 3 3b column, ministral 3 technical report table 5 (reasoning models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":72.1,"score_norm":65.12,"config":"2025, ministral 3 3b column, ministral 3 technical report table 5 (reasoning models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":77.5,"score_norm":71.88,"config":"aime 2024, ministral 3 3b row (card reports 0.775), reasoning table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":72.1,"score_norm":65.12,"config":"aime 2025, ministral 3 3b row (card reports 0.721), reasoning table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":53.4,"score_norm":40,"config":"ministral 3 3b column, ministral 3 technical report table 5 (reasoning models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":53.4,"score_norm":40,"config":"ministral 3 3b row (card reports 0.534), reasoning table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":51.7,"score_norm":null,"config":"2025, ministral 3 3b column, ministral 3 technical report table 5 (reasoning models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":54.8,"score_norm":null,"config":"ministral 3 3b column, ministral 3 technical report table 5 (reasoning models), v6","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":54.8,"score_norm":null,"config":"ministral 3 3b row (card reports 0.548), reasoning table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"phybench","score_raw":15,"score_norm":null,"config":"ministral 3 3b column, ministral 3 technical report table 5 (reasoning models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/collections/mistralai/ministral-3","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"ministral-3-8b","display_name":"Ministral 3 8B","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"3","size_class":"8b","params_total_b":8,"params_active_b":8,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":16,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.72,"expected_int8_gb":14.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":20.71,"inputs":[{"benchmark":"aime","score_raw":31.67,"score_norm":14.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"2025, artificial analysis independent evaluation"},{"benchmark":"gpqa_diamond","score_raw":47.07,"score_norm":31.08,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":4.26,"score_norm":3.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":64.15,"score_norm":40.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"artificial analysis independent evaluation"},{"benchmark":"tau_bench","score_raw":26.61,"score_norm":8.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"artificial analysis independent evaluation, tau2/telecom"},{"benchmark":"terminal_bench","score_raw":4.12,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-8b","config":"2.1, artificial analysis independent evaluation"}],"coverage":0.5635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16,"kv_cache_gb_per_8k":null,"min_serve_gb":17.32,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":8.48,"kv_cache_gb_per_8k":null,"min_serve_gb":9.8,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.8,"kv_cache_gb_per_8k":null,"min_serve_gb":6.12,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.48,"kv_cache_gb_per_8k":null,"min_serve_gb":9.8,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":76,"bw_limited_tok_s":45.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":20.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34.13,"bw_limited_tok_s":20.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.8,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":418.75,"bw_limited_tok_s":251.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":25.67,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"agieval","score_raw":59.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":31.67,"score_norm":14.59,"config":"2025, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":78.7,"score_norm":73.38,"config":"aime25, reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":50.9,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu_lite","score_raw":66.32,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":47.07,"score_norm":31.08,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.8,"score_norm":58.87,"config":"reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.26,"score_norm":3.32,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":29.12,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":30.26,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":61.6,"score_norm":null,"config":"reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":87.6,"score_norm":null,"config":"instruct, maj@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":76.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":64.15,"score_norm":40.25,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":79.3,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":45.95,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multilingual_mmlu","score_raw":70.6,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":26.61,"score_norm":8.81,"config":"artificial analysis independent evaluation, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.12,"score_norm":0,"config":"2.1, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"triviaqa","score_raw":68.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildbench","score_raw":66.8,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/mistral-3/","publisher":"Mistral AI"},{"url":"https://arxiv.org/html/2601.08584","publisher":"arXiv"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"ministral-3-8b-2512-reasoning","display_name":"Ministral-3-8B-Reasoning-2512","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"3","size_class":"8b","params_total_b":8.92,"params_active_b":8.92,"params_exact":8918026240,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.8,"int8_gb":9.5,"int4_gb":5.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Ministral-3-8B-Reasoning-2512","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-3-8b-reasoning-2512"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":17.84,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":19.981,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.836,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":19.977,"method":"vendor","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512/tree/main"},{"quant":"fp8","weights_gb":9.455,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":11.596,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.352,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":7.493,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.455,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":11.596,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":68.16,"bw_limited_tok_s":40.9,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.03,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":30.61,"bw_limited_tok_s":18.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":183.9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":375.56,"bw_limited_tok_s":225.34,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":86,"score_norm":82.5,"config":"2024, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":78.7,"score_norm":73.38,"config":"aime25, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.8,"score_norm":58.87,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":61.6,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/collections/mistralai/ministral-3","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"ministral-8B-8b-instruct","display_name":"Ministral-8B-Instruct","family":{"id":"ministral","vendor":"Mistral AI"},"generation":"8B","size_class":"8b","params_total_b":8,"params_active_b":8,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":16,"int8_gb":8.5,"int4_gb":4.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2024-10-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-8b-instruct"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16,"kv_cache_gb_per_8k":null,"min_serve_gb":17.32,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":8.48,"kv_cache_gb_per_8k":null,"min_serve_gb":9.8,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.8,"kv_cache_gb_per_8k":null,"min_serve_gb":6.12,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.48,"kv_cache_gb_per_8k":null,"min_serve_gb":9.8,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":43.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":76,"bw_limited_tok_s":45.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":20.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":34.13,"bw_limited_tok_s":20.47,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":200.8,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":418.75,"bw_limited_tok_s":251.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://llm-stats.com/models/compare/mistral-small-3.1-24b-base-2503-vs-ministral-8b-instruct-2410","publisher":"LLM Stats"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"mistral-1-nemo","display_name":"Mistral Nemo","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"1","size_class":"frontier","params_total_b":12.25,"params_active_b":12.25,"params_exact":12247782400,"params_source":"hf_safetensors","vram_est":{"bf16_gb":24.5,"int8_gb":13,"int4_gb":7.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Mistral-Nemo-Instruct-2407","openrouter_id":"mistralai/mistral-nemo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-07-19","release_status":"released","knowledge_cutoff":"2024-04-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-nemo","mistral-nemo-instruct-2407"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.019,"completion_usd_per_m":0.03,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"mistralai/mistral-nemo","as_of":"2026-10-06","blended_usd_per_m":0.02175,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":24.5,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":27.067,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":24.496,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":27.063,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407/tree/main"},{"quant":"fp8","weights_gb":12.985,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":15.327,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.35,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":9.692,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":12.985,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":15.327,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":28.92,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":12.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":49.63,"bw_limited_tok_s":29.78,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.19,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.29,"bw_limited_tok_s":13.37,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":140.95,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12.25,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":273.47,"bw_limited_tok_s":164.08,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"commonsenseqa","score_raw":70.4,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":83.5,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":68,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"naturalquestions","score_raw":31.2,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"openbookqa","score_raw":60.6,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"triviaqa","score_raw":73.8,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":50.3,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":76.8,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-nemo","publisher":"OpenRouter (listed 2024-07-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-2501-small-24b-instruct","display_name":"Mistral Small 3","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"2501","size_class":"24b","params_total_b":23.57,"params_active_b":23.57,"params_exact":23572403200,"params_source":"hf_safetensors","vram_est":{"bf16_gb":47.1,"int8_gb":25,"int4_gb":14.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.44,"expected_int8_gb":11.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Small-24B-Instruct-2501","openrouter_id":"mistralai/mistral-small-24b-instruct-2501","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2025-01-30","release_status":"released","knowledge_cutoff":"2023-10-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-1-small-24b-2501-instruct","mistral-3-small-24b-2501-instruct","mistral-small-24b-instruct-2501"],"intelligence":{"index":15.26,"inputs":[{"benchmark":"aime","score_raw":4.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3","config":"2025, artificial analysis independent run"},{"benchmark":"arena_elo","score_raw":1274.4,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":46.16,"score_norm":29.8,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3","config":"artificial analysis independent run"},{"benchmark":"hle","score_raw":3.76,"score_norm":2.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3","config":"artificial analysis independent run, no tools"},{"benchmark":"livebench","score_raw":42.5,"score_norm":5.56,"trust":3,"source_url":"https://livebench.ai/table_2024_11_25.csv","config":"global average of the 6 category averages, leaderboard row mistral-small-2501, livebench 2024-11-25 release"},{"benchmark":"mmlu_pro","score_raw":66.3,"score_norm":43.83,"trust":1,"source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","config":"5-shot cot, instruct"},{"benchmark":"tau_bench","score_raw":19.59,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3","config":"artificial analysis independent run, tau2, with tools"}],"coverage":0.7143,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.08,"cache_read_usd_per_m":null,"context_length":32768,"source":"openrouter","source_model_id":"mistralai/mistral-small-24b-instruct-2501","as_of":"2026-10-06","blended_usd_per_m":0.05750000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":47.14,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":50.839,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":47.145,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":50.844,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501/tree/main"},{"quant":"fp8","weights_gb":24.984,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":27.575,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":14.142,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":16.484,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":24.984,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":27.575,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":15.24,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":23.57,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":25.8,"bw_limited_tok_s":15.48,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":6.9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":23.57,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":11.58,"bw_limited_tok_s":6.95,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":78.58,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":23.57,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":142.13,"bw_limited_tok_s":85.28,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-24b-instruct-2501","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":4.33,"score_norm":0,"config":"2025, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1274.4,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1274.3,"score_norm":0,"config":"14681 votes, arena entry mistral-small-24b-instruct-2501, leaderboard text-overall-style_control, rank 286, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":87.3,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.16,"score_norm":29.8,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_main","score_raw":45.3,"score_norm":null,"config":"5-shot cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.76,"score_norm":2.59,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":84.8,"score_norm":null,"config":"instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":26.39,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":82.9,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":42.5,"score_norm":5.56,"config":"global average of the 6 category averages, leaderboard row mistral-small-2501, livebench 2024-11-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2024_11_25.csv","measured_date":"2024-11-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":25.19,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":70.6,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":66.3,"score_norm":43.83,"config":"5-shot cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":19.59,"score_norm":0,"config":"artificial analysis independent run, tau2, with tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"wildbench","score_raw":52.27,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-24b-instruct-2501","publisher":"OpenRouter (listed 2025-01-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-2502-saba","display_name":"Mistral Saba","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"2502","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-saba","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":32768,"release_date":"2025-02-17","release_status":"released","knowledge_cutoff":"2024-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-1-saba-2502","mistral-saba","mistral-saba-2502"],"intelligence":{"index":21.75,"inputs":[{"benchmark":"gpqa_diamond","score_raw":42.42,"score_norm":24.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-saba","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.424242424242424 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-saba","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.043 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":61.1,"score_norm":35.17,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro","config":"artificial analysis independent eval"}],"coverage":0.3889,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":0.6,"cache_read_usd_per_m":0.02,"context_length":32768,"source":"openrouter","source_model_id":"mistralai/mistral-saba","as_of":"2026-10-06","blended_usd_per_m":0.30000000000000004,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.5,"score_norm":6.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-saba","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.18,"score_norm":6.18,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.42,"score_norm":24.54,"config":"aa intelligence index eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.42,"score_norm":24.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.424242424242424 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"config":"aa eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.043 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":61.1,"score_norm":35.17,"config":"artificial analysis independent eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":24.07,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.240740740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-saba","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-saba","publisher":"OpenRouter (listed 2025-02-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-2512-large","display_name":"Mistral Large 3 2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"2512","size_class":"frontier-large","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-large-2512","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2025-12-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["mistral-1-large","mistral-2407-large"],"superseded_by":null,"aliases":["mistral-1-large-2512","mistral-3-large-2512","mistral-large-2512"],"intelligence":{"index":49.28,"inputs":[{"benchmark":"aime","score_raw":42.92,"score_norm":28.65,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.884, task=overall, updated=2026-04-16"},{"benchmark":"gpqa_diamond","score_raw":68.43,"score_norm":61.17,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.372, task=overall, updated=2026-08-19"},{"benchmark":"mmlu_pro","score_raw":79.82,"score_norm":66.37,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.472, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.205, task=overall, updated=2026-08-19"}],"coverage":0.5159,"subset_bias":{"offset":12.3,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":1.5,"cache_read_usd_per_m":0.05,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/mistral-large-2512","as_of":"2026-10-06","blended_usd_per_m":0.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":20.1,"score_norm":20.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":20.1,"score_norm":20.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.3,"score_norm":9.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-09-25","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.3,"score_norm":9.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":42.92,"score_norm":28.65,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.884, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1115,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1128,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-18","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1080,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1088,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-19","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1147,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1155,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-21","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1145,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1152,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-21","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1093,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1104,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-19","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":992,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1016,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-19","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1107,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1118,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-21","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1169,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512:batch","measured_date":null,"observed_at":"2026-09-19","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":68.43,"score_norm":61.17,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.372, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.82,"score_norm":66.37,"config":"edition=mmlu pro v1, runner=platform, stderr=0.472, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"config":"edition=swe-bench verified v1, runner=external, stderr=2.205, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2512","publisher":"OpenRouter (listed 2025-12-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-2603-small","display_name":"Mistral Small 4","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"2603","size_class":"frontier-small","params_total_b":119.4,"params_active_b":null,"params_exact":119401317952,"params_source":"hf_safetensors","vram_est":{"bf16_gb":239,"int8_gb":127,"int4_gb":71.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.07,"expected_int8_gb":135,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Small-4-119B-2603","openrouter_id":"mistralai/mistral-small-2603","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-03-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-1-small-2603","mistral-4-small-2603","mistral-small-2603","mistral-small-4-119b-2603"],"intelligence":{"index":41.44,"inputs":[{"benchmark":"gpqa_diamond","score_raw":76.87,"score_norm":73.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa intelligence index eval, reasoning_effort=high"},{"benchmark":"hle","score_raw":9.87,"score_norm":11.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa eval, no tools, reasoning_effort=high"},{"benchmark":"tau_bench","score_raw":41.23,"score_norm":28.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa eval, reasoning_effort=high, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":20.97,"score_norm":18.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa eval, reasoning_effort=high, terminal-bench 2.1"}],"coverage":0.3651,"subset_bias":{"offset":-7.16,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.6,"cache_read_usd_per_m":0.015,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/mistral-small-2603","as_of":"2026-10-06","blended_usd_per_m":0.26249999999999996,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":238.8,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":250.929,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":126.564,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":133.081,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":120.923,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":127.158,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Small-4-119B-2603/tree/main"},{"quant":"int4","weights_gb":71.64,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":75.411,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":126.564,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":133.081,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.58,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":119.4,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.57,"bw_limited_tok_s":2.59,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":32.57,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":119.4,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":56.11,"bw_limited_tok_s":33.67,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.8,"score_norm":0.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-2603","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":26.6,"score_norm":26.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-2603","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.3,"score_norm":11.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-2603","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":49.67,"score_norm":null,"config":"aa eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":538.9,"score_norm":null,"config":"aa eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.87,"score_norm":73.06,"config":"aa intelligence index eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.2,"score_norm":65.07,"config":"reasoning: high","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-4-119B-2603","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":9.87,"score_norm":11.57,"config":"aa eval, no tools, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":48.16,"score_norm":null,"config":"aa eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":56.82,"score_norm":null,"config":"aa eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":4.95,"score_norm":0,"config":"aa eval, banking, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":41.23,"score_norm":28.31,"config":"aa eval, reasoning_effort=high, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":17.42,"score_norm":14.61,"config":"aa eval, reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":20.97,"score_norm":18.79,"config":"aa eval, reasoning_effort=high, terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-2603","publisher":"OpenRouter (listed 2026-03-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3-14b-2512-instruct","display_name":"Ministral-3-14B-Instruct-2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"14b","params_total_b":13.95,"params_active_b":13.95,"params_exact":13945032240,"params_source":"hf_safetensors","vram_est":{"bf16_gb":27.9,"int8_gb":14.8,"int4_gb":8.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1,"expected_int8_gb":14.8,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Ministral-3-14B-Instruct-2512","openrouter_id":"mistralai/ministral-14b-2512","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-1-14b-2512","ministral-14b-2512","ministral-3-14b-2512","ministral-3-14b-instruct-2512","mistral-14-b-2512-14b"],"intelligence":{"index":21.01,"inputs":[{"benchmark":"aime","score_raw":30,"score_norm":12.5,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-14b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":57.17,"score_norm":45.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-14b","config":"artificial analysis harness"},{"benchmark":"hle","score_raw":4.63,"score_norm":3.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-14b","config":"artificial analysis harness, no tools"},{"benchmark":"tau_bench","score_raw":27.19,"score_norm":9.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-14b","config":"artificial analysis harness, tau2"},{"benchmark":"terminal_bench","score_raw":9.74,"score_norm":5.58,"trust":2,"source_url":"https://artificialanalysis.ai/models/ministral-3-14b","config":"artificial analysis harness, v2.1"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":0.2,"cache_read_usd_per_m":0.02,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/ministral-14b-2512","as_of":"2026-10-06","blended_usd_per_m":0.2,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":27.9,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":30.637,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":14.787,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.129,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":15.727,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":18.069,"method":"vendor","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512/tree/main"},{"quant":"int4","weights_gb":8.37,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":10.712,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":14.787,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.129,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":25.48,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":43.58,"bw_limited_tok_s":26.15,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":19.57,"bw_limited_tok_s":11.74,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":125.94,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":240.14,"bw_limited_tok_s":144.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.1,"score_norm":1.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":14.4,"score_norm":14.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6,"score_norm":6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":26.33,"score_norm":null,"config":"artificial analysis harness, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":30,"score_norm":12.5,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":55.1,"score_norm":null,"config":"instruct table, ministral 3 14b row (card reports 0.551)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1005,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1071,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1051,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1088,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-14b-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.17,"score_norm":45.31,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.63,"score_norm":3.87,"config":"artificial analysis harness, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":32.04,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":35.13,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":90.4,"score_norm":null,"config":"instruct table (card reports 0.904), maj@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":49.83,"score_norm":null,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":27.19,"score_norm":9.59,"config":"artificial analysis harness, tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.55,"score_norm":0,"config":"artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":9.74,"score_norm":5.58,"config":"artificial analysis harness, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ministral-3-14b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"wildbench","score_raw":68.5,"score_norm":null,"config":"instruct table, ministral 3 14b row","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512","publisher":"Hugging Face"},{"url":"https://mistral.ai/news/mistral-3/","publisher":"Mistral AI"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3-14b-2512-reasoning","display_name":"Ministral-3-14B-Reasoning-2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"14b","params_total_b":13.95,"params_active_b":13.95,"params_exact":13945031680,"params_source":"hf_safetensors","vram_est":{"bf16_gb":27.9,"int8_gb":14.8,"int4_gb":8.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Ministral-3-14B-Reasoning-2512","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-3-14b-reasoning-2512"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":27.9,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":30.637,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":27.89,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":30.627,"method":"vendor","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512/tree/main"},{"quant":"fp8","weights_gb":14.787,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.129,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.37,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":10.712,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":14.787,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.129,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":25.48,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":43.58,"bw_limited_tok_s":26.15,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":19.57,"bw_limited_tok_s":11.74,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":125.94,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":13.95,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":240.14,"bw_limited_tok_s":144.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":89.8,"score_norm":87.25,"config":"2024, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":85,"score_norm":81.25,"config":"aime25, reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.2,"score_norm":65.07,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":67.5,"score_norm":null,"config":"2025, ministral 3 paper table 5 (reasoning models), pass@16","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":64.6,"score_norm":null,"config":"reasoning mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"phybench","score_raw":26,"score_norm":null,"config":"ministral 3 paper table 5 (reasoning models), pass@16","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2601.08584v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"mistral-3-8b-2512-instruct","display_name":"Ministral-3-8B-Instruct-2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"8b","params_total_b":8.92,"params_active_b":8.92,"params_exact":8918026716,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.8,"int8_gb":9.5,"int4_gb":5.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.55,"expected_int8_gb":14.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Ministral-3-8B-Instruct-2512","openrouter_id":"mistralai/ministral-8b-2512","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-1-8b-2512","ministral-3-8b-2512","ministral-3-8b-2512-instruct","ministral-3-8b-instruct-2512","ministral-8b-2512","mistral-8-b-2512-8b"],"intelligence":{"index":20.76,"inputs":[{"benchmark":"aime","score_raw":31.7,"score_norm":14.63,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=ministral-3-8b","config":"aime 2025, artificial analysis harness"},{"benchmark":"gpqa_diamond","score_raw":47.1,"score_norm":31.13,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=ministral-3-8b","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=ministral-3-8b","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"mmlu_pro","score_raw":64.2,"score_norm":40.33,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=ministral-3-8b","config":"artificial analysis harness, mmlu-pro"},{"benchmark":"tau_bench","score_raw":26.6,"score_norm":8.8,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=ministral-3-8b","config":"artificial analysis harness, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":4.5,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=ministral-3-8b","config":"artificial analysis harness, terminal-bench hard"}],"coverage":0.5635,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.15,"cache_read_usd_per_m":0.015,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/ministral-8b-2512","as_of":"2026-10-06","blended_usd_per_m":0.15,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":17.84,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":19.981,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":9.455,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":11.596,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":10.421,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":12.562,"method":"vendor","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512/tree/main"},{"quant":"int4","weights_gb":5.352,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":7.493,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.455,"kv_cache_gb_per_8k":1.1409,"min_serve_gb":11.596,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.29,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":68.16,"bw_limited_tok_s":40.9,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.03,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":30.61,"bw_limited_tok_s":18.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":183.9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.92,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":375.56,"bw_limited_tok_s":225.34,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.6,"score_norm":0.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":9.7,"score_norm":9.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":5.5,"score_norm":5.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"agieval","score_raw":59.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":31.7,"score_norm":14.63,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":78.7,"score_norm":73.38,"config":"aime25, reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":50.9,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1047,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1059,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1003,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1072,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-8b-2512","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":47.1,"score_norm":31.13,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.8,"score_norm":58.87,"config":"reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":61.6,"score_norm":null,"config":"reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":87.6,"score_norm":null,"config":"instruct, maj@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":76.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":64.2,"score_norm":40.33,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":79.3,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multilingual_mmlu","score_raw":70.6,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":26.6,"score_norm":8.8,"config":"artificial analysis harness, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.5,"score_norm":0,"config":"artificial analysis harness, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=ministral-3-8b","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"triviaqa","score_raw":68.1,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildbench","score_raw":66.8,"score_norm":null,"config":"instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/mistral-3/","publisher":"Mistral AI"},{"url":"https://openrouter.ai/mistralai","publisher":"OpenRouter"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3-b-2512-3b","display_name":"Ministral 3 3B 2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"3b","params_total_b":3.85,"params_active_b":3.85,"params_exact":3849090048,"params_source":"hf_safetensors","vram_est":{"bf16_gb":7.7,"int8_gb":4.1,"int4_gb":2.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Ministral-3-3B-Instruct-2512","openrouter_id":"mistralai/ministral-3b-2512","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["ministral-1-3b-2512","ministral-3-3b-2512","ministral-3-3b-2512-instruct","ministral-3-3b-instruct-2512","ministral-3b-2512"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.1,"cache_read_usd_per_m":0.01,"context_length":131072,"source":"openrouter","source_model_id":"mistralai/ministral-3b-2512","as_of":"2026-10-06","blended_usd_per_m":0.1,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":7.7,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":9.572,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":4.081,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":5.953,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.31,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":4.182,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.081,"kv_cache_gb_per_8k":0.8724,"min_serve_gb":5.953,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":86.55,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":157.92,"bw_limited_tok_s":94.75,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":40.81,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":70.91,"bw_limited_tok_s":42.55,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":343,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.85,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":870.13,"bw_limited_tok_s":522.08,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.8,"score_norm":0.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":4.8,"score_norm":4.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.8,"score_norm":4.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":30.5,"score_norm":null,"config":"ministral 3 3b instruct, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2601.08584","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":978,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1016,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":961,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1034,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"math","score_raw":83,"score_norm":null,"config":"maj@1, ministral 3 3b instruct, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2601.08584","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mm_mtbench","score_raw":78.3,"score_norm":null,"config":"ministral 3 3b instruct, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2601.08584","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildbench","score_raw":56.8,"score_norm":null,"config":"ministral 3 3b instruct, table 4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2601.08584","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/ministral-3b-2512","publisher":"OpenRouter (listed 2025-12-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3-large-675b-2512-instruct","display_name":"Mistral-Large-3-675B-Instruct-2512","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"675b","params_total_b":675,"params_active_b":41,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":1350,"int8_gb":716,"int4_gb":405,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.16,"expected_int8_gb":112,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Large-3-675B-Instruct-2512","openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-large-3-675b-instruct-2512"],"intelligence":{"index":35.56,"inputs":[{"benchmark":"aime","score_raw":42.92,"score_norm":28.65,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"mistralai/mistral-large-2512, overall accuracy (95% ci 31.3-54.6), vals ai aime leaderboard"},{"benchmark":"arena_elo","score_raw":1415,"score_norm":60.34,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"arena (lmarena) text leaderboard, default view, ±3"},{"benchmark":"gpqa_diamond","score_raw":67.17,"score_norm":59.39,"trust":1,"source_url":"https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512","config":"official model card evaluation results (asterisked)"},{"benchmark":"hle","score_raw":4,"score_norm":2.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/comparisons/mistral-large-3-vs-llama-4-maverick","config":"artificial analysis independent eval, no tools, site displays rounded whole percent (4%)"},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"overall resolved rate (207/500), vals ai swe-bench verified leaderboard"},{"benchmark":"terminal_bench","score_raw":8.99,"score_norm":4.69,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2","config":"overall accuracy (8/89 tasks), terminus 2 harness, vals ai terminal-bench 2.0"}],"coverage":0.6349,"subset_bias":{"offset":-2.49,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":1350,"kv_cache_gb_per_8k":null,"min_serve_gb":1419.14,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":715.5,"kv_cache_gb_per_8k":null,"min_serve_gb":752.915,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":405,"kv_cache_gb_per_8k":null,"min_serve_gb":426.89,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":715.5,"kv_cache_gb_per_8k":null,"min_serve_gb":752.915,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":42.92,"score_norm":28.65,"config":"mistralai/mistral-large-2512, overall accuracy (95% ci 31.3-54.6), vals ai aime leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1415,"score_norm":60.34,"config":"arena (lmarena) text leaderboard, default view, ±3","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.17,"score_norm":59.39,"config":"official model card evaluation results (asterisked)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4,"score_norm":2.94,"config":"artificial analysis independent eval, no tools, site displays rounded whole percent (4%)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/comparisons/mistral-large-3-vs-llama-4-maverick","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"config":"overall resolved rate (207/500), vals ai swe-bench verified leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":8.99,"score_norm":4.69,"config":"overall accuracy (8/89 tasks), terminus 2 harness, vals ai terminal-bench 2.0","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/mistral-3/","publisher":"Mistral AI"},{"url":"https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"mistral-3-large-vision","display_name":"Mistral Large 3","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"frontier-large","params_total_b":675,"params_active_b":41,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":1350,"int8_gb":716,"int4_gb":405,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.04,"expected_int8_gb":27.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Large-3-675B-Instruct-2512","openrouter_id":"mistralai/mistral-large-2512","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262000,"release_date":"2025-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-3-large","mistral-3-large-675b","mistral-large-3","mistral-large-3-675b-base-2512"],"intelligence":{"index":38.2,"inputs":[{"benchmark":"aime","score_raw":38,"score_norm":22.5,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-3","config":"aime 2025, artificial analysis independent run, no tools, non-reasoning. raw field aime25=0.38. [raw: aime 2025]"},{"benchmark":"arena_elo","score_raw":1413.9,"score_norm":59.85,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":67.98,"score_norm":60.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-3","config":"artificial analysis independent run (intelligence index v4.1.1 component), no tools, non-reasoning model (0 reasoning tokens). raw value 0.67979797979798. [raw: gpqa diamond]"},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-3","config":"artificial analysis independent run, text-only / no tools (intelligence index v4.1.1 component). raw value 0.0417052826691381. [raw: humanity's last exam]"},{"benchmark":"mmlu_pro","score_raw":79.823,"score_norm":66.37,"trust":2,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"not a per-subject sub-score. [raw: mmlu pro], overall leaderboard accuracy (rank 89), vals.ai independent run"},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"trust":2,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"bash-only harness, single attempt (resolved rate). [raw: swe-bench verified], single tool, vals.ai independent run on the full 500-instance swe-bench verified split"},{"benchmark":"tau_bench","score_raw":24.56,"score_norm":6.08,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-3","config":"artificial analysis independent run, single aggregate number (raw field tau2=0.245614035087719). aa's separate τ3-banking domain score is 5.77. [raw: tau-2-bench], τ2-bench"},{"benchmark":"terminal_bench","score_raw":11.99,"score_norm":8.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-3","config":"artificial analysis independent run (intelligence index v4.1.1 component). raw value 0.119850187265918. aa's terminal-bench hard variant for the same model is 15.91. [raw: terminal-bench v2.1], terminal-bench v2.1"}],"coverage":0.8016,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":1350,"kv_cache_gb_per_8k":null,"min_serve_gb":1419.14,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":715.5,"kv_cache_gb_per_8k":null,"min_serve_gb":752.915,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":405,"kv_cache_gb_per_8k":null,"min_serve_gb":426.89,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":715.5,"kv_cache_gb_per_8k":null,"min_serve_gb":752.915,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":38,"score_norm":22.5,"config":"aime 2025, artificial analysis independent run, no tools, non-reasoning. raw field aime25=0.38. [raw: aime 2025]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1413.9,"score_norm":59.85,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1362.2,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1200.6,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1229.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.98,"score_norm":60.54,"config":"artificial analysis independent run (intelligence index v4.1.1 component), no tools, non-reasoning model (0 reasoning tokens). raw value 0.67979797979798. [raw: gpqa diamond]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"config":"artificial analysis independent run, text-only / no tools (intelligence index v4.1.1 component). raw value 0.0417052826691381. [raw: humanity's last exam]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.823,"score_norm":66.37,"config":"not a per-subject sub-score. [raw: mmlu pro], overall leaderboard accuracy (rank 89), vals.ai independent run","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":41.4,"score_norm":34.89,"config":"bash-only harness, single attempt (resolved rate). [raw: swe-bench verified], single tool, vals.ai independent run on the full 500-instance swe-bench verified split","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":24.56,"score_norm":6.08,"config":"artificial analysis independent run, single aggregate number (raw field tau2=0.245614035087719). aa's separate τ3-banking domain score is 5.77. [raw: tau-2-bench], τ2-bench","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":11.99,"score_norm":8.22,"config":"artificial analysis independent run (intelligence index v4.1.1 component). raw value 0.119850187265918. aa's terminal-bench hard variant for the same model is 15.91. [raw: terminal-bench v2.1], terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-3","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.mistral.ai/models","publisher":"Mistral AI"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"mistral-3.1-medium","display_name":"Mistral Medium 3.1","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3.1","size_class":"frontier-medium","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-medium-3.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2025-08-13","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":["mistral-3-medium"],"superseded_by":null,"aliases":["mistral-medium-3.1"],"intelligence":{"index":38.26,"inputs":[{"benchmark":"arena_elo","score_raw":1409,"score_norm":57.66,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"92749 votes, arena entry mistral-medium-2508, no style control, rank 137, text/overall"},{"benchmark":"hle","score_raw":4.7,"score_norm":3.97,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","config":"artificial analysis independent eval"},{"benchmark":"mmlu_pro","score_raw":74.4,"score_norm":57.33,"trust":1,"source_url":"https://www.noota.io/en/mistral-review-guide","config":"mistral-reported figure as quoted by the review"}],"coverage":0.3094,"subset_bias":{"offset":-7.08,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":2,"cache_read_usd_per_m":0.04,"context_length":131072,"source":"openrouter","source_model_id":"mistralai/mistral-medium-3.1","as_of":"2026-10-06","blended_usd_per_m":0.8,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.9,"score_norm":1.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":20.5,"score_norm":20.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.2,"score_norm":9.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1409,"score_norm":57.66,"config":"92749 votes, arena entry mistral-medium-2508, no style control, rank 137, text/overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1096,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1008,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1126,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1153,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1087,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":992,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1106,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":4.7,"score_norm":3.97,"config":"artificial analysis independent eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":74.4,"score_norm":57.33,"config":"mistral-reported figure as quoted by the review","trust":1,"source_type":"vendor_claim","source_url":"https://www.noota.io/en/mistral-review-guide","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3.1","publisher":"OpenRouter (listed 2025-08-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3.2-small-24b-2506-instruct","display_name":"Mistral Small 3.2 24B","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3.2","size_class":"24b","params_total_b":24.01,"params_active_b":24.01,"params_exact":24011361280,"params_source":"hf_safetensors","vram_est":{"bf16_gb":48,"int8_gb":25.5,"int4_gb":14.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.04,"expected_int8_gb":26.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Small-3.2-24B-Instruct-2506","openrouter_id":"mistralai/mistral-small-3.2-24b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["image","text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2025-06-20","release_status":"released","knowledge_cutoff":"2023-10-31","status":"current","supersedes":["mistral-3.1-small-24b-2503-instruct"],"superseded_by":null,"aliases":["mistral-small-3.2-24b-instruct","mistral-small-3.2-24b-instruct-2506"],"intelligence":{"index":37.03,"inputs":[{"benchmark":"arena_elo","score_raw":1357,"score_norm":34.48,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"17399 votes, arena entry mistral-small-2506, no style control, rank 194, text/overall"},{"benchmark":"gpqa_diamond","score_raw":46.13,"score_norm":29.76,"trust":1,"source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","config":"5-shot cot"},{"benchmark":"mmlu_pro","score_raw":69.06,"score_norm":48.43,"trust":1,"source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","config":"5-shot cot"}],"coverage":0.3572,"subset_bias":{"offset":13.06,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.09375,"completion_usd_per_m":0.25,"cache_read_usd_per_m":null,"context_length":256000,"source":"openrouter","source_model_id":"mistralai/mistral-small-3.2-24b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.1328125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":48.02,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":51.763,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":48.023,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":51.766,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506/tree/main"},{"quant":"fp8","weights_gb":25.451,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":28.066,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":14.406,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":16.748,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":25.451,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":28.066,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":14.97,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":24.01,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":25.32,"bw_limited_tok_s":15.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":6.78,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":24.01,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":11.37,"bw_limited_tok_s":6.82,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":77.25,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":24.01,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":139.53,"bw_limited_tok_s":83.72,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"ai2d","score_raw":92.91,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1357,"score_norm":34.48,"config":"17399 votes, arena entry mistral-small-2506, no style control, rank 194, text/overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":43.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":87.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":913,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":936,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":901,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":912,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":902,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":94.86,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.13,"score_norm":29.76,"config":"5-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_main","score_raw":44.22,"score_norm":null,"config":"5-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval_plus","score_raw":92.9,"score_norm":null,"config":"pass@5","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":69.42,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mathvista","score_raw":67.09,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp_plus","score_raw":78.33,"score_norm":null,"config":"pass@5","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":80.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.06,"score_norm":48.43,"config":"5-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmmu","score_raw":62.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":12.1,"score_norm":null,"config":"totalacc","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildbench_v2","score_raw":65.33,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.2-24b-instruct","publisher":"OpenRouter (listed 2025-06-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3.5-medium-128b","display_name":"Mistral-Medium-3.5-128B","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3.5","size_class":"128b","params_total_b":127.7,"params_active_b":127.7,"params_exact":127704210176,"params_source":"hf_safetensors","vram_est":{"bf16_gb":255,"int8_gb":135,"int4_gb":76.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.37,"expected_int8_gb":49.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mistral-Medium-3.5-128B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-medium-3.5-128b"],"intelligence":{"index":57.84,"inputs":[{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa intelligence index eval"},{"benchmark":"hle","score_raw":13.76,"score_norm":17.29,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa intelligence index eval, no tools"},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"trust":1,"source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","config":null},{"benchmark":"terminal_bench","score_raw":50.56,"score_norm":53.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa eval, terminal-bench 2.1"}],"coverage":0.4762,"subset_bias":{"offset":-2.3,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":255.4,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":271.123,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":135.362,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":145.083,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":133.606,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":143.239,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/tree/main"},{"quant":"int4","weights_gb":76.62,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":83.404,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":135.362,"kv_cache_gb_per_8k":2.9528,"min_serve_gb":145.083,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.56,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":127.7,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.28,"bw_limited_tok_s":2.57,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_lcr","score_raw":69.33,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":878.9,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"config":"aa intelligence index eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":13.76,"score_norm":17.29,"config":"aa intelligence index eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":68.78,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":64.86,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.16,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"config":"with tools (agentic)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench_telecom","score_raw":91.4,"score_norm":null,"config":"tau-cubed telecom","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.56,"score_norm":53.6,"config":"aa eval, terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"mistral-3.5-medium-vision","display_name":"Mistral Medium 3.5","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3.5","size_class":"frontier-medium","params_total_b":128,"params_active_b":128,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":256,"int8_gb":136,"int4_gb":76.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.35,"expected_int8_gb":47.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":null,"openrouter_id":"mistralai/mistral-medium-3-5","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_large","license":"Modified MIT","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-3.5-medium","mistral-3.5-medium-128b","mistral-medium-3-5","mistral-medium-3.5"],"intelligence":{"index":56.46,"inputs":[{"benchmark":"arena_elo","score_raw":1426.9,"score_norm":65.64,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.748484848484849 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":13.76,"score_norm":17.29,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.137627432808156 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":64.7,"score_norm":60.78,"trust":3,"source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","config":"vals.ai independently-run subset"},{"benchmark":"tau_bench","score_raw":94.15,"score_norm":98.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.941520467836257 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":30.3,"score_norm":29.76,"trust":3,"source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","config":"vals.ai independent run"}],"coverage":0.6032,"subset_bias":{"offset":-1.29,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.5,"completion_usd_per_m":7.5,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"mistralai/mistral-medium-3-5","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":256,"kv_cache_gb_per_8k":null,"min_serve_gb":272.8,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":135.68,"kv_cache_gb_per_8k":null,"min_serve_gb":146.464,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":76.8,"kv_cache_gb_per_8k":null,"min_serve_gb":84.64,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":135.68,"kv_cache_gb_per_8k":null,"min_serve_gb":146.464,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.55,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":128,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":4.27,"bw_limited_tok_s":2.56,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":7,"score_norm":7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":19.21,"score_norm":19.21,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":46.9,"score_norm":46.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.2,"score_norm":14.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":30.39,"score_norm":30.39,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, source value 0.653333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":12.5,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1426.9,"score_norm":65.64,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1199.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1262.9,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":13.66,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.13655123642959935 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":33.72,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.3372127723067741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"finance_agent","score_raw":46.1,"score_norm":null,"config":"vals.ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","measured_date":"2026-05-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":21.68,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, source value 0.21675 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.748484848484849 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":69.1,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.690982776089159 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":13.76,"score_norm":17.29,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.137627432808156 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":68.78,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, source value 0.687755102040816 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":1.67,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0166666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":64.86,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, source value 0.648554913294798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"sage","score_raw":37.6,"score_norm":null,"config":"vals.ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","measured_date":"2026-05-01","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.58,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, scicode, source value 0.395833333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":64.7,"score_norm":60.78,"config":"vals.ai independently-run subset","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","measured_date":"2026-05-01","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.6,"score_norm":75.11,"config":"% resolved, as published on model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","measured_date":"2026-04-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau3_telecom","score_raw":91.4,"score_norm":null,"config":"agentic tool-use, as published","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Medium-3.5-128B","measured_date":"2026-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":94.15,"score_norm":98.87,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.941520467836257 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":15.05,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.150515463917526 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":30.3,"score_norm":29.76,"config":"vals.ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","measured_date":"2026-05-01","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.56,"score_norm":53.6,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.50561797752809 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"vals_index","score_raw":52.77,"score_norm":null,"config":"vals.ai independent run, rank 32/46","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/mistralai_mistral-medium-3.5","measured_date":"2026-05-01","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.mistral.ai/models","publisher":"Mistral AI"},{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3-5","publisher":"OpenRouter (listed 2026-04-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-4-small-instruct","display_name":"Mistral Small 4","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"4","size_class":"frontier-small","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2026-03","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["mistral-4-small","mistral-small-4"],"intelligence":{"index":51.49,"inputs":[{"benchmark":"aime","score_raw":84,"score_norm":80,"trust":2,"source_url":"https://docsbot.ai/models/mistral-small-4","config":"aime 2025, cites mistral.ai/news/mistral-small-4, mistral-reported figure transcribed by aggregator, no tools, non-reasoning (instruct) variant listed as 36 [raw: aime 2025], reasoning_effort=high"},{"benchmark":"gpqa_diamond","score_raw":76.87,"score_norm":73.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.768686868686869 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":9.87,"score_norm":11.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0987025023169602 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":78,"score_norm":63.33,"trust":2,"source_url":"https://docsbot.ai/models/mistral-small-4","config":"cites mistral.ai/news/mistral-small-4, mistral-reported figure transcribed by aggregator, non-reasoning (instruct) variant listed as 73.5 [raw: mmlu pro], reasoning mode (high)"},{"benchmark":"tau_bench","score_raw":41.23,"score_norm":28.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.412280701754386 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":20.97,"score_norm":18.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-4","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.209737827715356 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5635,"subset_bias":{"offset":1.17,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":4.63,"score_norm":4.63,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.71,"score_norm":19.71,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20,"score_norm":20,"config":"mistral small 4 (reasoning)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":47.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.473333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":84,"score_norm":80,"config":"aime 2025, cites mistral.ai/news/mistral-small-4, mistral-reported figure transcribed by aggregator, no tools, non-reasoning (instruct) variant listed as 36 [raw: aime 2025], reasoning_effort=high","trust":2,"source_type":"aggregator","source_url":"https://docsbot.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":1.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":4.49,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.044920000000000015 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.87,"score_norm":73.06,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.768686868686869 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.2,"score_norm":65.07,"config":"reasoning enabled; vendor model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/mistralai/Mistral-Small-4-119B-2603","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":9.87,"score_norm":11.57,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0987025023169602 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":48.16,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.481632653061224 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":78,"score_norm":63.33,"config":"cites mistral.ai/news/mistral-small-4, mistral-reported figure transcribed by aggregator, non-reasoning (instruct) variant listed as 73.5 [raw: mmlu pro], reasoning mode (high)","trust":2,"source_type":"aggregator","source_url":"https://docsbot.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":56.82,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.568208092485549 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":37.96,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.37962962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":41.23,"score_norm":28.31,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.412280701754386 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":36.7,"score_norm":22.27,"config":"openrouter \"exacto benchmarks\" section on the mistralai/mistral-small-2603 model page, value for the mistral first-party endpoint (auto-routing row shows 41.3%). domain/version not stated by openroute [raw: tau-bench]","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/mistralai/mistral-small-2603","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":4.95,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0494845360824742 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":17.42,"score_norm":14.61,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.174242424242424 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":20.97,"score_norm":18.79,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.209737827715356 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.mistral.ai/models","publisher":"Mistral AI"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"mixtral-8-x22b-instruct","display_name":"Mixtral 8x22B Instruct","family":{"id":"mixtral","vendor":"Mistral AI"},"generation":"8","size_class":"frontier","params_total_b":140.63,"params_active_b":39,"params_exact":140630071296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":281,"int8_gb":149,"int4_gb":84.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.16,"expected_int8_gb":23.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"mistralai/Mixtral-8x22B-Instruct-v0.1","openrouter_id":"mistralai/mixtral-8x22b-instruct","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text","file"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2024-04","release_status":"released","knowledge_cutoff":"2024-01-31","status":"current","supersedes":["mixtral-8-x22b-v0.1-8x22b-instruct"],"superseded_by":null,"aliases":["mixtral-8-x22b","mixtral-8-x22b-8x22b","mixtral-8-x22b-v0.1-8x22b-instruct","mixtral-8x22b","mixtral-8x22b-instruct","mixtral-8x22b-instruct-v0.1","mixtral-8x22b-v0.1"],"intelligence":{"index":6.21,"inputs":[{"benchmark":"arena_elo","score_raw":1229.5,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":33.23,"score_norm":11.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.332323232323232 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.01,"score_norm":2.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0401 is a 0-1 fraction, x100 at harvest"}],"coverage":0.3492,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.2,"context_length":65536,"source":"openrouter","source_model_id":"mistralai/mixtral-8x22b-instruct","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":281.26,"kv_cache_gb_per_8k":1.879,"min_serve_gb":297.202,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":281.26,"kv_cache_gb_per_8k":1.879,"min_serve_gb":297.202,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1/tree/main"},{"quant":"fp8","weights_gb":149.068,"kv_cache_gb_per_8k":1.879,"min_serve_gb":158.4,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":84.378,"kv_cache_gb_per_8k":1.879,"min_serve_gb":90.476,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":149.068,"kv_cache_gb_per_8k":1.879,"min_serve_gb":158.4,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":7.86,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":39,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":14,"bw_limited_tok_s":7.92,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.7,"score_norm":5.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mixtral-8x22b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.01,"score_norm":4.01,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1229.5,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1229.2,"score_norm":0,"config":"51416 votes, arena entry mixtral-8x22b-instruct-v0.1, leaderboard text-overall-style_control, rank 310, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":33.23,"score_norm":11.59,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.332323232323232 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.01,"score_norm":2.96,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0401 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":14.81,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.148148148148148 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":18.75,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.1875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-8x22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mixtral-8x22b-instruct","publisher":"OpenRouter (listed 2024-04-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"morph-3-fast","display_name":"Morph V3 Fast","family":{"id":"morph","vendor":"morph"},"generation":"3","size_class":"frontier-fast","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"morph/morph-v3-fast","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":81920,"release_date":"2025-07-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["morph-v3-fast"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.8,"completion_usd_per_m":1.2,"cache_read_usd_per_m":null,"context_length":81920,"source":"openrouter","source_model_id":"morph/morph-v3-fast","as_of":"2026-10-06","blended_usd_per_m":0.9000000000000001,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#morph/morph-v3-fast","publisher":"OpenRouter (listed 2025-07-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"morph-3-large","display_name":"Morph V3 Large","family":{"id":"morph","vendor":"morph"},"generation":"3","size_class":"frontier-large","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"morph/morph-v3-large","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2025-07-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["morph-v3-large"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.9,"completion_usd_per_m":1.9,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"morph/morph-v3-large","as_of":"2026-10-06","blended_usd_per_m":1.15,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#morph/morph-v3-large","publisher":"OpenRouter (listed 2025-07-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-glimmer-1-30b","display_name":"Muse Glimmer 30B","family":{"id":"muse","vendor":"Meta"},"generation":"1","size_class":"30b","params_total_b":29.78,"params_active_b":29.78,"params_exact":29776626688,"params_source":"hf_safetensors","vram_est":{"bf16_gb":59.6,"int8_gb":31.6,"int4_gb":17.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.2,"expected_int8_gb":69.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"meta-models/Muse-Glimmer-30B","openrouter_id":"meta/muse-glimmer-30b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2026-08-09","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["muse-glimmer-30b"],"intelligence":{"index":68.43,"inputs":[{"benchmark":"aime","score_raw":94.7,"score_norm":93.38,"trust":1,"source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","config":"aime 2026, high reasoning"},{"benchmark":"gpqa_diamond","score_raw":83.5,"score_norm":82.39,"trust":1,"source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","config":"artificial analysis harness, high reasoning"},{"benchmark":"hle","score_raw":22,"score_norm":29.41,"trust":1,"source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","config":"artificial analysis harness, text only"},{"benchmark":"swe_bench_verified","score_raw":76,"score_norm":73.33,"trust":1,"source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","config":null},{"benchmark":"terminal_bench","score_raw":51.7,"score_norm":54.94,"trust":1,"source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","config":"terminalbench 2.1, terminus2 harness"}],"coverage":0.5555,"subset_bias":{"offset":-0.43,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.35,"completion_usd_per_m":1.5,"cache_read_usd_per_m":0.04,"context_length":131072,"source":"openrouter","source_model_id":"meta/muse-glimmer-30b","as_of":"2026-10-06","blended_usd_per_m":0.6375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":59.56,"kv_cache_gb_per_8k":0.1908,"min_serve_gb":62.729,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":59.553,"kv_cache_gb_per_8k":0.1908,"min_serve_gb":62.721,"method":"vendor","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B/tree/main"},{"quant":"fp8","weights_gb":31.567,"kv_cache_gb_per_8k":0.1908,"min_serve_gb":33.336,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":17.868,"kv_cache_gb_per_8k":0.1908,"min_serve_gb":19.059,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":31.567,"kv_cache_gb_per_8k":0.1908,"min_serve_gb":33.336,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":23.91,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":29.78,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":40.83,"bw_limited_tok_s":24.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":29.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9.17,"bw_limited_tok_s":5.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":63.23,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":29.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":112.49,"bw_limited_tok_s":67.49,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":8.5,"score_norm":8.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-glimmer-30b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":49,"score_norm":49,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-glimmer-30b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":17.5,"score_norm":17.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-glimmer-30b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":80,"score_norm":null,"config":"artificial analysis harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":94.7,"score_norm":93.38,"config":"aime 2026, high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"charxiv_reasoning","score_raw":78.8,"score_norm":null,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepsearch_qa","score_raw":74.6,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gaia2","score_raw":43.3,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.5,"score_norm":82.39,"config":"artificial analysis harness, high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22,"score_norm":29.41,"config":"artificial analysis harness, text only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77,"score_norm":null,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":75.5,"score_norm":null,"config":"public split, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74,"score_norm":null,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"omnidocbench","score_raw":75.8,"score_norm":null,"config":"v1.5","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":65.9,"score_norm":null,"config":"verified, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.6,"score_norm":null,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"screenspot_pro","score_raw":75.4,"score_norm":null,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":51.2,"score_norm":77,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":51.2,"score_norm":77,"config":"high reasoning, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76,"score_norm":73.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76,"score_norm":73.33,"config":"high reasoning, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":23.5,"score_norm":null,"config":"tau3/banking, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":51.7,"score_norm":54.94,"config":"terminalbench 2.1, terminus2 harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"wildclawbench","score_raw":47.6,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/meta-models/Muse-Glimmer-30B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta/muse-glimmer-30b","publisher":"OpenRouter (listed 2026-08-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-spark-1.3","display_name":"Muse Spark 1.3","family":{"id":"muse","vendor":"Meta"},"generation":"1.3","size_class":"frontier-spark","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"meta/muse-spark-1.3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":"Proprietary","is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["muse-spark-1.1","muse-spark-1.2","muse-spark-1.2-contributor"],"superseded_by":null,"aliases":[],"intelligence":{"index":86.96,"inputs":[{"benchmark":"gpqa_diamond","score_raw":93.84,"score_norm":96.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","config":"artificial analysis harness, max reasoning effort"},{"benchmark":"hle","score_raw":49.07,"score_norm":69.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","config":"artificial analysis harness, max reasoning effort, no tools"},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","config":"artificial analysis harness, max reasoning effort, v2.1"}],"coverage":0.3174,"subset_bias":{"offset":-0.67,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":4.25,"cache_read_usd_per_m":0.15,"context_length":1048576,"source":"openrouter","source_model_id":"meta/muse-spark-1.3","as_of":"2026-10-06","blended_usd_per_m":2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":55.5,"score_norm":55.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":75.8,"score_norm":75.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":48.1,"score_norm":48.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":84.33,"score_norm":null,"config":"artificial analysis harness, max reasoning effort, v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1624.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":24.86,"score_norm":null,"config":"artificial analysis harness, max reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1413,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1261,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1360,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1350,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1311,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1357,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1241,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1219,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1244,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1321,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1359,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1289,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1364,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.84,"score_norm":96.96,"config":"artificial analysis harness, max reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":49.07,"score_norm":69.22,"config":"artificial analysis harness, max reasoning effort, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":58.33,"score_norm":null,"config":"artificial analysis harness, max reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":52.37,"score_norm":null,"config":"artificial analysis harness, max reasoning effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"config":"artificial analysis harness, max reasoning effort, v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://research.meta.ai/blog/introducing-muse-spark-1-3","publisher":"Meta"},{"url":"https://developer.meta.com/ai/models/muse-spark/","publisher":"Meta"},{"url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3","publisher":"OpenRouter (listed 2026-09-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-spark-1.3-contributor","display_name":"Muse Spark 1.3 Contributor","family":{"id":"muse","vendor":"Meta"},"generation":"1.3","size_class":"frontier-spark","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"meta/muse-spark-1.3-contributor","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file","audio"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":87.08,"inputs":[{"benchmark":"gpqa_diamond","score_raw":94,"score_norm":97.18,"trust":2,"source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","config":"artificial analysis intelligence index component, xhigh reasoning"},{"benchmark":"hle","score_raw":49,"score_norm":69.12,"trust":2,"source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","config":"+2 vs xhigh, max"},{"benchmark":"terminal_bench","score_raw":86,"score_norm":95.29,"trust":2,"source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","config":"+6 vs 1.2 80%, 2.1, max"}],"coverage":0.3174,"subset_bias":{"offset":-0.67,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.2,"cache_read_usd_per_m":0.002,"context_length":1048576,"source":"openrouter","source_model_id":"meta/muse-spark-1.3-contributor","as_of":"2026-10-06","blended_usd_per_m":0.125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":62,"score_norm":62,"config":"limited partner preview, max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":61,"score_norm":61,"config":"xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79,"score_norm":null,"config":"vs 1.2 83%, xhigh and max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"agentic_if_index","score_raw":57.8,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":49.4,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":26,"score_norm":null,"config":"vs 1.2 18%, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"deepsearchqa","score_raw":89.4,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":75.4,"score_norm":null,"config":"1.1, max, vendor comparison table","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/muse-spark-1-3","measured_date":"2026-09-03","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":75.4,"score_norm":null,"config":"1.1, max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1754,"score_norm":null,"config":"62% more reasoning tokens vs xhigh, elo, max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1709,"score_norm":null,"config":"elo, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1754,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94,"score_norm":97.18,"config":"artificial analysis intelligence index component, xhigh reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94,"score_norm":97.18,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/meta/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":49,"score_norm":69.12,"config":"+2 vs xhigh, max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":47,"score_norm":66.18,"config":"vs muse spark 1.2 45%, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":49,"score_norm":69.12,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/meta/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"jobbench","score_raw":64.9,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mrcr_256k_512k","score_raw":98.5,"score_norm":null,"config":"max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":66.9,"score_norm":null,"config":"2.0, max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":59,"score_norm":null,"config":"vs 1.2 56%, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":59,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://opper.ai/meta/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"swe_atlas","score_raw":59.4,"score_norm":null,"config":"codebase qna, max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":52,"score_norm":null,"config":"max, more turns and 28% more reasoning tokens vs xhigh, tools on","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":47,"score_norm":null,"config":"agentic banking domain, tools on, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":86,"score_norm":95.29,"config":"+6 vs 1.2 80%, 2.1, max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85,"score_norm":94.12,"config":"2.1, vs 1.2 80%, xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/muse-spark-1-3","measured_date":"2026-09-02","observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.8,"score_norm":98.59,"config":"2.1, max, meta published comparison table","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/muse-spark-1-3","measured_date":"2026-09-03","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.8,"score_norm":98.59,"config":"2.1, max reasoning tier; reclassified 2026-09-05: third-party blog restating base-model (max tier) figures","trust":0,"source_type":"community","source_url":"https://www.explainx.ai/blog/meta-muse-spark-1-3-launch-benchmarks-pricing-september-2026","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.3-contributor","publisher":"OpenRouter (listed 2026-09-02)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-09-29T06:07:54.087Z"},{"id":"muse-spark-1.3-max","display_name":"Muse Spark 1.3 Max","family":{"id":"muse","vendor":"Meta"},"generation":"1.3","size_class":"frontier-spark-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-09-05","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_elo","score_raw":1494.3,"score_norm":95.68,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1470.7,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-28","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1645.5,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-28","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1290.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1656.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://vals.ai/models/meta_muse_spark_1_3_max","publisher":"Vals AI"},{"url":"https://artificialanalysis.ai/articles/muse-spark-1-3","publisher":"Artificial Analysis"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"namazu-1","display_name":"Sakana Namazu","family":{"id":"namazu","vendor":"sakana"},"generation":"1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"sakana/sakana-namazu","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-08-11","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["namazu","sakana-namazu"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.95,"completion_usd_per_m":4,"cache_read_usd_per_m":0.15,"context_length":262144,"source":"openrouter","source_model_id":"sakana/sakana-namazu","as_of":"2026-10-06","blended_usd_per_m":1.7125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"fairpoliticsqa","score_raw":56.3,"score_norm":null,"config":"sakana internal benchmark","trust":1,"source_type":"vendor_claim","source_url":"https://sakana.ai/namazu-api/","measured_date":"2026-08-03","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#sakana/sakana-namazu","publisher":"OpenRouter (listed 2026-08-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-2-nano-12b-vision","display_name":"nemotron-nano-12b-v2-vl","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"2","size_class":"12b","params_total_b":13.18,"params_active_b":12,"params_exact":13181860358,"params_source":"hf_safetensors","vram_est":{"bf16_gb":26.4,"int8_gb":14,"int4_gb":7.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","openrouter_id":"nvidia/nemotron-nano-12b-v2-vl:free","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["image","text","video"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-10-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nemotron-nano-12b-v2-vl","nvidia-nemotron-nano-12b-v2-vl-bf16"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":26.36,"kv_cache_gb_per_8k":null,"min_serve_gb":28.158,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":26.364,"kv_cache_gb_per_8k":null,"min_serve_gb":28.162,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/tree/main"},{"quant":"fp8","weights_gb":13.971,"kv_cache_gb_per_8k":null,"min_serve_gb":15.451,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.908,"kv_cache_gb_per_8k":null,"min_serve_gb":9.388,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":13.971,"kv_cache_gb_per_8k":null,"min_serve_gb":15.451,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":29.5,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":50.67,"bw_limited_tok_s":30.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.75,"bw_limited_tok_s":13.65,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":143.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":279.17,"bw_limited_tok_s":167.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"ai2d","score_raw":87.11,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":89.72,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":94.39,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mathvista","score_raw":76.9,"score_norm":null,"config":"chatgpt as judge","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu","score_raw":68,"score_norm":null,"config":"chatgpt as judge","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ocr_reasoning","score_raw":36.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ocrbench_v2","score_raw":62,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"video_mme","score_raw":65.9,"score_norm":null,"config":"without subtitles","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16","measured_date":"2025-10-28","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-nano-12b-v2-vl:free","publisher":"OpenRouter (listed 2025-10-28)"}],"last_verified":"2026-08-24","last_enriched_at":null},{"id":"nemotron-2-nano-9b","display_name":"nemotron-nano-9b-v2","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"2","size_class":"9b","params_total_b":8.89,"params_active_b":8.89,"params_exact":8888227328,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.8,"int8_gb":9.4,"int4_gb":5.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.13,"expected_int8_gb":20.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/NVIDIA-Nemotron-Nano-9B-v2","openrouter_id":"nvidia/nemotron-nano-9b-v2:free","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-09-05","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nemotron-nano-9b-v2","nvidia-nemotron-nano-9b-v2"],"intelligence":{"index":28.71,"inputs":[{"benchmark":"aime","score_raw":69.67,"score_norm":62.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.696666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":56.97,"score_norm":45.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.56969696969697 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.87,"score_norm":4.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0486561631139944 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":23.39,"score_norm":4.52,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.233918128654971 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":17.78,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":18.914,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.776,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":18.91,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2/tree/main"},{"quant":"fp8","weights_gb":9.423,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":10.557,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.334,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":6.468,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.423,"kv_cache_gb_per_8k":0.1342,"min_serve_gb":10.557,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.42,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.89,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":68.39,"bw_limited_tok_s":41.03,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.09,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.89,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":30.71,"bw_limited_tok_s":18.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":184.4,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.89,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":376.83,"bw_limited_tok_s":226.1,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":8.68,"score_norm":8.68,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":26,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0.26 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":69.67,"score_norm":62.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.696666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":56.97,"score_norm":45.03,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.56969696969697 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.87,"score_norm":4.22,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0486561631139944 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":27.62,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.276190476190476 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":72.38,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.723809523809524 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":21.99,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.219907407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":23.39,"score_norm":4.52,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.233918128654971 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-nano-9b-v2-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-nano-9b-v2:free","publisher":"OpenRouter (listed 2025-09-05)"}],"last_verified":"2026-08-24","last_enriched_at":null},{"id":"nemotron-3-nano-30b-a3b","display_name":"Nemotron 3 Nano 30B A3B","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3","size_class":"30b-a3b","params_total_b":31.58,"params_active_b":3,"params_exact":31577937344,"params_source":"hf_safetensors","vram_est":{"bf16_gb":63.2,"int8_gb":33.5,"int4_gb":18.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":4.32,"expected_int8_gb":145,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16","openrouter_id":"nvidia/nemotron-3-nano-30b-a3b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-12-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nvidia-nemotron-3-nano-30b-a3b-bf16"],"intelligence":{"index":43.75,"inputs":[{"benchmark":"aime","score_raw":91,"score_norm":88.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.91 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1313.5,"score_norm":15.1,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":75.66,"score_norm":71.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.756565656565657 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":11.4,"score_norm":13.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.113994439295644 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":40.94,"score_norm":27.92,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.409356725146199 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-2.28,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.2,"cache_read_usd_per_m":0.03,"context_length":262144,"source":"openrouter","source_model_id":"nvidia/nemotron-3-nano-30b-a3b","as_of":"2026-10-06","blended_usd_per_m":0.08750000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":63.16,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":66.368,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":63.157,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":66.365,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/tree/main"},{"quant":"fp8","weights_gb":33.475,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":35.199,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.948,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":19.998,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":33.475,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":35.199,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-30b-a3b","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.99,"score_norm":1.99,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":14.4,"score_norm":14.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-30b-a3b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.9,"score_norm":8.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-30b-a3b","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.54,"score_norm":14.54,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":37.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.373333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91,"score_norm":88.75,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.91 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1313.5,"score_norm":15.1,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1314.9,"score_norm":15.72,"config":"15581 votes, arena entry nvidia-nemotron-3-nano-30b-a3b-bf16, leaderboard text-overall-style_control, rank 248, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.66,"score_norm":71.35,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.756565656565657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.4,"score_norm":13.82,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.113994439295644 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.09,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.710884353741497 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":74.07,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.740740740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":29.63,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.296296296296296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":40.94,"score_norm":27.92,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.409356725146199 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.98,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0597938144329897 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.74,"score_norm":2.05,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0674157303370786 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-30b-a3b","publisher":"OpenRouter (listed 2025-12-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3-nano-omni-30b-a3b-reasoning","display_name":"Nemotron 3 Nano Omni","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3","size_class":"30b-a3b","params_total_b":33.02,"params_active_b":3,"params_exact":33015632214,"params_source":"hf_safetensors","vram_est":{"bf16_gb":66,"int8_gb":35,"int4_gb":19.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.76,"expected_int8_gb":61.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16","openrouter_id":"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text","audio","image","video"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":256000,"release_date":"2026-04-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nemotron-3-nano-omni-30b-a3b-reasoning-bf16"],"intelligence":{"index":19.58,"inputs":[{"benchmark":"gpqa_diamond","score_raw":46.87,"score_norm":30.8,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.468686868686869 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.82,"score_norm":4.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0481927710843374 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":45.32,"score_norm":33.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.453216374269006 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-6.36,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0,"completion_usd_per_m":0,"cache_read_usd_per_m":null,"context_length":256000,"source":"openrouter","source_model_id":"nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free","as_of":"2026-10-06","blended_usd_per_m":0,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":66.04,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":69.392,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":66.032,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":69.384,"method":"vendor","source_url":"https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16/tree/main"},{"quant":"fp8","weights_gb":35.001,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":36.801,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.812,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":20.862,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":35.001,"kv_cache_gb_per_8k":0.0503,"min_serve_gb":36.801,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":13.8,"score_norm":13.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.3,"score_norm":10.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.01,"score_norm":15.01,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":40.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.406666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.87,"score_norm":30.8,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.468686868686869 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.82,"score_norm":4.15,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0481927710843374 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":63.2,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.631972789115646 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":53.18,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.531791907514451 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":27.78,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":45.32,"score_norm":33.76,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.453216374269006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.74,"score_norm":2.05,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0674157303370786 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-nano-omni-30b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free","publisher":"OpenRouter (listed 2026-04-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3-super-120b-a12b","display_name":"Nemotron 3 Super","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3","size_class":"120b-a12b","params_total_b":123.61,"params_active_b":12,"params_exact":123611012096,"params_source":"hf_safetensors","vram_est":{"bf16_gb":247,"int8_gb":131,"int4_gb":74.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.41,"expected_int8_gb":185,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8","openrouter_id":"nvidia/nemotron-3-super-120b-a12b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-03-11","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nvidia-nemotron-3-super-120b-a12b-fp8"],"intelligence":{"index":52.06,"inputs":[{"benchmark":"arena_elo","score_raw":1360.3,"score_norm":35.96,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"7535 votes, arena entry march26-chatbot1, leaderboard text-overall-style_control, rank 187, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":80,"score_norm":77.46,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.8 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":20.76,"score_norm":27.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.207599629286376 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":67.84,"score_norm":63.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.678362573099415 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":38.58,"score_norm":39.51,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.385767790262172 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-5.14,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.08,"completion_usd_per_m":0.45,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"nvidia/nemotron-3-super-120b-a12b","as_of":"2026-10-06","blended_usd_per_m":0.1725,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":247.22,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":259.648,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":131.027,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":137.645,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":128.35,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":134.835,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8/tree/main"},{"quant":"int4","weights_gb":74.166,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":77.941,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":131.027,"kv_cache_gb_per_8k":0.0671,"min_serve_gb":137.645,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":25.1,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":12,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":45.5,"bw_limited_tok_s":25.74,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":250.94,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":12,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":558.33,"bw_limited_tok_s":335,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.7,"score_norm":1.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":8.79,"score_norm":8.79,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":37.7,"score_norm":37.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-super-120b-a12b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12.8,"score_norm":12.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.67,"score_norm":25.67,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":60.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.603333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":1.84,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0184365781710914 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1360.3,"score_norm":35.96,"config":"7535 votes, arena entry march26-chatbot1, leaderboard text-overall-style_control, rank 187, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":3.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0314285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":9.9,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.09902999999999998 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80,"score_norm":77.46,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.8 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":20.76,"score_norm":27.59,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.207599629286376 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.5,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.714965986394558 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":1.13,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.0112994350282486 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.359953703703704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":67.84,"score_norm":63.79,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.678362573099415 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":10.31,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.103092783505155 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.79,"score_norm":27.99,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.287878787878788 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38.58,"score_norm":39.51,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.385767790262172 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-super-120b-a12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-super-120b-a12b","publisher":"OpenRouter (listed 2026-03-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3-ultra-550b-a55b","display_name":"Nemotron 3 Ultra","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3","size_class":"550b-a55b","params_total_b":560.52,"params_active_b":55,"params_exact":560524578816,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1121,"int8_gb":594,"int4_gb":336,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.51,"expected_int8_gb":306,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","openrouter_id":"nvidia/nemotron-3-ultra-550b-a55b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-06-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nvidia-nemotron-3-ultra","nvidia-nemotron-3-ultra-550b-a55b-bf16","nvidia-nemotron-3-ultra-550b-a55b-nvfp4"],"intelligence":{"index":68.19,"inputs":[{"benchmark":"arena_elo","score_raw":1426.2,"score_norm":65.33,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":86.11,"score_norm":86.07,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.738, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":28.41,"score_norm":38.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.284059314179796 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":85.76,"score_norm":76.27,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.345, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":69,"score_norm":65.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.07, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":83.33,"score_norm":84.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":50.94,"score_norm":54.05,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=2.925, task=overall, updated=2026-08-19"}],"coverage":0.7223,"subset_bias":{"offset":1.59,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":2.2,"cache_read_usd_per_m":0.1,"context_length":262144,"source":"openrouter","source_model_id":"nvidia/nemotron-3-ultra-550b-a55b","as_of":"2026-10-06","blended_usd_per_m":0.925,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1121.04,"kv_cache_gb_per_8k":null,"min_serve_gb":1179.292,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1121.056,"kv_cache_gb_per_8k":null,"min_serve_gb":1179.309,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/tree/main"},{"quant":"fp8","weights_gb":594.151,"kv_cache_gb_per_8k":null,"min_serve_gb":626.059,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":336.312,"kv_cache_gb_per_8k":null,"min_serve_gb":355.328,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":594.151,"kv_cache_gb_per_8k":null,"min_serve_gb":626.059,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":20.1,"score_norm":20.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":27.5,"score_norm":27.5,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":49.3,"score_norm":49.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.9,"score_norm":22.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.32,"score_norm":38.32,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.71 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_lcr","score_raw":65.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":6.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0625 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1426.2,"score_norm":65.33,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1427,"score_norm":65.68,"config":"1427 +/-7, entry nvidia-nemotron-3-ultra-550b-a55b-nvfp4, lmarena text leaderboard","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":5.67,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.05670562350908316 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":44.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0314285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1146,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1105,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1150,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1152,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1068,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1146,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1146,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":28.89,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.28886899433005075 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":46.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":33.15,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.331475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.11,"score_norm":86.07,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.738, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.67,"score_norm":86.86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.866666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87,"score_norm":87.32,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":81.72,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.81719496309162 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":28.41,"score_norm":38.84,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.284059314179796 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37.4,"score_norm":52.06,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":26.7,"score_norm":36.32,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":81.36,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.813605442176871 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":89,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":11.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.111111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.76,"score_norm":76.27,"config":"edition=mmlu pro v1, runner=platform, stderr=0.345, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.8,"score_norm":78,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ruler","score_raw":94.7,"score_norm":null,"config":"1m context","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.93,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.399305555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69,"score_norm":65.56,"config":"edition=swe-bench verified v1, runner=external, stderr=2.07, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":71.9,"score_norm":68.78,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":83.33,"score_norm":84.44,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":70.9,"score_norm":67.87,"config":"taubench v3 average of airline/retail/telecom/banking, vendor-reported via nemo evaluator sdk","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":14.23,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.142268041237113 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench_v3_average","score_raw":70.9,"score_norm":null,"config":"airline/retail/telecom/banking average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.94,"score_norm":54.05,"config":"edition=terminal-bench 2.1, runner=external, stderr=2.925, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":36.36,"score_norm":36.89,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.363636363636364 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.93,"score_norm":57.56,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.539325842696629 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nvidia-nemotron-3-ultra-550b-a55b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.4,"score_norm":60.47,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3-ultra-550b-a55b","publisher":"OpenRouter (listed 2026-06-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3.5-content-safety","display_name":"Nemotron 3.5 Content Safety","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3.5","size_class":"frontier","params_total_b":4.3,"params_active_b":4.3,"params_exact":4300079472,"params_source":"hf_safetensors","vram_est":{"bf16_gb":8.6,"int8_gb":4.6,"int4_gb":2.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"nvidia/Nemotron-3.5-Content-Safety","openrouter_id":"nvidia/nemotron-3.5-content-safety","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_edge","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2026-06-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":0.2,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"nvidia/nemotron-3.5-content-safety","as_of":"2026-10-06","blended_usd_per_m":0.2,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":8.6,"kv_cache_gb_per_8k":0.2894,"min_serve_gb":9.889,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":8.6,"kv_cache_gb_per_8k":0.2894,"min_serve_gb":9.889,"method":"vendor","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety/tree/main"},{"quant":"fp8","weights_gb":4.558,"kv_cache_gb_per_8k":0.2894,"min_serve_gb":5.847,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.58,"kv_cache_gb_per_8k":0.2894,"min_serve_gb":3.869,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.558,"kv_cache_gb_per_8k":0.2894,"min_serve_gb":5.847,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":78.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":141.4,"bw_limited_tok_s":84.84,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":36.7,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":63.49,"bw_limited_tok_s":38.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":318.54,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":4.3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":779.07,"bw_limited_tok_s":467.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aegis_2_0","score_raw":86,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"linguasafe","score_raw":71,"score_norm":null,"config":"average f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mm_safetybench","score_raw":71,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"polyguard","score_raw":80,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"vlguard","score_raw":90,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildguard","score_raw":85,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"xstest","score_raw":85,"score_norm":null,"config":"prompt harmful f1 (x100)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3.5-content-safety:free","publisher":"OpenRouter (listed 2026-06-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3.5-lightning","display_name":"Nemotron 3.5 Lightning","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3.5","size_class":"frontier","params_total_b":31.58,"params_active_b":3,"params_exact":31577937344,"params_source":"hf_safetensors","vram_est":{"bf16_gb":63.2,"int8_gb":33.5,"int4_gb":18.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":5.05,"expected_int8_gb":169,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","openrouter_id":"nvidia/nemotron-3.5-lightning","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-08-11","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["nemotron-3-super"],"superseded_by":null,"aliases":["nvidia-nemotron-3.5-lightning-30b-a3b-bf16"],"intelligence":{"index":49,"inputs":[{"benchmark":"gpqa_diamond","score_raw":74.34,"score_norm":69.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.743434343434343 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":10.57,"score_norm":12.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.105653382761817 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":81.94,"score_norm":69.9,"trust":1,"source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","config":"bf16, nvidia nemo gym harness"},{"benchmark":"swe_bench_verified","score_raw":51.56,"score_norm":46.18,"trust":1,"source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","config":"bf16"},{"benchmark":"terminal_bench","score_raw":24.34,"score_norm":22.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.243445692883895 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5953,"subset_bias":{"offset":-1.43,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.06,"completion_usd_per_m":0.16,"cache_read_usd_per_m":0.03,"context_length":262144,"source":"openrouter","source_model_id":"nvidia/nemotron-3.5-lightning","as_of":"2026-10-06","blended_usd_per_m":0.08499999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":63.16,"kv_cache_gb_per_8k":null,"min_serve_gb":66.438,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.827,"kv_cache_gb_per_8k":null,"min_serve_gb":69.238,"method":"vendor","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/tree/main"},{"quant":"fp8","weights_gb":33.475,"kv_cache_gb_per_8k":null,"min_serve_gb":35.269,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.948,"kv_cache_gb_per_8k":null,"min_serve_gb":20.068,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":33.475,"kv_cache_gb_per_8k":null,"min_serve_gb":35.269,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":72.4,"quant":"int4","batch_size":1,"context_tokens":262144,"method":"community","trust":0,"source_url":"https://blog.kubesimplify.com/nemotron-3-5-lightning-on-dgx-spark","derivation":null},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":3.5,"score_norm":3.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3.5-lightning","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":13.85,"score_norm":13.85,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":26.8,"score_norm":26.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3.5-lightning","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12.9,"score_norm":12.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3.5-lightning","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23.63,"score_norm":23.63,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":55.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.553333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_lcr","score_raw":52,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_omniscience","score_raw":17.5,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":36.97,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":16.18,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.16179500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":832,"score_norm":null,"config":"elo, bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.34,"score_norm":69.49,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.743434343434343 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.57,"score_norm":71.23,"config":"nvfp4 quantized checkpoint","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.44,"score_norm":71.04,"config":"bf16, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.57,"score_norm":12.6,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.105653382761817 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.72,"score_norm":14.29,"config":"bf16, text-only, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.88,"score_norm":null,"config":"bf16, loose","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.94,"score_norm":69.9,"config":"bf16, nvidia nemo gym harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"pinchbench","score_raw":85.37,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":31.6,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.315972222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":32.6,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":39.33,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":51.56,"score_norm":46.18,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":9.28,"score_norm":null,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":8.87,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.088659793814433 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.34,"score_norm":22.75,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.243445692883895 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nemotron-3-5-lightning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.58,"score_norm":23.04,"config":"bf16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","measured_date":"2026-08-11","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nvidia/nemotron-3.5-lightning","publisher":"OpenRouter (listed 2026-08-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nex-n2-mini","display_name":"nex-n2-mini","family":{"id":"nex","vendor":"nex-agi"},"generation":"2","size_class":"frontier-n-mini","params_total_b":35.11,"params_active_b":null,"params_exact":35107181936,"params_source":"hf_safetensors","vram_est":{"bf16_gb":70.2,"int8_gb":37.2,"int4_gb":21.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":10.49,"expected_int8_gb":390,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nex-agi/Nex-N2-Mini","openrouter_id":"nex-agi/nex-n2-mini","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-06-24","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":74.94,"inputs":[{"benchmark":"gpqa_diamond","score_raw":82.6,"score_norm":81.13,"trust":1,"source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","config":"agentic eval suite"},{"benchmark":"swe_bench_verified","score_raw":74.4,"score_norm":71.56,"trust":1,"source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","config":"with tools (agentic)"},{"benchmark":"terminal_bench","score_raw":60.7,"score_norm":65.53,"trust":1,"source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","config":"terminal-bench 2.1"}],"coverage":0.3651,"subset_bias":{"offset":5.93,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":70.22,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":73.899,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":70.214,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":73.893,"method":"vendor","source_url":"https://huggingface.co/nex-agi/Nex-N2-Mini/tree/main"},{"quant":"fp8","weights_gb":37.217,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":39.246,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":21.066,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":22.287,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":37.217,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":39.246,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":20.36,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":35.11,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":34.63,"bw_limited_tok_s":20.78,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.38,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":35.11,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":7.78,"bw_limited_tok_s":4.4,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":54.15,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":35.11,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":95.41,"bw_limited_tok_s":57.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"apex","score_raw":9.4,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":74.1,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":8,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1402,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.6,"score_norm":81.13,"config":"agentic eval suite","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":89.1,"score_norm":null,"config":"default","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":50.2,"score_norm":75.33,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74.4,"score_norm":71.56,"config":"with tools (agentic)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench","score_raw":65.9,"score_norm":null,"config":"tau3 (tau-cubed-bench)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.7,"score_norm":65.53,"config":"terminal-bench 2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon","score_raw":33.3,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"widesearch","score_raw":62,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"wildclawbench","score_raw":47.7,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2-mini","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-mini","publisher":"OpenRouter (listed 2026-06-24)"}],"last_verified":"2026-09-08","last_enriched_at":null},{"id":"nex-n2-pro","display_name":"nex-n2-pro","family":{"id":"nex","vendor":"nex-agi"},"generation":"2","size_class":"frontier-n-pro","params_total_b":396.8,"params_active_b":null,"params_exact":396802360816,"params_source":"hf_safetensors","vram_est":{"bf16_gb":794,"int8_gb":421,"int4_gb":238,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.89,"expected_int8_gb":374,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"nex-agi/Nex-N2-Pro","openrouter_id":"nex-agi/nex-n2-pro","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-06-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":73.86,"inputs":[{"benchmark":"gpqa_diamond","score_raw":89.19,"score_norm":90.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/nex-n2-pro","config":"artificial analysis independent run"},{"benchmark":"hle","score_raw":33.73,"score_norm":46.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/nex-n2-pro","config":"artificial analysis independent run, no tools"},{"benchmark":"tau_bench","score_raw":81.58,"score_norm":82.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/nex-n2-pro","config":"artificial analysis independent run, tau2, with tools"},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/nex-n2-pro","config":"2.1, artificial analysis independent run"}],"coverage":0.3651,"subset_bias":{"offset":-1.1,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":793.6,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":833.532,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":793.605,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":833.537,"method":"vendor","source_url":"https://huggingface.co/nex-agi/Nex-N2-Pro/tree/main"},{"quant":"fp8","weights_gb":420.608,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":441.89,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":238.08,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":250.236,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":420.608,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":441.89,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":59.1,"score_norm":59.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81.67,"score_norm":null,"config":"artificial analysis aa-lcr v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8.57,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1282,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1126,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1250,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1253,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1223,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","measured_date":null,"observed_at":"2026-09-08","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.19,"score_norm":90.41,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.73,"score_norm":46.66,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":66.19,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":81.58,"score_norm":82.11,"config":"artificial analysis independent run, tau2, with tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"config":"2.1, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nex-n2-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2-pro","publisher":"OpenRouter (listed 2026-06-08)"}],"last_verified":"2026-09-08","last_enriched_at":null},{"id":"nex-n2.5-mini","display_name":"Nex-N2.5-Mini","family":{"id":"nex","vendor":"nex-agi"},"generation":"2.5","size_class":"frontier-n-mini","params_total_b":35.11,"params_active_b":null,"params_exact":35107181936,"params_source":"hf_safetensors","vram_est":{"bf16_gb":70.2,"int8_gb":37.2,"int4_gb":21.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"nex-agi/Nex-N2.5-mini","openrouter_id":"nex-agi/nex-n2.5-mini","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-09-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.025,"completion_usd_per_m":0.1,"cache_read_usd_per_m":0.0025,"context_length":262144,"source":"openrouter","source_model_id":"nex-agi/nex-n2.5-mini","as_of":"2026-10-06","blended_usd_per_m":0.043750000000000004,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":70.22,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":73.899,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":70.214,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":73.893,"method":"vendor","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini/tree/main"},{"quant":"fp8","weights_gb":37.217,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":39.246,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":21.066,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":22.287,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":37.217,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":39.246,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":20.36,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":35.11,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":34.63,"bw_limited_tok_s":20.78,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.38,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":35.11,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":7.78,"bw_limited_tok_s":4.4,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":54.15,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":35.11,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":95.41,"bw_limited_tok_s":57.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"automationbench","score_raw":32.3,"score_norm":null,"config":"1.0.6, public version, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":83.4,"score_norm":null,"config":"summary context-compaction when token usage exceeds 60% of context window, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":36.1,"score_norm":null,"config":"1.1, nexau harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1446,"score_norm":null,"config":"elo, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":28.5,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"omnidoc","score_raw":89.7,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":30.5,"score_norm":null,"config":"2, grounding coordinates normalized 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":71.2,"score_norm":null,"config":"grounding coordinates normalized 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95, verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld_g","score_raw":82.9,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":43.8,"score_norm":64.67,"config":"nexau harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"swe_mm","score_raw":25.5,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":73.4,"score_norm":80.47,"config":"2.1, coding tasks, nexau harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":54.6,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"vision2web","score_raw":52.9,"score_norm":null,"config":"and website, average across frontend, gemini-3.5-flash vlm judge, glm-5v-turbo (claude code) gui agent, temperature=0.7, top_k=40, top_p=0.95, webpage","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"webarena_verified","score_raw":63.4,"score_norm":null,"config":"grounding coordinates normalized 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"webtest","score_raw":48.6,"score_norm":null,"config":"nexcua harness, oracle mode using ground-truth checklist for defect detection only, temperature=0.7, top_k=40, top_p=0.95, without checklist generation","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-mini","measured_date":"2026-09-08","observed_at":"2026-09-30","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2.5-mini:free","publisher":"OpenRouter (listed 2026-09-08)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-06T06:09:09.959Z"},{"id":"nex-n2.5-pro","display_name":"Nex-N2.5-Pro","family":{"id":"nex","vendor":"nex-agi"},"generation":"2.5","size_class":"frontier-n-pro","params_total_b":396.8,"params_active_b":null,"params_exact":396802360816,"params_source":"hf_safetensors","vram_est":{"bf16_gb":794,"int8_gb":421,"int4_gb":238,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"nex-agi/Nex-N2.5-Pro","openrouter_id":"nex-agi/nex-n2.5-pro","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-09-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.075,"completion_usd_per_m":0.25,"cache_read_usd_per_m":0.015,"context_length":262144,"source":"openrouter","source_model_id":"nex-agi/nex-n2.5-pro","as_of":"2026-10-06","blended_usd_per_m":0.11875,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":793.6,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":833.532,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":420.608,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":441.89,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":407.131,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":427.739,"method":"vendor","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-Pro/tree/main"},{"quant":"int4","weights_gb":238.08,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":250.236,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":420.608,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":441.89,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"automationbench","score_raw":44.2,"score_norm":null,"config":"1.0.6, public version, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":89.7,"score_norm":null,"config":"summary context-compaction when token usage exceeds 60% of context window, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":55.8,"score_norm":null,"config":"1.1, nexau harness, reasoning_effort default medium (adaptive thinking), temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1628,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":41.4,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"omnidoc","score_raw":92.2,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":56.4,"score_norm":null,"config":"2, grounding coordinates normalized to 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":82.2,"score_norm":null,"config":"grounding coordinates normalized to 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95, verified","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"osworld_g","score_raw":87.4,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":61.2,"score_norm":93.67,"config":"nexau harness, reasoning_effort default medium (adaptive thinking), temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":61.2,"score_norm":93.67,"config":"nexau harness.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_mm","score_raw":38.2,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"2.1, nexau harness, reasoning_effort default medium (adaptive thinking), temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"2.1, nexau harness.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nex-agi/Nex-N2.5-Pro","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":68.5,"score_norm":null,"config":"temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"vision2web","score_raw":68.2,"score_norm":null,"config":"and website, average across frontend, gemini-3.5-flash vlm judge, glm-5v-turbo (claude code) gui agent, temperature=0.7, top_k=40, top_p=0.95, webpage","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"webarena_verified","score_raw":67.6,"score_norm":null,"config":"grounding coordinates normalized to 0–1000, nexcua harness, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"webtest","score_raw":52.8,"score_norm":null,"config":"defect detection only, nexcua harness, no checklist generation), oracle mode (ground-truth checklist, temperature=0.7, top_k=40, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://github.com/nex-agi/Nex-N2.5","measured_date":"2026-09-10","observed_at":"2026-09-30","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nex-agi/nex-n2.5-pro:free","publisher":"OpenRouter (listed 2026-09-08)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-06T06:09:14.906Z"},{"id":"nova-1-micro","display_name":"Nova Micro 1.0","family":{"id":"nova","vendor":"Amazon"},"generation":"1","size_class":"frontier-micro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"amazon/nova-micro-v1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-12-05","release_status":"released","knowledge_cutoff":"2024-10-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["nova-micro-v1"],"intelligence":{"index":6.38,"inputs":[{"benchmark":"aime","score_raw":6,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-micro","config":"aa eval, aime 2025"},{"benchmark":"gpqa_diamond","score_raw":35.76,"score_norm":15.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-micro","config":"aa intelligence index eval"},{"benchmark":"hle","score_raw":4.63,"score_norm":3.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-micro","config":"aa eval, no tools"},{"benchmark":"tau_bench","score_raw":14.04,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-micro","config":"aa eval, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-micro","config":"aa eval, terminal-bench hard"}],"coverage":0.4445,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.035,"completion_usd_per_m":0.14,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"amazon/nova-micro-v1","as_of":"2026-10-06","blended_usd_per_m":0.061250000000000006,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.9,"score_norm":5.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-micro-v1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":6,"score_norm":0,"config":"aa eval, aime 2025","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_c","score_raw":90.2,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":79.5,"score_norm":null,"config":"3-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":79.3,"score_norm":null,"config":"6-shot cot, f1","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":35.76,"score_norm":15.15,"config":"aa intelligence index eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40,"score_norm":21.13,"config":"0-shot cot, main set","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":92.3,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":4.63,"score_norm":3.87,"config":"aa eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":87.2,"score_norm":null,"config":"0-shot, instruction-level loose acc","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":13.97,"score_norm":null,"config":"aa eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":69.3,"score_norm":null,"config":"0-shot cot, math5k exact match","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":77.6,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":14.04,"score_norm":0,"config":"aa eval, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa eval, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-micro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#amazon/nova-micro-v1","publisher":"OpenRouter (listed 2024-12-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nova-1-premier","display_name":"Nova Premier 1.0","family":{"id":"nova","vendor":"Amazon"},"generation":"1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"amazon/nova-premier-v1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-10-31","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["nova-premier-v1"],"intelligence":{"index":29.59,"inputs":[{"benchmark":"aime","score_raw":17.3,"score_norm":0,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"'nova 1 premier' row, aime 2025, aime 2025 (acc) column [raw: aime 2025], amazon nova 2 technical report, no tools, table 1"},{"benchmark":"gpqa_diamond","score_raw":57.1,"score_norm":45.21,"trust":1,"source_url":"https://cdn.amazon.science/e5/e6/ccc5378c42dca467d1abe1628ec9/amazon-nova-premier-technical-report-and-model-card.pdf","config":"'nova premier' column (first data column), 0-shot cot, amazon nova premier technical report & model card, exact-match accuracy, no tools, reported as 57.1 +/- 6.9 [raw: gpqa diamond], table 1"},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-premier","config":"artificial analysis intelligence index v4.1.1 hle eval, i.e. 4.17% [raw: humanity's last exam (hle)], no tools, page data payload for slug 'nova-premier' carries hle=0.0417052826691381 (fraction)"},{"benchmark":"mmlu_pro","score_raw":73.3,"score_norm":55.5,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"'nova 1 premier' row, amazon nova 2 technical report, mmlu-pro (acc) column [raw: mmlu-pro], table 1"},{"benchmark":"swe_bench_verified","score_raw":42.4,"score_norm":36,"trust":1,"source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","config":"internal agentic scaffold, resolved rate"},{"benchmark":"tau_bench","score_raw":38.3,"score_norm":24.4,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"'nova 1 premier' row. footnote a: [raw: tau2-bench (telecom)], retail-verified and airline-verified are '-' (not evaluated) for nova 1 premier. amazon nova 2 technical report, table 2, telecom domain only - no average available"},{"benchmark":"terminal_bench","score_raw":11.3,"score_norm":7.41,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"'nova 1 premier' row [raw: terminal-bench 1.0], accuracy, amazon nova 2 technical report, table 4 (software engineering), terminal-bench 1.0"}],"coverage":0.7223,"subset_bias":{"offset":0.71,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":12.5,"cache_read_usd_per_m":0.625,"context_length":1000000,"source":"openrouter","source_model_id":"amazon/nova-premier-v1","as_of":"2026-10-06","blended_usd_per_m":5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.2,"score_norm":9.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-premier-v1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13,"score_norm":13,"config":"9-eval composite, current aa index version (aug 2026)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-premier","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":17.3,"score_norm":0,"config":"'nova 1 premier' row, aime 2025, aime 2025 (acc) column [raw: aime 2025], amazon nova 2 technical report, no tools, table 1","trust":1,"source_type":"paper","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"bfcl","score_raw":63.7,"score_norm":null,"config":"berkeley function calling leaderboard","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"charxiv","score_raw":84.6,"score_norm":null,"config":"chart understanding","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":842,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-premier-v1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.1,"score_norm":45.21,"config":"'nova premier' column (first data column), 0-shot cot, amazon nova premier technical report & model card, exact-match accuracy, no tools, reported as 57.1 +/- 6.9 [raw: gpqa diamond], table 1","trust":1,"source_type":"paper","source_url":"https://cdn.amazon.science/e5/e6/ccc5378c42dca467d1abe1628ec9/amazon-nova-premier-technical-report-and-model-card.pdf","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.17,"score_norm":3.19,"config":"artificial analysis intelligence index v4.1.1 hle eval, i.e. 4.17% [raw: humanity's last exam (hle)], no tools, page data payload for slug 'nova-premier' carries hle=0.0417052826691381 (fraction)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-premier","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"math_500","score_raw":82,"score_norm":null,"config":"0-shot cot, exact match","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":87.4,"score_norm":null,"config":"0-shot cot, vendor announcement","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":73.3,"score_norm":55.5,"config":"'nova 1 premier' row, amazon nova 2 technical report, mmlu-pro (acc) column [raw: mmlu-pro], table 1","trust":1,"source_type":"paper","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"simpleqa","score_raw":86.3,"score_norm":null,"config":"with rag","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":42.4,"score_norm":36,"config":"internal agentic scaffold, resolved rate","trust":1,"source_type":"vendor_claim","source_url":"https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-nova-premier-complex-tasks-model-distillation","measured_date":"2025-04-30","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":38.3,"score_norm":24.4,"config":"'nova 1 premier' row. footnote a: [raw: tau2-bench (telecom)], retail-verified and airline-verified are '-' (not evaluated) for nova 1 premier. amazon nova 2 technical report, table 2, telecom domain only - no average available","trust":1,"source_type":"paper","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":11.3,"score_norm":7.41,"config":"'nova 1 premier' row [raw: terminal-bench 1.0], accuracy, amazon nova 2 technical report, table 4 (software engineering), terminal-bench 1.0","trust":1,"source_type":"paper","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#amazon/nova-premier-v1","publisher":"OpenRouter (listed 2025-10-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nova-1-pro","display_name":"Nova Pro 1.0","family":{"id":"nova","vendor":"Amazon"},"generation":"1","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"amazon/nova-pro-v1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":300000,"release_date":"2024-12-05","release_status":"released","knowledge_cutoff":"2024-10-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["nova-pro-v1"],"intelligence":{"index":18.61,"inputs":[{"benchmark":"aime","score_raw":7,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=nova-pro","config":"aime 2025, artificial analysis harness"},{"benchmark":"arena_elo","score_raw":1290,"score_norm":4.62,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"24, 745 votes, arena entry amazon-nova-pro-v1.0, rank 275, text overall"},{"benchmark":"gpqa_diamond","score_raw":49.9,"score_norm":35.07,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=nova-pro","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":3.2,"score_norm":1.76,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=nova-pro","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"mmlu_pro","score_raw":69.1,"score_norm":48.5,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=nova-pro","config":"artificial analysis harness, mmlu-pro"},{"benchmark":"tau_bench","score_raw":14,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=nova-pro","config":"artificial analysis harness, tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":6.1,"score_norm":1.29,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=nova-pro","config":"artificial analysis harness, terminal-bench hard"}],"coverage":0.6428,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.8,"completion_usd_per_m":3.2,"cache_read_usd_per_m":null,"context_length":300000,"source":"openrouter","source_model_id":"amazon/nova-pro-v1","as_of":"2026-10-06","blended_usd_per_m":1.4000000000000001,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7,"score_norm":7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-pro-v1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":7,"score_norm":0,"config":"aime 2025, artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/aime-2025?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_c","score_raw":94.8,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1290,"score_norm":4.62,"config":"24, 745 votes, arena entry amazon-nova-pro-v1.0, rank 275, text overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bigbench_hard","score_raw":86.9,"score_norm":null,"config":"3-shot cot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":803,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-pro-v1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":85.4,"score_norm":null,"config":"6-shot cot, f1","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49.9,"score_norm":35.07,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.9,"score_norm":30.85,"config":"0-shot cot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":94.8,"score_norm":null,"config":"0-shot cot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.2,"score_norm":1.76,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":89,"score_norm":null,"config":"0-shot pass@1","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":92.1,"score_norm":null,"config":"0-shot, instruction-level loose accuracy","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":76.6,"score_norm":null,"config":"0-shot cot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":85.9,"score_norm":null,"config":"0-shot cot","trust":2,"source_type":"paper","source_url":"https://ar5iv.labs.arxiv.org/html/2506.12103","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.1,"score_norm":48.5,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":14,"score_norm":0,"config":"artificial analysis harness, tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/tau2-bench?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.1,"score_norm":1.29,"config":"artificial analysis harness, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/terminalbench-hard?models=nova-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#amazon/nova-pro-v1","publisher":"OpenRouter (listed 2024-12-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nova-2-lite-v1","display_name":"Nova 2 Lite","family":{"id":"nova","vendor":"Amazon"},"generation":"2","size_class":"frontier-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"amazon/nova-2-lite-v1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-12-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["nova-1-lite"],"superseded_by":null,"aliases":[],"intelligence":{"index":54.53,"inputs":[{"benchmark":"aime","score_raw":91,"score_norm":88.75,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"acc, aime 2025"},{"benchmark":"arena_elo","score_raw":1335.6,"score_norm":24.95,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"12294 votes, arena entry nova-2-lite, leaderboard text-overall-style_control, rank 221, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":79.6,"score_norm":76.9,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"acc"},{"benchmark":"hle","score_raw":11.63,"score_norm":14.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/nova-2-0-lite-reasoning","config":"aa intelligence index eval, nova 2.0 lite (high)"},{"benchmark":"mmlu_pro","score_raw":80.9,"score_norm":68.17,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"acc"},{"benchmark":"swe_bench_verified","score_raw":53.6,"score_norm":48.44,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"no inference-time scaling / no internal agentic scaffolding"},{"benchmark":"tau_bench","score_raw":76,"score_norm":74.67,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"tau2-bench telecom"},{"benchmark":"terminal_bench","score_raw":32.5,"score_norm":32.35,"trust":1,"source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","config":"terminal-bench 1.0"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":2.5,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"amazon/nova-2-lite-v1","as_of":"2026-10-06","blended_usd_per_m":0.85,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":23,"score_norm":23,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-2-lite-v1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.4,"score_norm":13.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-2-lite-v1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19,"score_norm":19,"config":"reasoning, medium effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-2-0-lite-reasoning-medium","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":91,"score_norm":88.75,"config":"acc, aime 2025","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":91,"score_norm":88.75,"config":"amazon nova 2 technical report","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1335.6,"score_norm":24.95,"config":"12294 votes, arena entry nova-2-lite, leaderboard text-overall-style_control, rank 221, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"bfcl","score_raw":60.3,"score_norm":null,"config":"bfcl v4 overall fc","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.6,"score_norm":76.9,"config":"acc","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.6,"score_norm":76.9,"config":"amazon nova 2 technical report","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":11.63,"score_norm":14.16,"config":"aa intelligence index eval, nova 2.0 lite (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/nova-2-0-lite-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"if_bench","score_raw":70.8,"score_norm":null,"config":"prompt loose acc","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":70.8,"score_norm":null,"config":"amazon nova 2 technical report (if-bench)","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":71,"score_norm":null,"config":"code generation acc, v5 (2024-07-01 to 2025-01-01 subset)","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"longcodebench","score_raw":84,"score_norm":null,"config":"1m match","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"longcodebench_1m","score_raw":84,"score_norm":null,"config":"amazon nova 2 technical report","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.9,"score_norm":68.17,"config":"acc","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.9,"score_norm":68.17,"config":"amazon nova 2 technical report text-benchmark table","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"multichallenge","score_raw":76.6,"score_norm":null,"config":"acc","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multichallenge","score_raw":76.6,"score_norm":null,"config":"amazon nova 2 technical report","trust":0,"source_type":"vendor_claim","source_url":"https://assets.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":"2025-12-02","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":53.6,"score_norm":48.44,"config":"no inference-time scaling / no internal agentic scaffolding","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76,"score_norm":74.67,"config":"tau2-bench telecom","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":32.5,"score_norm":32.35,"config":"terminal-bench 1.0","trust":1,"source_type":"vendor_claim","source_url":"https://cdn.amazon.science/c5/3d/84514a224666b5be6de4b43ef4aa/nova-2-0-technical-report2.pdf","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#amazon/nova-2-lite-v1","publisher":"OpenRouter (listed 2025-12-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"olmo-2-0325-32b-instruct","display_name":"OLMo-2-0325-32B-Instruct","family":{"id":"olmo","vendor":"Allen Institute"},"generation":"2","size_class":"32b","params_total_b":32.23,"params_active_b":32,"params_exact":32234279936,"params_source":"hf_safetensors","vram_est":{"bf16_gb":64.5,"int8_gb":34.2,"int4_gb":19.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"allenai/OLMo-2-0325-32B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":4096,"release_date":"2025-03","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":64.46,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":69.83,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":64.469,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":69.84,"method":"vendor","source_url":"https://huggingface.co/allenai/OLMo-2-0325-32B-Instruct/tree/main"},{"quant":"fp8","weights_gb":34.164,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.02,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.338,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":22.486,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.164,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.02,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":22.29,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":38,"bw_limited_tok_s":22.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.09,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.53,"bw_limited_tok_s":5.12,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":59.1,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":104.69,"bw_limited_tok_s":62.81,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arena_elo","score_raw":1251.6,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/allenai/OLMo-2-0325-32B","publisher":"Hugging Face"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"olmo-3-32b-reasoning","display_name":"olmo-3-32b-think","family":{"id":"olmo","vendor":"Allen Institute"},"generation":"3","size_class":"32b","params_total_b":32.23,"params_active_b":32.23,"params_exact":32233522176,"params_source":"hf_safetensors","vram_est":{"bf16_gb":64.5,"int8_gb":34.2,"int4_gb":19.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.59,"expected_int8_gb":20.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"allenai/Olmo-3-32B-Think","openrouter_id":"allenai/olmo-3-32b-think","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2025-11-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["olmo-3-32b-think"],"intelligence":{"index":28.73,"inputs":[{"benchmark":"aime","score_raw":73.67,"score_norm":67.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.736666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1306.4,"score_norm":11.93,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":61.01,"score_norm":50.72,"trust":2,"source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.61010101010101 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.44,"score_norm":6.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0644114921223355 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-3.01,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":64.46,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":69.025,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":64.467,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":69.033,"method":"vendor","source_url":"https://huggingface.co/allenai/Olmo-3-32B-Think/tree/main"},{"quant":"fp8","weights_gb":34.164,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":37.214,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.338,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":21.68,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.164,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":37.214,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":22.14,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32.23,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":37.73,"bw_limited_tok_s":22.64,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32.23,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.47,"bw_limited_tok_s":5.08,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":58.7,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32.23,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":103.94,"bw_limited_tok_s":62.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.15,"score_norm":6.15,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":73.67,"score_norm":67.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.736666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1306.4,"score_norm":11.93,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1305.3,"score_norm":11.44,"config":"5968 votes, arena entry olmo-3-32b-think, leaderboard text-overall-style_control, rank 262, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.01,"score_norm":50.72,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.61010101010101 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.44,"score_norm":6.53,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0644114921223355 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":49.12,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.491156462585034 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":67.2,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.671957671957672 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":28.59,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.28587962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":1.52,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/olmo-3-32b-think","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#allenai/olmo-3-32b-think","publisher":"OpenRouter (listed 2025-11-21)"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"oluo-2-70b","display_name":"Oluo 2 70B","family":{"id":"oluo","vendor":"Allen Institute for AI"},"generation":"2","size_class":"70b","params_total_b":70,"params_active_b":70,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":140,"int8_gb":74.2,"int4_gb":42,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"allenai/Oluo-2-1124-70B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"Apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2025-11-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["oluo-2-1124-70b-instruct"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":140,"kv_cache_gb_per_8k":null,"min_serve_gb":149.8,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":74.2,"kv_cache_gb_per_8k":null,"min_serve_gb":80.71,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42,"kv_cache_gb_per_8k":null,"min_serve_gb":46.9,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.2,"kv_cache_gb_per_8k":null,"min_serve_gb":80.71,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":70,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.9,"bw_limited_tok_s":2.34,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":54.31,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":70,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":95.71,"bw_limited_tok_s":57.43,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/allenai/Oluo-2-1124-70B-Instruct","publisher":"Hugging Face"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"palmyra-x5","display_name":"Palmyra X5","family":{"id":"palmyra","vendor":"writer"},"generation":"5","size_class":"frontier-x","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"writer/palmyra-x5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1040000,"release_date":"2026-01-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":6,"cache_read_usd_per_m":null,"context_length":1040000,"source":"openrouter","source_model_id":"writer/palmyra-x5","as_of":"2026-10-06","blended_usd_per_m":1.95,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"bbh","score_raw":70.99,"score_norm":null,"config":"labelled 'bbh (big-bench hard)'","trust":1,"source_type":"vendor_claim","source_url":"https://dev.writer.com/home/models","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":47.2,"score_norm":31.27,"config":"labelled 'gpqa (graduate-level google-proof q&a)', subset not stated (not identified as diamond)","trust":1,"source_type":"vendor_claim","source_url":"https://dev.writer.com/home/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math_hard","score_raw":71.57,"score_norm":null,"config":"labelled math_hard on writer's model docs page","trust":1,"source_type":"vendor_claim","source_url":"https://dev.writer.com/home/models","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":65.02,"score_norm":41.7,"config":"labelled mmlu_pro on writer's model docs page","trust":1,"source_type":"vendor_claim","source_url":"https://dev.writer.com/home/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#writer/palmyra-x5","publisher":"OpenRouter (listed 2026-01-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"pareto-26.10","display_name":"Pareto 26.10 Preview","family":{"id":"pareto","vendor":"Unbiased"},"generation":"26.10","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"unbiased/pareto-26.10-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-10-01","release_status":"preview","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["pareto-26.10-preview"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.8,"completion_usd_per_m":3.2,"cache_read_usd_per_m":0.03,"context_length":1048576,"source":"openrouter","source_model_id":"unbiased/pareto-26.10-preview","as_of":"2026-10-06","blended_usd_per_m":1.4000000000000001,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#unbiased/pareto-26.10-preview","publisher":"OpenRouter (listed 2026-10-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"perceptron-mk1","display_name":"Perceptron Mk1","family":{"id":"perceptron","vendor":"perceptron"},"generation":"1","size_class":"frontier-mk","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perceptron/perceptron-mk1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":32768,"release_date":"2026-05-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":1.5,"cache_read_usd_per_m":null,"context_length":32768,"source":"openrouter","source_model_id":"perceptron/perceptron-mk1","as_of":"2026-10-06","blended_usd_per_m":0.4875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"blink","score_raw":70.8,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":87.1,"score_norm":null,"config":"document/ocr/chart, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"cosmos_reason1","score_raw":76.8,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"countbench","score_raw":92.8,"score_norm":null,"config":"counting, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":97.2,"score_norm":null,"config":"document/ocr/chart, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"egoschema","score_raw":80.6,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"embspatialbench","score_raw":85.1,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"erqa","score_raw":62.5,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mvbench","score_raw":74,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"next_qa","score_raw":85.8,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ocrbench","score_raw":920.7,"score_norm":null,"config":"document/ocr/chart, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"perception_test","score_raw":80.8,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"realworldqa","score_raw":80.2,"score_norm":null,"config":"general vl reference, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"refcoco","score_raw":92.2,"score_norm":null,"config":"grounding and localization, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"refspatialbench","score_raw":72.4,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"robospatial_pointing","score_raw":82.8,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"robospatial_vqa","score_raw":84.1,"score_norm":null,"config":"er benchmarks table, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"screenspot_pro","score_raw":66,"score_norm":null,"config":"grounding and localization, image benchmarks, perceptron mk1 column","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tempcompass","score_raw":77.2,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tomato","score_raw":42.9,"score_norm":null,"config":"perceptron mk1 column, standard subset, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"vsi_bench","score_raw":88.5,"score_norm":null,"config":"hard subset, perceptron mk1 column, video benchmarks","trust":1,"source_type":"vendor_claim","source_url":"https://www.perceptron.inc/blog/introducing-perceptron-mk1","measured_date":"2026-05-12","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perceptron/perceptron-mk1","publisher":"OpenRouter (listed 2026-05-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"perceptron-mk1.5","display_name":"Perceptron Mk1.5","family":{"id":"perceptron","vendor":"perceptron"},"generation":"1.5","size_class":"frontier-mk","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perceptron/perceptron-mk1.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","audio"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":36864,"release_date":"2026-09-25","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":1.5,"cache_read_usd_per_m":null,"context_length":36864,"source":"openrouter","source_model_id":"perceptron/perceptron-mk1.5","as_of":"2026-10-06","blended_usd_per_m":0.4875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perceptron/perceptron-mk1.5","publisher":"OpenRouter (listed 2026-09-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"phi-4-14b-reasoning","display_name":"Phi-4-reasoning","family":{"id":"phi","vendor":"Microsoft"},"generation":"4","size_class":"14b","params_total_b":14,"params_active_b":14,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":28,"int8_gb":14.8,"int4_gb":8.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.56,"expected_int8_gb":52.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"MIT","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["phi-4-reasoning"],"intelligence":{"index":59.95,"inputs":[{"benchmark":"aime","score_raw":75.3,"score_norm":69.12,"trust":1,"source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","config":"aime 24, phi-4-reasoning column"},{"benchmark":"gpqa_diamond","score_raw":65.8,"score_norm":57.46,"trust":1,"source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","config":"gpqa-d, phi-4-reasoning column"},{"benchmark":"mmlu_pro","score_raw":74.3,"score_norm":57.17,"trust":1,"source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","config":"mmlupro, phi-4-reasoning column"}],"coverage":0.3572,"subset_bias":{"offset":16.01,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":28,"kv_cache_gb_per_8k":null,"min_serve_gb":29.96,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":14.84,"kv_cache_gb_per_8k":null,"min_serve_gb":16.4,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.4,"kv_cache_gb_per_8k":null,"min_serve_gb":9.96,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":14.84,"kv_cache_gb_per_8k":null,"min_serve_gb":16.4,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":25.4,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":43.43,"bw_limited_tok_s":26.06,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.56,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":19.5,"bw_limited_tok_s":11.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":125.55,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":14,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":239.29,"bw_limited_tok_s":143.57,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":62.9,"score_norm":53.63,"config":"aime 25, phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":75.3,"score_norm":69.12,"config":"aime 24, phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arenahard","score_raw":73.3,"score_norm":null,"config":"phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"flenqa","score_raw":97.7,"score_norm":null,"config":"3k-token subset","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":65.8,"score_norm":57.46,"config":"gpqa-d, phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humanevalplus","score_raw":92.9,"score_norm":null,"config":"phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":83.4,"score_norm":null,"config":"strict","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":53.8,"score_norm":null,"config":"8/1/24-2/1/25 window","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":74.3,"score_norm":57.17,"config":"mmlupro, phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"omnimath","score_raw":76.6,"score_norm":null,"config":"phi-4-reasoning column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.microsoft.com/en-us/research/publication/phi-4-reasoning-technical-report/","publisher":"Microsoft Research"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"phi-4-mini-3.8b-instruct","display_name":"Phi-4-mini-instruct","family":{"id":"phi","vendor":"Microsoft"},"generation":"4","size_class":"3.8b","params_total_b":3.84,"params_active_b":3.84,"params_exact":3836021760,"params_source":"hf_safetensors","vram_est":{"bf16_gb":7.7,"int8_gb":4.1,"int4_gb":2.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.02,"expected_int8_gb":8.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"microsoft/Phi-4-mini-instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-02-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["phi-4-mini-instruct"],"intelligence":{"index":10.67,"inputs":[{"benchmark":"gpqa_diamond","score_raw":33,"score_norm":11.27,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent"},{"benchmark":"hle","score_raw":4,"score_norm":2.94,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, no tools"},{"benchmark":"mmlu_pro","score_raw":52.8,"score_norm":21.33,"trust":1,"source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","config":"0-shot cot"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/leaderboards/models","config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, terminal-bench v2.1"}],"coverage":0.4365,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":7.68,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.754,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":7.672,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.746,"method":"vendor","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct/tree/main"},{"quant":"fp8","weights_gb":4.07,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.144,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.304,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":4.378,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.07,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.144,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":86.76,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":158.33,"bw_limited_tok_s":95,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":40.91,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":71.09,"bw_limited_tok_s":42.66,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":343.59,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":872.4,"bw_limited_tok_s":523.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc_challenge","score_raw":83.7,"score_norm":null,"config":"10-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_hard","score_raw":32.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bigbench_hard","score_raw":70.4,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":33,"score_norm":11.27,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":25.2,"score_norm":0.28,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":88.6,"score_norm":null,"config":"8-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":69.1,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":4,"score_norm":2.94,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math","score_raw":64,"score_norm":null,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":67.3,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":52.8,"score_norm":21.33,"config":"0-shot cot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"artificial analysis independent eval, leaderboard cell displayed rounded to whole percent, terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/leaderboards/models","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/microsoft/Phi-4-mini-instruct","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"phi-4-mini-3.8b-reasoning","display_name":"Phi-4-mini-reasoning","family":{"id":"phi","vendor":"Microsoft"},"generation":"4","size_class":"3.8b","params_total_b":3.84,"params_active_b":3.84,"params_exact":3836021760,"params_source":"hf_safetensors","vram_est":{"bf16_gb":7.7,"int8_gb":4.1,"int4_gb":2.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"microsoft/Phi-4-mini-reasoning","openrouter_id":null,"is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_edge","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-04-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["phi-4-mini-reasoning"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":7.68,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.754,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":7.672,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.746,"method":"vendor","source_url":"https://huggingface.co/microsoft/Phi-4-mini-reasoning/tree/main"},{"quant":"fp8","weights_gb":4.07,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.144,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.304,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":4.378,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.07,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.144,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":86.76,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":158.33,"bw_limited_tok_s":95,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":40.91,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":71.09,"bw_limited_tok_s":42.66,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":343.59,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.84,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":872.4,"bw_limited_tok_s":523.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":57.5,"score_norm":46.88,"config":"2024, pass@1, phi-4-mini-reasoning paper, table 3 (+rl final)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2504.21233v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":57.5,"score_norm":46.88,"config":"aime 2024, max seq len 32768","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":52,"score_norm":38.03,"config":"pass@1, phi-4-mini-reasoning paper, table 3 (+rl final)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2504.21233v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":52,"score_norm":38.03,"config":"max seq len 32768","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math_500","score_raw":94.6,"score_norm":null,"config":"pass@1, phi-4-mini-reasoning paper, table 3 (+rl final)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2504.21233v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":94.6,"score_norm":null,"config":"max seq len 32768","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/microsoft/Phi-4-mini-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/unsloth/Phi-4-mini-reasoning-unsloth-bnb-4bit","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-2.5-0.5b","display_name":"Qwen2.5-0.5B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"0.5b","params_total_b":0.49,"params_active_b":0.49,"params_exact":494032768,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1,"int8_gb":0.5,"int4_gb":0.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":5.39,"expected_int8_gb":2.8,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-0.5B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-09-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-0.5b","qwen2.5-0.5b-instruct"],"intelligence":{"index":2.7,"inputs":[{"benchmark":"gpqa_diamond","score_raw":29.8,"score_norm":6.76,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)"},{"benchmark":"livebench","score_raw":12.6,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)"},{"benchmark":"mmlu_pro","score_raw":15,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)"}],"coverage":0.3969,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":0.98,"kv_cache_gb_per_8k":0.1007,"min_serve_gb":2.081,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":0.988,"kv_cache_gb_per_8k":0.1007,"min_serve_gb":2.089,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/tree/main"},{"quant":"fp8","weights_gb":0.519,"kv_cache_gb_per_8k":0.1007,"min_serve_gb":1.62,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.294,"kv_cache_gb_per_8k":0.1007,"min_serve_gb":1.395,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.519,"kv_cache_gb_per_8k":0.1007,"min_serve_gb":1.62,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":426.77,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.49,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1240.82,"bw_limited_tok_s":744.49,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":250.54,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.49,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":557.14,"bw_limited_tok_s":334.29,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":804,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.49,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":6836.73,"bw_limited_tok_s":4102.04,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"gpqa_diamond","score_raw":29.8,"score_norm":6.76,"config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":49.6,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":35.4,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":27.9,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":12.6,"score_norm":0,"config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":5.1,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":34.4,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":49.6,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":15,"score_norm":0,"config":"qwen2.5-0.5b-instruct, table 10 (instruction-tuned models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":24.1,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multipl_e","score_raw":28.5,"score_norm":null,"config":"qwen2.5-0.5b-instruct, table 10","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen2.5-0.5B","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-2.5-1.5b","display_name":"Qwen2.5-1.5B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"1.5b","params_total_b":1.54,"params_active_b":1.54,"params_exact":1543714304,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3.1,"int8_gb":1.6,"int4_gb":0.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":4.2,"expected_int8_gb":6.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-1.5B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-09-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-1.5b","qwen2.5-1.5b-instruct"],"intelligence":{"index":8.56,"inputs":[{"benchmark":"aime","score_raw":0.4,"score_norm":25,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"aime'25, measured by the qwen3 team"},{"benchmark":"gpqa_diamond","score_raw":29.8,"score_norm":6.76,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"measured by the qwen3 team, non-thinking, qwen2.5-1.5b-instruct"},{"benchmark":"livebench","score_raw":18,"score_norm":0,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"livebench 2024-11-25, measured by the qwen3 team"}],"coverage":0.3572,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":3.08,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":4.315,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":3.087,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":4.322,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/tree/main"},{"quant":"fp8","weights_gb":1.632,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":2.867,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.924,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":2.159,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1.632,"kv_cache_gb_per_8k":0.2349,"min_serve_gb":2.867,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":191.52,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":1.54,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":394.81,"bw_limited_tok_s":236.88,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":96.14,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":1.54,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":177.27,"bw_limited_tok_s":106.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":566.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":1.54,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2175.32,"bw_limited_tok_s":1305.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":0.4,"score_norm":25,"config":"aime'25, measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":9,"score_norm":null,"config":"measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl","score_raw":47.8,"score_norm":null,"config":"bfcl v3, measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":29.8,"score_norm":6.76,"config":"measured by the qwen3 team, non-thinking, qwen2.5-1.5b-instruct","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":42.5,"score_norm":null,"config":"measured by the qwen3 team, strict prompt","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":18,"score_norm":0,"config":"livebench 2024-11-25, measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":5.3,"score_norm":null,"config":"measured by the qwen3 team, v5","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":55,"score_norm":null,"config":"measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":24.2,"score_norm":0,"config":"mcf, qwen2.5-1.5b-instruct as evaluated by the smollm2 team","trust":0,"source_type":"community","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_redux","score_raw":50.7,"score_norm":null,"config":"measured by the qwen3 team","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/collections/Qwen/qwen25","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-2.5-32b-instruct-code","display_name":"Qwen2.5 Coder 32B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"32b","params_total_b":32.76,"params_active_b":32.76,"params_exact":32763876352,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.5,"int8_gb":34.7,"int4_gb":19.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.31,"expected_int8_gb":10.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-Coder-32B-Instruct","openrouter_id":"qwen/qwen-2.5-coder-32b-instruct","is_moe":false,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-11-11","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-2.5-coder-32b-instruct","qwen2.5-coder-32b-instruct"],"intelligence":{"index":14.95,"inputs":[{"benchmark":"aime","score_raw":20,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/html/2409.12186v3","config":"aime24, qwen2.5-coder technical report table 20"},{"benchmark":"arena_elo","score_raw":1270.6,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=qwen2-5-coder-32b-instruct","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":3.5,"score_norm":2.21,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=qwen2-5-coder-32b-instruct","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"livebench","score_raw":45,"score_norm":11.11,"trust":2,"source_url":"https://arxiv.org/html/2406.19314v2","config":"global average, qwen2.5-coder-32b-instruct, table 2"},{"benchmark":"mmlu_pro","score_raw":63.5,"score_norm":39.17,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=qwen2-5-coder-32b-instruct","config":"artificial analysis harness, mmlu-pro"}],"coverage":0.6666,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.66,"completion_usd_per_m":1,"cache_read_usd_per_m":null,"context_length":32768,"source":"openrouter","source_model_id":"qwen/qwen-2.5-coder-32b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.745,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":65.52,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.943,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.528,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.952,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct/tree/main"},{"quant":"fp8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.656,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":22.804,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.79,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32.76,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":37.12,"bw_limited_tok_s":22.27,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.98,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.33,"bw_limited_tok_s":5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":57.81,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":102.26,"bw_limited_tok_s":61.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen-2.5-coder-32b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aider","score_raw":60.9,"score_norm":null,"config":"code editing, pass@1, qwen2.5-coder technical report table 19","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aider","score_raw":73.7,"score_norm":null,"config":"code repair","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-coder-family/","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":20,"score_norm":0,"config":"aime24, qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"amc23","score_raw":55,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1270.6,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1270.6,"score_norm":0,"config":"5432 votes, arena entry qwen2.5-coder-32b-instruct, leaderboard text-overall-style_control, rank 289, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"c_eval","score_raw":68.9,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"collegemath","score_raw":47.7,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gaokao2023en","score_raw":68.3,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.8,"score_norm":23.66,"config":"qwen2.5-coder technical report table 20, unqualified gpqa (subset not stated)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=qwen2-5-coder-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":93,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.5,"score_norm":2.21,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=qwen2-5-coder-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":79.9,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":45,"score_norm":11.11,"config":"global average, qwen2.5-coder-32b-instruct, table 2","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2406.19314v2","measured_date":"2025-04-18","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math","score_raw":76.4,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mceval","score_raw":65.9,"score_norm":null,"config":"40+ languages, multi-language","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-coder-family/","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mdeval","score_raw":75.2,"score_norm":null,"config":"multi-language code repair","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-coder-family/","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":77.6,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":62.3,"score_norm":37.17,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":63.5,"score_norm":39.17,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=qwen2-5-coder-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"olympiadbench","score_raw":42.5,"score_norm":null,"config":"qwen2.5-coder technical report table 20","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2409.12186v3","measured_date":"2024-11-12","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen-2.5-coder-32b-instruct","publisher":"OpenRouter (listed 2024-11-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-2.5-3b","display_name":"Qwen2.5-3B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"3b","params_total_b":3.09,"params_active_b":3.09,"params_exact":3085938688,"params_source":"hf_safetensors","vram_est":{"bf16_gb":6.2,"int8_gb":3.3,"int4_gb":1.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.34,"expected_int8_gb":4.4,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-3B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-09-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-3b","qwen2.5-3b-instruct"],"intelligence":{"index":4.84,"inputs":[{"benchmark":"gpqa_diamond","score_raw":30.3,"score_norm":7.46,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"5-shot"},{"benchmark":"livebench","score_raw":26.8,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"0831 edition"},{"benchmark":"mmlu_pro","score_raw":43.7,"score_norm":6.17,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"5-shot"}],"coverage":0.3969,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":6.18,"kv_cache_gb_per_8k":0.302,"min_serve_gb":7.482,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":6.172,"kv_cache_gb_per_8k":0.302,"min_serve_gb":7.474,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/tree/main"},{"quant":"fp8","weights_gb":3.275,"kv_cache_gb_per_8k":0.302,"min_serve_gb":4.577,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.854,"kv_cache_gb_per_8k":0.302,"min_serve_gb":3.156,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":3.275,"kv_cache_gb_per_8k":0.302,"min_serve_gb":4.577,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":105.59,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.09,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":196.76,"bw_limited_tok_s":118.06,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":50.34,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.09,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":88.35,"bw_limited_tok_s":53.01,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":394.12,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.09,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1084.14,"bw_limited_tok_s":650.49,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"gpqa_diamond","score_raw":30.3,"score_norm":7.46,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":74.4,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":26.8,"score_norm":0,"config":"0831 edition","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math","score_raw":65.9,"score_norm":null,"config":"4-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":43.7,"score_norm":6.17,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://arxiv.org/html/2412.15115","publisher":"arXiv"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-2.5-72b","display_name":"Qwen2.5-72B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"72b","params_total_b":72.71,"params_active_b":72.71,"params_exact":72706203648,"params_source":"hf_safetensors","vram_est":{"bf16_gb":145,"int8_gb":77.1,"int4_gb":43.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen2.5-72B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-09-18","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-72b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":145.42,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":155.375,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":145.413,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":155.368,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-72B/tree/main"},{"quant":"fp8","weights_gb":77.073,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":83.611,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":43.626,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":48.492,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":77.073,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":83.611,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.25,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":72.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.75,"bw_limited_tok_s":2.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":52.39,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":72.71,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":92.15,"bw_limited_tok_s":55.29,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc_challenge","score_raw":72.4,"score_norm":null,"config":"base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":86.3,"score_norm":null,"config":"3-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":45.9,"score_norm":29.44,"config":"5-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":91.5,"score_norm":null,"config":"4-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":87.6,"score_norm":null,"config":"base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":59.1,"score_norm":null,"config":"0-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":62.1,"score_norm":null,"config":"4-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":84.7,"score_norm":null,"config":"0-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":86.1,"score_norm":null,"config":"5-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":58.1,"score_norm":30.17,"config":"5-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":83.9,"score_norm":null,"config":"5-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multipl_e","score_raw":60.5,"score_norm":null,"config":"0-shot, base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":60.4,"score_norm":null,"config":"base model","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2412.15115","measured_date":"2024-12-19","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen2.5-llm/","publisher":"Qwen"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"qwen-2.5-72b-instruct","display_name":"Qwen2.5 72B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"72b","params_total_b":72.71,"params_active_b":72.71,"params_exact":72706203648,"params_source":"hf_safetensors","vram_est":{"bf16_gb":145,"int8_gb":77.1,"int4_gb":43.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.3,"expected_int8_gb":23.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-72B-Instruct","openrouter_id":"qwen/qwen-2.5-72b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-09-19","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-72b-instruct"],"intelligence":{"index":32.78,"inputs":[{"benchmark":"arena_elo","score_raw":1302.8,"score_norm":10.33,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":49,"score_norm":33.8,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"5-shot"},{"benchmark":"livebench","score_raw":52.3,"score_norm":27.33,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"0831 edition"},{"benchmark":"mmlu_pro","score_raw":71.1,"score_norm":51.83,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"5-shot"}],"coverage":0.4762,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.36,"completion_usd_per_m":0.4,"cache_read_usd_per_m":null,"context_length":32768,"source":"openrouter","source_model_id":"qwen/qwen-2.5-72b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.37,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":145.42,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":155.375,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":145.413,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":155.368,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/tree/main"},{"quant":"fp8","weights_gb":77.073,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":83.611,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":43.626,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":48.492,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":77.073,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":83.611,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.25,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":72.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.75,"bw_limited_tok_s":2.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":52.39,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":72.71,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":92.15,"bw_limited_tok_s":55.29,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.7,"score_norm":7.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen-2.5-72b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1302.8,"score_norm":10.33,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1302.9,"score_norm":10.37,"config":"39406 votes, arena entry qwen2.5-72b-instruct, leaderboard text-overall-style_control, rank 266, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":81.2,"score_norm":null,"config":null,"trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":49,"score_norm":33.8,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":86.6,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":52.3,"score_norm":27.33,"config":"0831 edition","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"math","score_raw":83.1,"score_norm":null,"config":"4-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":88.2,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":71.1,"score_norm":51.83,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen-2.5-72b-instruct","publisher":"OpenRouter (listed 2024-09-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-2.5-72b-instruct-vision","display_name":"Qwen2.5 VL 72B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"72b","params_total_b":73.41,"params_active_b":73.41,"params_exact":73410777344,"params_source":"hf_safetensors","vram_est":{"bf16_gb":147,"int8_gb":77.8,"int4_gb":44,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen2.5-VL-72B-Instruct","openrouter_id":"qwen/qwen2.5-vl-72b-instruct","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-01-26","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen2.5-vl-72b-instruct"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.8,"completion_usd_per_m":1,"cache_read_usd_per_m":0.4,"context_length":128000,"source":"openrouter","source_model_id":"qwen/qwen2.5-vl-72b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.8500000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":146.82,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":156.845,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":146.822,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":156.848,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/tree/main"},{"quant":"fp8","weights_gb":77.815,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":84.39,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":44.046,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":48.933,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":77.815,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":84.39,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.23,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":73.41,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.72,"bw_limited_tok_s":2.23,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":51.92,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":73.41,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":91.27,"bw_limited_tok_s":54.76,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"ai2d","score_raw":88.4,"score_norm":null,"config":"test","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1121.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1121.6,"score_norm":null,"config":"3768 votes, arena entry qwen2.5-vl-72b-instruct, leaderboard vision-overall-style_control, rank 111, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/vision","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"chartqa","score_raw":89.5,"score_norm":null,"config":"test","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"docvqa","score_raw":96.4,"score_norm":null,"config":"val","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"lvbench","score_raw":47.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mathvision","score_raw":38.1,"score_norm":null,"config":"full","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mathvista","score_raw":74.8,"score_norm":null,"config":"mini","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu","score_raw":70.2,"score_norm":null,"config":"val","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":51.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmstar","score_raw":70.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mvbench","score_raw":70.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ocrbench","score_raw":885,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":8.83,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"screenspot","score_raw":87.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"screenspot_pro","score_raw":43.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"video_mme","score_raw":73.3,"score_norm":null,"config":"without subtitles","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct","measured_date":"2025-01-01","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen2.5-vl-72b-instruct","publisher":"OpenRouter (listed 2025-02-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-2.5-7b-instruct","display_name":"Qwen2.5 7B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"7b","params_total_b":7.62,"params_active_b":7.62,"params_exact":7615616512,"params_source":"hf_safetensors","vram_est":{"bf16_gb":15.2,"int8_gb":8.1,"int4_gb":4.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.32,"expected_int8_gb":10.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen2.5-7B-Instruct","openrouter_id":"qwen/qwen-2.5-7b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2024-09-18","release_status":"released","knowledge_cutoff":"2024-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-2.5-7b","qwen2.5-7b","qwen2.5-7b-instruct"],"intelligence":{"index":14.58,"inputs":[{"benchmark":"gpqa_diamond","score_raw":36.4,"score_norm":16.06,"trust":1,"source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","config":null},{"benchmark":"livebench","score_raw":35.9,"score_norm":0,"trust":1,"source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","config":"livebench 0831 release"},{"benchmark":"mmlu_pro","score_raw":56.3,"score_norm":27.17,"trust":1,"source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","config":null}],"coverage":0.3969,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.2,"cache_read_usd_per_m":null,"context_length":32768,"source":"openrouter","source_model_id":"qwen/qwen-2.5-7b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":15.24,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":16.71,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":15.231,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":16.701,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/tree/main"},{"quant":"fp8","weights_gb":8.077,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":9.547,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.572,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":6.042,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.077,"kv_cache_gb_per_8k":0.4698,"min_serve_gb":9.547,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":45.69,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":7.62,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":79.79,"bw_limited_tok_s":47.87,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":21.04,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":7.62,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":35.83,"bw_limited_tok_s":21.5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":208.72,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":7.62,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":439.63,"bw_limited_tok_s":263.78,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arena_hard","score_raw":52,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":36.4,"score_norm":16.06,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":91.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":84.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":71.2,"score_norm":null,"config":"strict-prompt","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":35.9,"score_norm":0,"config":"livebench 0831 release","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":28.7,"score_norm":null,"config":"2305-2409 subset","trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":75.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":79.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":56.3,"score_norm":27.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":75.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multipl_e","score_raw":70.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://qwenlm.github.io/blog/qwen2.5-llm/","measured_date":"2024-09-19","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen-2.5-7b-instruct","publisher":"OpenRouter (listed 2024-10-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-2025-plus","display_name":"Qwen-Plus","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2025","size_class":"frontier-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen-plus","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-09-08","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-plus","qwen-plus-2025-01-25","qwen-plus-2025-07-28"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.26,"completion_usd_per_m":0.78,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen-plus-2025-07-28","as_of":"2026-10-06","blended_usd_per_m":0.39,"endpoint_expires_at":"2026-10-09"},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen-plus-2025-07-28","publisher":"OpenRouter (listed 2025-09-08)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-2025-plus-07-28-reasoning","display_name":"qwen-plus-2025-07-28-thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2025","size_class":"frontier-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen-plus-2025-07-28:thinking","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-09-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-28-plus-2025-07-reasoning","qwen-plus-2025-07-28-thinking"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen-plus-2025-07-28:thinking","publisher":"OpenRouter (listed 2025-09-08)"}],"last_verified":"2026-08-25","last_enriched_at":null},{"id":"qwen-3-0.6b","display_name":"Qwen3-0.6B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"0.6b","params_total_b":0.75,"params_active_b":0.75,"params_exact":751632384,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1.5,"int8_gb":0.8,"int4_gb":0.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.36,"expected_int8_gb":2.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-0.6B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2025-04-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-0.6b"],"intelligence":{"index":2.54,"inputs":[{"benchmark":"aime","score_raw":15.1,"score_norm":0,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"aime'25, thinking mode"},{"benchmark":"gpqa_diamond","score_raw":27.9,"score_norm":4.08,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"avg of 10 samples, thinking mode"},{"benchmark":"hle","score_raw":5.64,"score_norm":5.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-0.6b-instruct-reasoning","config":"aa eval, no tools, qwen3 0.6b (reasoning)"},{"benchmark":"livebench","score_raw":30.3,"score_norm":0,"trust":1,"source_url":"https://arxiv.org/abs/2505.09388","config":"livebench 2024-11-25, thinking mode"},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-0.6b-instruct-reasoning","config":"aa eval, qwen3 0.6b (reasoning), tau2-bench telecom"}],"coverage":0.5159,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":1.5,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":3.44,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1.503,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":3.443,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-0.6B/tree/main"},{"quant":"fp8","weights_gb":0.795,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":2.735,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.45,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":2.39,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.795,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":2.735,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":327.23,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.75,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":810.67,"bw_limited_tok_s":486.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":179.25,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.75,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":364,"bw_limited_tok_s":218.4,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":728.26,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.75,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":4466.67,"bw_limited_tok_s":2680,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":15.1,"score_norm":0,"config":"aime'25, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":8.5,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl","score_raw":46.4,"score_norm":null,"config":"bfcl v3, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":27.9,"score_norm":4.08,"config":"avg of 10 samples, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.64,"score_norm":5.35,"config":"aa eval, no tools, qwen3 0.6b (reasoning)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-0.6b-instruct-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":59.2,"score_norm":null,"config":"strict prompt, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":30.3,"score_norm":0,"config":"livebench 2024-11-25, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":12.3,"score_norm":null,"config":"thinking mode, v5","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":77.6,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_redux","score_raw":55.6,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/abs/2505.09388","measured_date":"2025-05-14","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"config":"aa eval, qwen3 0.6b (reasoning), tau2-bench telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-0.6b-instruct-reasoning","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://huggingface.co/Qwen/Qwen3-0.6B","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-3-1.7b","display_name":"Qwen3-1.7B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"1.7b","params_total_b":2.03,"params_active_b":2.03,"params_exact":2031739904,"params_source":"hf_safetensors","vram_est":{"bf16_gb":4.1,"int8_gb":2.2,"int4_gb":1.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":8.3,"expected_int8_gb":17.9,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-1.7B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":32768,"release_date":"2025-04-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-1.7b"],"intelligence":{"index":25.54,"inputs":[{"benchmark":"aime","score_raw":48.3,"score_norm":35.37,"trust":2,"source_url":"https://arxiv.org/html/2505.09388v1","config":"24, thinking mode"},{"benchmark":"gpqa_diamond","score_raw":40.1,"score_norm":21.27,"trust":2,"source_url":"https://arxiv.org/html/2505.09388v1","config":"thinking mode"},{"benchmark":"livebench","score_raw":51.1,"score_norm":24.67,"trust":2,"source_url":"https://arxiv.org/html/2505.09388v1","config":"2024.11.25, thinking mode"}],"coverage":0.3572,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":4.06,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":5.999,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":4.064,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":6.004,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-1.7B/tree/main"},{"quant":"fp8","weights_gb":2.152,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":4.092,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.218,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":3.157,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":2.152,"kv_cache_gb_per_8k":0.9395,"min_serve_gb":4.092,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":152.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":2.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":299.51,"bw_limited_tok_s":179.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":74.66,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":2.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":134.48,"bw_limited_tok_s":80.69,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":497.52,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":2.03,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1650.25,"bw_limited_tok_s":990.15,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":36.8,"score_norm":21,"config":"25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":48.3,"score_norm":35.37,"config":"24, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard","score_raw":43.1,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":56.6,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40.1,"score_norm":21.27,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":51.1,"score_norm":24.67,"config":"2024.11.25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":33.2,"score_norm":null,"config":"5, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":93.4,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_redux","score_raw":73.9,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2505.09388v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-3-14b-instruct","display_name":"Qwen3-14B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"14b","params_total_b":14.77,"params_active_b":14.77,"params_exact":14768307200,"params_source":"hf_safetensors","vram_est":{"bf16_gb":29.5,"int8_gb":15.7,"int4_gb":8.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.31,"expected_int8_gb":20.6,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-14B","openrouter_id":"qwen/qwen3-14b","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":["qwen-2.5-14b-instruct","qwen-3-14b-04-28"],"superseded_by":null,"aliases":["qwen-3-14b","qwen-3-14b-04-28","qwen3-14b","qwen3-14b-04-28"],"intelligence":{"index":29.33,"inputs":[{"benchmark":"aime","score_raw":58,"score_norm":47.5,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.58 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":60.4,"score_norm":49.86,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.604040404040404 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.53,"score_norm":3.72,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0453 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":34.5,"score_norm":19.33,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.345029239766082 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":5.3,"score_norm":0.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.053030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.12,"completion_usd_per_m":0.24,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-14b","as_of":"2026-10-06","blended_usd_per_m":0.15,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":29.54,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":32.359,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":29.537,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":32.356,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-14B/tree/main"},{"quant":"fp8","weights_gb":15.656,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.998,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.862,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.204,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":15.656,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":17.998,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":24.1,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":41.16,"bw_limited_tok_s":24.7,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":10.97,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":18.48,"bw_limited_tok_s":11.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":119.79,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":14.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":226.81,"bw_limited_tok_s":136.09,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.9,"score_norm":0.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-14b","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.86,"score_norm":1.86,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":13.8,"score_norm":13.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-14b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.2,"score_norm":8.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-14b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.44,"score_norm":10.44,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":58,"score_norm":47.5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.58 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":60.4,"score_norm":49.86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.604040404040404 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.53,"score_norm":3.72,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0453 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":40.54,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.405442176870748 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":52.28,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.522751322751323 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":31.6,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.315972222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":34.5,"score_norm":19.33,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.345029239766082 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.57,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0556701030927835 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.87,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.048689138576779 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":5.3,"score_norm":0.35,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.053030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-14b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://arxiv.org/html/2505.09388v1","publisher":"arXiv"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-235b-a22b-07-25","display_name":"Qwen3-235B-A22B-Instruct-2507","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.09,"params_active_b":22,"params_exact":235093634560,"params_source":"hf_safetensors","vram_est":{"bf16_gb":470,"int8_gb":249,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.83,"expected_int8_gb":207,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-235B-A22B-Instruct-2507","openrouter_id":"qwen/qwen3-235b-a22b-2507","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-07-21","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":["qwen-3-235b-a22b","qwen-3-235b-a22b-2507-instruct"],"superseded_by":null,"aliases":["qwen-3-235b-a22b-2507","qwen-3-235b-a22b-2507-instruct","qwen3-235b-a22b-07-25","qwen3-235b-a22b-2507","qwen3-235b-a22b-instruct-2507"],"intelligence":{"index":55.88,"inputs":[{"benchmark":"aime","score_raw":91,"score_norm":88.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.91 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":1.25,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0044, eval=arc-agi-2 semi-private (v2_semi_private), org=alibaba, run: 25/07, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1422.4,"score_norm":63.63,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":78.99,"score_norm":76.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.78989898989899 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":15.89,"score_norm":20.43,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.158943466172382 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":75.4,"score_norm":78.67,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","config":"global average. label on the official model card is exactly 'livebench 20241125'. [raw: livebench 20241125], livebench 2024-11-25 release"},{"benchmark":"mmlu_pro","score_raw":83,"score_norm":71.67,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","config":"no tools. [raw: mmlu-pro], official model card benchmark table. non-thinking mode"},{"benchmark":"tau_bench","score_raw":53.22,"score_norm":44.29,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.532163742690059 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","config":"label confirmed by page title of https://artificialanalysis.ai/evaluations/terminalbench-hard = 'terminal [raw: terminal-bench hard], measured independently by artificial analysis (field 'terminalbenchhard', terminal-bench hard"}],"coverage":0.8412,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.09,"completion_usd_per_m":0.55,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-235b-a22b-2507","as_of":"2026-10-06","blended_usd_per_m":0.20500000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":470.18,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.266,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":470.192,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.279,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/tree/main"},{"quant":"fp8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.054,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":149.684,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":3.83,"score_norm":3.83,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12,"score_norm":12,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.89,"score_norm":19.89,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.706666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91,"score_norm":88.75,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.91 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":71.67,"score_norm":64.59,"config":"aime 2025, measured independently by artificial analysis, no tools. raw value on page: 0.716666666666667 (= 43/60). [raw: aime 2025]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":11,"score_norm":null,"config":"cost_per_task_usd=0.0025, eval=arc-agi-1 semi-private (v1_semi_private), org=alibaba, run: 25/07, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":1.25,"score_norm":0,"config":"cost_per_task_usd=0.0044, eval=arc-agi-2 semi-private (v2_semi_private), org=alibaba, run: 25/07, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":1.25,"score_norm":0,"config":"arc prize official semi-private eval (datasetid 'v2_semi_private', costpertask $0.0044. this json is the data file that backs https://arcpri [raw: arc-agi-2], modelid 'qwen3-235b-a22b-instruct-2507'), raw score 0.0125","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/media/data/evaluations.json","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1422.4,"score_norm":63.63,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1422.8,"score_norm":63.81,"config":"97398 votes, arena entry qwen3-235b-a22b-instruct-2507, leaderboard text-overall-style_control, rank 109, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: qwen3 235b a22b 2507 instruct, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1009,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1044,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1078,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":964,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":972,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1064,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":2.22,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.022180000000000005 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.99,"score_norm":76.04,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.78989898989899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.5,"score_norm":73.94,"config":"column 'qwen3-235b-a22b-instruct-2507'. printed label is bare 'gpqa', official model card benchmark table, qwen's post-training eval suite uses the gpqa-diamond subset (qwen3 tech report arxiv:2505.09 [raw: gpqa]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":15.89,"score_norm":20.43,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.158943466172382 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.12,"score_norm":13.41,"config":"hle as run by artificial analysis (component of aa intelligence index), no tools. raw value on page: 0.111214087117702. only independent hle measurement found for this exact non-thinking variant. [raw: humanity's last exam]","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":51.22,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.512244897959184 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":75.4,"score_norm":78.67,"config":"global average. label on the official model card is exactly 'livebench 20241125'. [raw: livebench 20241125], livebench 2024-11-25 release","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":78.84,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.788359788359788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83,"score_norm":71.67,"config":"no tools. [raw: mmlu-pro], official model card benchmark table. non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":42.36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.423611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":53.22,"score_norm":44.29,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.532163742690059 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":32.5,"score_norm":16.67,"config":"tau1-airline 44.0. telecom chosen because [raw: tau2-telecom], tau1-retail 71.3, tau2-airline 50.0, tau2-bench (τ2-bench), telecom domain only - no average is published. official model card also reports tau2-retail 74.6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":7.84,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0783505154639175 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":11.99,"score_norm":8.22,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.119850187265918 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: qwen3 235b a22b 2507 instruct, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"config":"label confirmed by page title of https://artificialanalysis.ai/evaluations/terminalbench-hard = 'terminal [raw: terminal-bench hard], measured independently by artificial analysis (field 'terminalbenchhard', terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-2507","publisher":"OpenRouter (listed 2025-07-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-235b-a22b-2507-reasoning","display_name":"Qwen3-235B-A22B-Thinking-2507","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.09,"params_active_b":22,"params_exact":235093634560,"params_source":"hf_safetensors","vram_est":{"bf16_gb":470,"int8_gb":249,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.24,"expected_int8_gb":308,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-235B-A22B-Thinking-2507","openrouter_id":"qwen/qwen3-235b-a22b-thinking-2507","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07-25","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-235b-a22b-thinking-2507"],"intelligence":{"index":68.44,"inputs":[{"benchmark":"aime","score_raw":92.3,"score_norm":90.38,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"81, 920 max output tokens. [raw: aime25], aime 2025. official model card \"reasoning\" section, no tools"},{"benchmark":"arena_elo","score_raw":1400,"score_norm":53.65,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":81.1,"score_norm":79.01,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"81, 920 max o [raw: gpqa], no tools, official model card table, qwen's gpqa rows are conventionally the diamond subset). thinking mode, row labeled \"gpqa\" (no subset label printed, temp 0.6 / top-p 0.95 / top-k 20"},{"benchmark":"hle","score_raw":18.2,"score_norm":23.82,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"card footnote: \"scores marked with # refer to models that are not multi-modal and were evaluated only on the text-only subset\". text-only, no tools. [raw: hle], printed as \"18.2#\""},{"benchmark":"livebench","score_raw":78.4,"score_norm":85.33,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"global average as printed on the official model card. no tools. [raw: livebench 20241125], livebench 2024-11-25 release"},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"\"knowledge\" section. thinking mode, no tools. [raw: mmlu-pro], official model card table"},{"benchmark":"tau_bench","score_raw":71.9,"score_norm":69.2,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"ta [raw: tau2-retail], tau-2 (τ2-bench) retail domain only — no cross-domain average is published anywhere for this model. other published domains on the same card: tau2-airline 58.0, tau1-retail 67.8, tau2-telecom 45.6"}],"coverage":0.7143,"subset_bias":{"offset":4.18,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.23,"completion_usd_per_m":2.3,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-235b-a22b-thinking-2507","as_of":"2026-10-06","blended_usd_per_m":0.7474999999999999,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":470.18,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.266,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":470.192,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.279,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507/tree/main"},{"quant":"fp8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.054,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":149.684,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.3,"score_norm":1.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":22.1,"score_norm":22.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12.7,"score_norm":12.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92.3,"score_norm":90.38,"config":"81, 920 max output tokens. [raw: aime25], aime 2025. official model card \"reasoning\" section, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1400,"score_norm":53.65,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1398.8,"score_norm":53.12,"config":"9014 votes, arena entry qwen3-235b-a22b-thinking-2507, leaderboard text-overall-style_control, rank 145, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1013,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1039,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":957,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":970,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":952,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1059,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.1,"score_norm":79.01,"config":"81, 920 max o [raw: gpqa], no tools, official model card table, qwen's gpqa rows are conventionally the diamond subset). thinking mode, row labeled \"gpqa\" (no subset label printed, temp 0.6 / top-p 0.95 / top-k 20","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.2,"score_norm":23.82,"config":"card footnote: \"scores marked with # refer to models that are not multi-modal and were evaluated only on the text-only subset\". text-only, no tools. [raw: hle], printed as \"18.2#\"","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":78.4,"score_norm":85.33,"config":"global average as printed on the official model card. no tools. [raw: livebench 20241125], livebench 2024-11-25 release","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"config":"\"knowledge\" section. thinking mode, no tools. [raw: mmlu-pro], official model card table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":71.9,"score_norm":69.2,"config":"ta [raw: tau2-retail], tau-2 (τ2-bench) retail domain only — no cross-domain average is published anywhere for this model. other published domains on the same card: tau2-airline 58.0, tau1-retail 67.8, tau2-telecom 45.6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b-thinking-2507","publisher":"OpenRouter (listed 2025-07-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-235b-a22b-instruct-vision","display_name":"Qwen3 VL 235B A22B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.67,"params_active_b":22,"params_exact":235670022896,"params_source":"hf_safetensors","vram_est":{"bf16_gb":471,"int8_gb":250,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.73,"expected_int8_gb":183,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-235B-A22B-Instruct","openrouter_id":"qwen/qwen3-vl-235b-a22b-instruct","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-09-23","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":["qwen-3-235b-a22b-reasoning-vision"],"superseded_by":null,"aliases":["qwen3-vl-235b-a22b-instruct"],"intelligence":{"index":51.72,"inputs":[{"benchmark":"aime","score_raw":70.67,"score_norm":63.34,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.706666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1413.4,"score_norm":59.62,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":71.21,"score_norm":65.08,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.712121212121212 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.63,"score_norm":6.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0662650602409639 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":74.8,"score_norm":77.33,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"column \"qwen3-vl 235b-a22b instruct\". [raw: livebench 2024-11-25], global/overall average, livebench 2024-11-25 release, qwen3-vl technical report table 2"},{"benchmark":"mmlu_pro","score_raw":81.8,"score_norm":69.67,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"column \"qwen3-vl 235b-a22b instruct\" (non-thinking). vendor self-reported, no tools. [raw: mmlu-pro], pure-text evaluation, qwen3-vl technical report (arxiv:2511.21631) table 2"},{"benchmark":"tau_bench","score_raw":35.09,"score_norm":20.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.350877192982456 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7619,"subset_bias":{"offset":2.81,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.21,"completion_usd_per_m":1.9,"cache_read_usd_per_m":0.1,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-vl-235b-a22b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.6325,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":471.34,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":496.484,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":471.34,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":496.484,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct/tree/main"},{"quant":"fp8","weights_gb":249.81,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.878,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.402,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":150.049,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.81,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.878,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.9,"score_norm":9.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.37,"score_norm":14.37,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":32,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.32 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":70.67,"score_norm":63.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.706666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":74.7,"score_norm":68.38,"config":"aime 2025, column \"qwen3-vl 235b-a22b instruct\" (non-thinking). no tools stated. [raw: aime-25], qwen3-vl technical report table 2, reasoning section","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1413.4,"score_norm":59.62,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1414.4,"score_norm":60.07,"config":"11556 votes, arena entry qwen3-vl-235b-a22b-instruct, leaderboard text-overall-style_control, rank 124, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1215.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1215.1,"score_norm":null,"config":"12138 votes, arena entry qwen3-vl-235b-a22b-instruct, leaderboard vision-overall-style_control, rank 64, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.21,"score_norm":65.08,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.712121212121212 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.3,"score_norm":69.44,"config":"column \"qwen3-vl 235b-a22b instruct\". row labeled only \"gpqa\" (qwen convention = gpqa-diamond, knowledge section, paper does not state the subset explicitly). no too [raw: gpqa], qwen3-vl technical report table 2","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.63,"score_norm":6.81,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0662650602409639 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":42.65,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.426530612244898 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":74.8,"score_norm":77.33,"config":"column \"qwen3-vl 235b-a22b instruct\". [raw: livebench 2024-11-25], global/overall average, livebench 2024-11-25 release, qwen3-vl technical report table 2","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":59.37,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.593650793650794 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.8,"score_norm":69.67,"config":"column \"qwen3-vl 235b-a22b instruct\" (non-thinking). vendor self-reported, no tools. [raw: mmlu-pro], pure-text evaluation, qwen3-vl technical report (arxiv:2511.21631) table 2","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":67.57,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.675722543352601 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":35.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.358796296296296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":35.09,"score_norm":20.12,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.350877192982456 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":40.9,"score_norm":27.87,"config":"\"auto-routing\" row (model-level default routing). domain/version not stated by openrouter. per-provider spread on the same page: 38.8-41.0%. [raw: tau-bench], openrouter \"exacto benchmarks\" per-provider table","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-235b-a22b-instruct","publisher":"OpenRouter (listed 2025-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-2507-instruct","display_name":"Qwen3 30B A3B Instruct 2507","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":30.53,"params_active_b":3,"params_exact":30532122624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":61.1,"int8_gb":32.4,"int4_gb":18.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":6.64,"expected_int8_gb":215,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-30B-A3B-Instruct-2507","openrouter_id":"qwen/qwen3-30b-a3b-instruct-2507","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-07-29","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":["qwen-3-30b-a3b-instruct"],"superseded_by":null,"aliases":["qwen3-30b-a3b-instruct-2507"],"intelligence":{"index":57.1,"inputs":[{"benchmark":"aime","score_raw":61.3,"score_norm":51.63,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","config":"aime 2025, no tools [raw: aime25], non-thinking mode, official qwen model card 'reasoning' section"},{"benchmark":"arena_elo","score_raw":1382.7,"score_norm":45.94,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":70.4,"score_norm":63.94,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","config":"no tools [raw: gpqa], non-thinking, official qwen model card 'knowledge' section, row is labeled only 'gpqa' (card does not print the word 'diamond')"},{"benchmark":"livebench","score_raw":69,"score_norm":64.44,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","config":"global average, livebench 2024-11-25 release, non-thinking [raw: livebench 20241125], official qwen model card"},{"benchmark":"mmlu_pro","score_raw":78.4,"score_norm":64,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","config":"no tools [raw: mmlu-pro], non-thinking mode, official qwen model card benchmark table"},{"benchmark":"tau_bench","score_raw":39.8,"score_norm":26.4,"trust":2,"source_url":"https://openrouter.ai/qwen/qwen3-30b-a3b-instruct-2507","config":"auto-routing row, coreweave [raw: tau-bench], openrouter 'exacto benchmarks' per-endpoint table, single aggregate number (tau version/domain mix not stated on page). per-provider values on same table: siliconflow 39.1"}],"coverage":0.6032,"subset_bias":{"offset":9.59,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.3,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-30b-a3b-instruct-2507","as_of":"2026-10-06","blended_usd_per_m":0.15000000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":61.06,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.918,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":61.067,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.926,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507/tree/main"},{"quant":"fp8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.318,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.123,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.5,"score_norm":7.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-instruct-2507","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":61.3,"score_norm":51.63,"config":"aime 2025, no tools [raw: aime25], non-thinking mode, official qwen model card 'reasoning' section","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1382.7,"score_norm":45.94,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1383.3,"score_norm":46.21,"config":"23828 votes, arena entry qwen3-30b-a3b-instruct-2507, leaderboard text-overall-style_control, rank 166, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":70.4,"score_norm":63.94,"config":"no tools [raw: gpqa], non-thinking, official qwen model card 'knowledge' section, row is labeled only 'gpqa' (card does not print the word 'diamond')","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":69,"score_norm":64.44,"config":"global average, livebench 2024-11-25 release, non-thinking [raw: livebench 20241125], official qwen model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":78.4,"score_norm":64,"config":"no tools [raw: mmlu-pro], non-thinking mode, official qwen model card benchmark table","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":39.8,"score_norm":26.4,"config":"auto-routing row, coreweave [raw: tau-bench], openrouter 'exacto benchmarks' per-endpoint table, single aggregate number (tau version/domain mix not stated on page). per-provider values on same table: siliconflow 39.1","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3-30b-a3b-instruct-2507","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-instruct-2507","publisher":"OpenRouter (listed 2025-07-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-2507-reasoning","display_name":"Qwen3 30B A3B Thinking 2507","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":30.53,"params_active_b":3,"params_exact":30532122624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":61.1,"int8_gb":32.4,"int4_gb":18.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":10.7,"expected_int8_gb":346,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-30B-A3B-Thinking-2507","openrouter_id":"qwen/qwen3-30b-a3b-thinking-2507","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":81920,"release_date":"2025-08-28","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-30b-a3b-thinking-2507"],"intelligence":{"index":71.75,"inputs":[{"benchmark":"aime","score_raw":85,"score_norm":81.25,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","config":"aime25, thinking mode"},{"benchmark":"gpqa_diamond","score_raw":73.4,"score_norm":68.17,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","config":"thinking mode"},{"benchmark":"livebench","score_raw":76.8,"score_norm":81.78,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","config":"livebench 20241125, thinking mode"},{"benchmark":"mmlu_pro","score_raw":80.9,"score_norm":68.17,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","config":"thinking mode"},{"benchmark":"tau_bench","score_raw":58.8,"score_norm":51.73,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","config":"tau2-retail, with tools"}],"coverage":0.5239,"subset_bias":{"offset":10.74,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":2.4,"cache_read_usd_per_m":null,"context_length":81920,"source":"openrouter","source_model_id":"qwen/qwen3-30b-a3b-thinking-2507","as_of":"2026-10-06","blended_usd_per_m":0.75,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":61.06,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.918,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":61.067,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.926,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507/tree/main"},{"quant":"fp8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.318,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.123,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.9,"score_norm":0.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":12.1,"score_norm":12.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.8,"score_norm":9.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":85,"score_norm":81.25,"config":"aime25, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":56,"score_norm":null,"config":"win rate judged by gpt-4.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":72.4,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":933,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":937,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.4,"score_norm":68.17,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":71.4,"score_norm":null,"config":"25, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":88.9,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":76.8,"score_norm":81.78,"config":"livebench 20241125, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":"2024-11-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":66,"score_norm":null,"config":"thinking mode, v6 (25.02-25.05)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.9,"score_norm":68.17,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":91.4,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multiif","score_raw":76.4,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ojbench","score_raw":25.1,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"polymath","score_raw":52.6,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":56.8,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":58.8,"score_norm":51.73,"config":"tau2-retail, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"writingbench","score_raw":85,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b-thinking-2507","publisher":"OpenRouter (listed 2025-08-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-instruct-code","display_name":"Qwen3-Coder-30B-A3B-Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":30.53,"params_active_b":3,"params_exact":30532122624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":61.1,"int8_gb":32.4,"int4_gb":18.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.89,"expected_int8_gb":61.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-Coder-30B-A3B-Instruct","openrouter_id":"qwen/qwen3-coder-30b-a3b-instruct","is_moe":true,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-coder-30b-a3b-instruct"],"intelligence":{"index":19.43,"inputs":[{"benchmark":"aime","score_raw":29,"score_norm":11.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.29 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":51.62,"score_norm":37.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.516161616161616 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.85,"score_norm":2.72,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0384615384615385 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":34.5,"score_norm":19.33,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.345029239766082 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.07,"completion_usd_per_m":0.28,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-coder-30b-a3b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.12250000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":61.06,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.918,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":61.067,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.926,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/tree/main"},{"quant":"fp8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.318,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.123,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.6,"score_norm":9.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.63,"score_norm":13.63,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":31.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.316666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":29,"score_norm":11.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.29 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1091,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1053,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1094,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":51.62,"score_norm":37.49,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.516161616161616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.85,"score_norm":2.72,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0384615384615385 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":32.65,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.326530612244898 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":40.32,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.403174603174603 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":27.78,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":34.5,"score_norm":19.33,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.345029239766082 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-30b-a3b-instruct","publisher":"OpenRouter (listed 2025-07-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-instruct-vision","display_name":"Qwen3 VL 30B A3B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":31.07,"params_active_b":3,"params_exact":31070754032,"params_source":"hf_safetensors","vram_est":{"bf16_gb":62.1,"int8_gb":32.9,"int4_gb":18.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.44,"expected_int8_gb":113,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-30B-A3B-Instruct","openrouter_id":"qwen/qwen3-vl-30b-a3b-instruct","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-10-06","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-vl-30b-a3b-instruct"],"intelligence":{"index":35.77,"inputs":[{"benchmark":"aime","score_raw":72.33,"score_norm":65.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":69.49,"score_norm":62.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.694949494949495 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.3,"score_norm":6.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0630213160333642 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.6,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-vl-30b-a3b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.26249999999999996,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":62.14,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":66.052,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":62.142,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":66.054,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct/tree/main"},{"quant":"fp8","weights_gb":32.934,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":35.386,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.642,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.447,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.934,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":35.386,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.9,"score_norm":7.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.9,"score_norm":9.9,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":27.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.273333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":72.33,"score_norm":65.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":69.49,"score_norm":62.66,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.694949494949495 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.3,"score_norm":6.32,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0630213160333642 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":33.13,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.331292517006803 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":47.62,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.476190476190476 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":62.14,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.621387283236994 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":30.79,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.30787037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-30b-a3b-instruct","publisher":"OpenRouter (listed 2025-10-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-reasoning-vision","display_name":"Qwen3 VL 30B A3B Thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":31.07,"params_active_b":3,"params_exact":31070754032,"params_source":"hf_safetensors","vram_est":{"bf16_gb":62.1,"int8_gb":32.9,"int4_gb":18.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":9.85,"expected_int8_gb":324,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-30B-A3B-Thinking","openrouter_id":"qwen/qwen3-vl-30b-a3b-thinking","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-10-06","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-vl-30b-a3b-thinking"],"intelligence":{"index":69.92,"inputs":[{"benchmark":"aime","score_raw":83.1,"score_norm":78.88,"trust":2,"source_url":"https://arxiv.org/pdf/2511.21631","config":"aime-25, thinking mode"},{"benchmark":"gpqa_diamond","score_raw":74.4,"score_norm":69.58,"trust":2,"source_url":"https://arxiv.org/pdf/2511.21631","config":"thinking mode"},{"benchmark":"livebench","score_raw":72.1,"score_norm":71.33,"trust":2,"source_url":"https://arxiv.org/pdf/2511.21631","config":"livebench 2024-11-25, thinking mode"},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"trust":2,"source_url":"https://arxiv.org/pdf/2511.21631","config":"thinking mode"},{"benchmark":"tau_bench","score_raw":64,"score_norm":58.67,"trust":2,"source_url":"https://arxiv.org/pdf/2511.21631","config":"tau2-retail, with tools"}],"coverage":0.5239,"subset_bias":{"offset":10.74,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":2.4,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-vl-30b-a3b-thinking","as_of":"2026-10-06","blended_usd_per_m":0.75,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":62.14,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":66.052,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":62.142,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":66.054,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Thinking/tree/main"},{"quant":"fp8","weights_gb":32.934,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":35.386,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.642,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.447,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.934,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":35.386,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.5,"score_norm":9.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-30b-a3b-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":83.1,"score_norm":78.88,"config":"aime-25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":56.7,"score_norm":null,"config":"winrate","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":68.6,"score_norm":null,"config":"with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.4,"score_norm":69.58,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":67.6,"score_norm":null,"config":"25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":81.7,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":72.1,"score_norm":71.33,"config":"livebench 2024-11-25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":"2024-11-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":64.2,"score_norm":null,"config":"thinking mode, v6","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":90.9,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multiif","score_raw":73,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ojbench","score_raw":23.4,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"polymath","score_raw":51.7,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":56.4,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":64,"score_norm":58.67,"config":"tau2-retail, with tools","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"writingbench","score_raw":85.2,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2511.21631","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-30b-a3b-thinking","publisher":"OpenRouter (listed 2025-10-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-32b-instruct","display_name":"Qwen3-32B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"32b","params_total_b":32.76,"params_active_b":32.76,"params_exact":32762123264,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.5,"int8_gb":34.7,"int4_gb":19.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.72,"expected_int8_gb":25.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-32B","openrouter_id":"qwen/qwen3-32b","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":["qwen-2.5-32b-instruct","qwen-3-32b-04-28"],"superseded_by":null,"aliases":["qwen-3-32b","qwen-3-32b-04-28","qwen3-32b","qwen3-32b-04-28"],"intelligence":{"index":35.31,"inputs":[{"benchmark":"aime","score_raw":73,"score_norm":66.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1346.9,"score_norm":29.98,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":66.77,"score_norm":58.83,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.667676767676768 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":7.41,"score_norm":7.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0741 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":29.82,"score_norm":13.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.298245614035088 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":5.24,"score_norm":0.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.052434456928839 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-2.92,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.08,"completion_usd_per_m":0.28,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-32b","as_of":"2026-10-06","blended_usd_per_m":0.13,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":65.52,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.943,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.524,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.948,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-32B/tree/main"},{"quant":"fp8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.656,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":22.804,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.79,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":32.76,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":37.12,"bw_limited_tok_s":22.27,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.98,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.33,"bw_limited_tok_s":5,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":57.81,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":32.76,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":102.26,"bw_limited_tok_s":61.36,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.9,"score_norm":0.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-32b","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.79,"score_norm":1.79,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":15.3,"score_norm":15.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-32b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.6,"score_norm":8.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-32b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.45,"score_norm":11.45,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":73,"score_norm":66.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1346.9,"score_norm":29.98,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1347,"score_norm":30.03,"config":"3926 votes, arena entry qwen3-32b, leaderboard text-overall-style_control, rank 205, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.77,"score_norm":58.83,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.667676767676768 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.41,"score_norm":7.96,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":36.33,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.363265306122449 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":54.6,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.546031746031746 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":35.42,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.354166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":29.82,"score_norm":13.09,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.298245614035088 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.36,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0536082474226804 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":5.24,"score_norm":0.28,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.052434456928839 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-32b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://arxiv.org/html/2505.09388v1","publisher":"arXiv"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-32b-instruct-vision","display_name":"Qwen3 VL 32B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"32b","params_total_b":33.36,"params_active_b":33.36,"params_exact":33357390064,"params_source":"hf_safetensors","vram_est":{"bf16_gb":66.7,"int8_gb":35.4,"int4_gb":20,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.71,"expected_int8_gb":25.3,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-32B-Instruct","openrouter_id":"qwen/qwen3-vl-32b-instruct","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-10-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-vl-32b-instruct"],"intelligence":{"index":35.46,"inputs":[{"benchmark":"aime","score_raw":68.33,"score_norm":60.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":67.07,"score_norm":59.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.670707070707071 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.81,"score_norm":7.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0681186283595922 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":29.24,"score_norm":12.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.292397660818713 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.104,"completion_usd_per_m":0.416,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-vl-32b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.182,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":66.72,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":72.203,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":66.715,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":72.198,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct/tree/main"},{"quant":"fp8","weights_gb":35.362,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":39.278,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":20.016,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":23.164,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":35.362,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":39.278,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":21.4,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":33.36,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":36.45,"bw_limited_tok_s":21.87,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":4.89,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":33.36,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":8.18,"bw_limited_tok_s":4.91,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":56.83,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":33.36,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":100.42,"bw_limited_tok_s":60.25,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.9,"score_norm":11.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.99,"score_norm":10.99,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":34.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.343333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":68.33,"score_norm":60.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.07,"score_norm":59.25,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.670707070707071 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.81,"score_norm":7.07,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0681186283595922 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":39.18,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.391836734693878 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":51.43,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.514285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":64.28,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.642774566473988 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":30.09,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.300925925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":30.09,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: qwen3 vl 32b instruct, scicode, source value 0.300925925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":29.24,"score_norm":12.32,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.292397660818713 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-32b-instruct","publisher":"OpenRouter (listed 2025-10-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-480b-a35b-07-25-code","display_name":"Qwen3 Coder 480B A35B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"480b-a35b","params_total_b":480.15,"params_active_b":35,"params_exact":480154875392,"params_source":"hf_safetensors","vram_est":{"bf16_gb":960,"int8_gb":509,"int4_gb":288,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.25,"expected_int8_gb":127,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-Coder-480B-A35B-Instruct","openrouter_id":"qwen/qwen3-coder","is_moe":true,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-07-22","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-3-480b-a35b-instruct-code","qwen3-coder","qwen3-coder-480b-a35b-07-25","qwen3-coder-480b-a35b-instruct"],"intelligence":{"index":39.54,"inputs":[{"benchmark":"aime","score_raw":39.33,"score_norm":24.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.393333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1387.3,"score_norm":47.99,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":61.82,"score_norm":51.86,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.618181818181818 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.54,"score_norm":3.74,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0454124189063948 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":67,"score_norm":63.33,"trust":2,"source_url":"https://arxiv.org/html/2603.16790v1","config":"incoder-32b technical report table 3 (agentic coding + general tool use comparison table) [raw: swe-bench verified], third-party eval"},{"benchmark":"tau_bench","score_raw":43.57,"score_norm":31.43,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.435672514619883 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":23.9,"score_norm":22.24,"trust":3,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"23.9% +/- 2.8 [raw: terminal-bench 2.0], terminal-bench 2.0, terminus 2 reference agent (official harness)"}],"coverage":0.6826,"subset_bias":{"offset":-1.67,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1,"cache_read_usd_per_m":0.1,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-coder","as_of":"2026-10-06","blended_usd_per_m":0.475,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":960.3,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":1010.395,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":960.314,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":1010.41,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct/tree/main"},{"quant":"fp8","weights_gb":508.959,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":536.487,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":288.09,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":304.575,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":508.959,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":536.487,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.9,"score_norm":11.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":18.18,"score_norm":18.18,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":45.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.453333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":39.33,"score_norm":24.16,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.393333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1387.3,"score_norm":47.99,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1387.8,"score_norm":48.21,"config":"25788 votes, arena entry qwen3-coder-480b-a35b-instruct, leaderboard text-overall-style_control, rank 161, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1274.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1273.3,"score_norm":null,"config":"12291 votes, arena entry qwen3-coder-480b-a35b-instruct, code/overall, leaderboard webdev-overall-raw, rank 100, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1091,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1112,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1119,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1165,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.82,"score_norm":51.86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.618181818181818 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":62.1,"score_norm":52.25,"config":"62.1% +/- 3.0pp [raw: gpqa diamond], model-level default routing, openrouter gpqa diamond leaderboard","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/benchmarks/gpqa-diamond","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.54,"score_norm":3.74,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0454124189063948 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":40.48,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.404761904761905 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":58.52,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.585185185185185 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":35.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.358796296296296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":38.7,"score_norm":56.17,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":67,"score_norm":63.33,"config":"incoder-32b technical report table 3 (agentic coding + general tool use comparison table) [raw: swe-bench verified], third-party eval","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2603.16790v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":43.57,"score_norm":31.43,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.435672514619883 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":77.5,"score_norm":76.67,"config":"same table also reports airline 60.0 and telecom 65.8. incoder-32b technical report table 3 [raw: tau2-bench (retail)], tau2-bench retail domain only (no published average found)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2603.16790v1","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":23.9,"score_norm":22.24,"config":"23.9% +/- 2.8 [raw: terminal-bench 2.0], terminal-bench 2.0, terminus 2 reference agent (official harness)","trust":3,"source_type":"leaderboard","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":18.94,"score_norm":16.4,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.189393939393939 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-480b-a35b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder","publisher":"OpenRouter (listed 2025-07-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-4b","display_name":"Qwen3-4B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"4b","params_total_b":4.02,"params_active_b":4.02,"params_exact":4022468096,"params_source":"hf_safetensors","vram_est":{"bf16_gb":8,"int8_gb":4.3,"int4_gb":2.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.32,"expected_int8_gb":14.2,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-4B","openrouter_id":"qwen/qwen3-4b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-04-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-4b"],"intelligence":{"index":20.01,"inputs":[{"benchmark":"aime","score_raw":19.1,"score_norm":0,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","config":"aime25, non-thinking mode"},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","config":"non-thinking mode"},{"benchmark":"livebench","score_raw":48.4,"score_norm":18.67,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","config":"20241125 release, non-thinking mode"},{"benchmark":"mmlu_pro","score_raw":58,"score_norm":30,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","config":"non-thinking mode"}],"coverage":0.4762,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":8.04,"kv_cache_gb_per_8k":1.208,"min_serve_gb":10.248,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":8.045,"kv_cache_gb_per_8k":1.208,"min_serve_gb":10.253,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-4B/tree/main"},{"quant":"fp8","weights_gb":4.261,"kv_cache_gb_per_8k":1.208,"min_serve_gb":6.469,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":2.412,"kv_cache_gb_per_8k":1.208,"min_serve_gb":4.62,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":4.261,"kv_cache_gb_per_8k":1.208,"min_serve_gb":6.469,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":83.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":4.02,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":151.24,"bw_limited_tok_s":90.75,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":39.15,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":4.02,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":67.91,"bw_limited_tok_s":40.75,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":333.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":4.02,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":833.33,"bw_limited_tok_s":500,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":19.1,"score_norm":0,"config":"aime25, non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":9.5,"score_norm":null,"config":"non-thinking mode, win rate judged by gpt-4.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":57.6,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":81.2,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":48.4,"score_norm":18.67,"config":"20241125 release, non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":26.4,"score_norm":null,"config":"non-thinking mode, v6 (25.02-25.05)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":58,"score_norm":30,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":77.3,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multiif","score_raw":61.3,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":32,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"writingbench","score_raw":68.5,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"zebralogic","score_raw":35.2,"score_norm":null,"config":"non-thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"qwen-3-8b-instruct","display_name":"Qwen3-8B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"8b","params_total_b":8.19,"params_active_b":8.19,"params_exact":8190735360,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.4,"int8_gb":8.7,"int4_gb":4.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.62,"expected_int8_gb":14,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-8B","openrouter_id":"qwen/qwen3-8b","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-03-31","status":"current","supersedes":["qwen-3-8b-04-28"],"superseded_by":null,"aliases":["qwen-3-8b","qwen-3-8b-04-28","qwen3-8b","qwen3-8b-04-28"],"intelligence":{"index":19.82,"inputs":[{"benchmark":"aime","score_raw":24.33,"score_norm":5.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.243333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":58.89,"score_norm":47.73,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.588888888888889 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.89,"score_norm":2.78,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0389 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":27.78,"score_norm":10.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.117,"completion_usd_per_m":0.455,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-8b","as_of":"2026-10-06","blended_usd_per_m":0.2015,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":16.38,"kv_cache_gb_per_8k":1.208,"min_serve_gb":18.588,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.382,"kv_cache_gb_per_8k":1.208,"min_serve_gb":18.59,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-8B/tree/main"},{"quant":"fp8","weights_gb":8.681,"kv_cache_gb_per_8k":1.208,"min_serve_gb":10.889,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.914,"kv_cache_gb_per_8k":1.208,"min_serve_gb":7.122,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.681,"kv_cache_gb_per_8k":1.208,"min_serve_gb":10.889,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":42.64,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":74.24,"bw_limited_tok_s":44.54,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":19.61,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":33.33,"bw_limited_tok_s":20,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":197.06,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":409.04,"bw_limited_tok_s":245.42,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.8,"score_norm":0.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-8b","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.58,"score_norm":1.58,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":9,"score_norm":9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-8b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.3,"score_norm":7.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-8b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.29,"score_norm":8.29,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":24.33,"score_norm":5.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.243333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":58.89,"score_norm":47.73,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.588888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.89,"score_norm":2.78,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0389 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":33.47,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.33469387755102 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":40.63,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.406349206349206 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":22.57,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.225694444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":27.78,"score_norm":10.37,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":4.74,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0474226804123711 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.25,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0224719101123595 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://arxiv.org/html/2505.09388v1","publisher":"arXiv"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-8b-instruct-vision","display_name":"Qwen3 VL 8B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"8b","params_total_b":8.77,"params_active_b":8.77,"params_exact":8767123696,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.5,"int8_gb":9.3,"int4_gb":5.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.98,"expected_int8_gb":9.1,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-8B-Instruct","openrouter_id":"qwen/qwen3-vl-8b-instruct","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["image","text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-10-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-vl-8b-instruct"],"intelligence":{"index":12.13,"inputs":[{"benchmark":"aime","score_raw":27.33,"score_norm":9.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.273333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":42.73,"score_norm":24.97,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.427272727272727 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":2.69,"score_norm":1.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0268767377201112 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":29.24,"score_norm":12.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.292397660818713 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.117,"completion_usd_per_m":0.455,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-vl-8b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.2015,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":17.54,"kv_cache_gb_per_8k":1.208,"min_serve_gb":19.748,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.534,"kv_cache_gb_per_8k":1.208,"min_serve_gb":19.742,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct/tree/main"},{"quant":"fp8","weights_gb":9.296,"kv_cache_gb_per_8k":1.208,"min_serve_gb":11.504,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.262,"kv_cache_gb_per_8k":1.208,"min_serve_gb":7.47,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.296,"kv_cache_gb_per_8k":1.208,"min_serve_gb":11.504,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.94,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":69.33,"bw_limited_tok_s":41.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":31.13,"bw_limited_tok_s":18.68,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":186.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":381.98,"bw_limited_tok_s":229.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.3,"score_norm":7.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.24,"score_norm":8.24,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":16.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.166666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":27.33,"score_norm":9.16,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.273333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.73,"score_norm":24.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.427272727272727 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":2.69,"score_norm":1.01,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0268767377201112 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":32.31,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.32312925170068 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":33.23,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.332275132275132 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":47.34,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.473410404624277 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":17.36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.173611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":29.24,"score_norm":12.32,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.292397660818713 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.27,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0227272727272727 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-vl-8b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-8b-instruct","publisher":"OpenRouter (listed 2025-10-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-8b-reasoning-vision","display_name":"Qwen3 VL 8B Thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"8b","params_total_b":8.77,"params_active_b":8.77,"params_exact":8767123696,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.5,"int8_gb":9.3,"int4_gb":5.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":6.83,"expected_int8_gb":63.5,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-VL-8B-Thinking","openrouter_id":"qwen/qwen3-vl-8b-thinking","is_moe":false,"variant_axis":"vision","kind":"model","modalities":["image","text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-10-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-vl-8b-thinking"],"intelligence":{"index":65.74,"inputs":[{"benchmark":"aime","score_raw":80.3,"score_norm":75.37,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"25, thinking mode"},{"benchmark":"gpqa_diamond","score_raw":69.9,"score_norm":63.24,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"thinking mode"},{"benchmark":"livebench","score_raw":69.8,"score_norm":66.22,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"2024.11.25, thinking mode"},{"benchmark":"mmlu_pro","score_raw":77.3,"score_norm":62.17,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v1","config":"thinking mode"}],"coverage":0.4762,"subset_bias":{"offset":11.41,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.18,"completion_usd_per_m":2.1,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-vl-8b-thinking","as_of":"2026-10-06","blended_usd_per_m":0.66,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":17.54,"kv_cache_gb_per_8k":1.208,"min_serve_gb":19.748,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.534,"kv_cache_gb_per_8k":1.208,"min_serve_gb":19.742,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking/tree/main"},{"quant":"fp8","weights_gb":9.296,"kv_cache_gb_per_8k":1.208,"min_serve_gb":11.504,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.262,"kv_cache_gb_per_8k":1.208,"min_serve_gb":7.47,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.296,"kv_cache_gb_per_8k":1.208,"min_serve_gb":11.504,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":39.94,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":69.33,"bw_limited_tok_s":41.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.33,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":31.13,"bw_limited_tok_s":18.68,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":186.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":8.77,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":381.98,"bw_limited_tok_s":229.19,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":8.2,"score_norm":8.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-8b-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":80.3,"score_norm":75.37,"config":"25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":51.1,"score_norm":null,"config":"thinking mode, winrate","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":63,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":69.9,"score_norm":63.24,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":60.6,"score_norm":null,"config":"25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":83.2,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":69.8,"score_norm":66.22,"config":"2024.11.25, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":58.6,"score_norm":null,"config":"6, thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":77.3,"score_norm":62.17,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_prox","score_raw":70.7,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_redux","score_raw":88.8,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"multiif","score_raw":75.1,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"polymath","score_raw":47.5,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":51.2,"score_norm":null,"config":"thinking mode","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-8b-thinking","publisher":"OpenRouter (listed 2025-10-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-flash-code","display_name":"Qwen3 Coder Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3-coder-flash","is_moe":false,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-09-17","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-coder-flash"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.195,"completion_usd_per_m":0.975,"cache_read_usd_per_m":0.039,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3-coder-flash","as_of":"2026-10-06","blended_usd_per_m":0.39,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-flash","publisher":"OpenRouter (listed 2025-09-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-max-reasoning","display_name":"Qwen3 Max Thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3-max-thinking","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-02-09","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-max-thinking"],"intelligence":{"index":68.95,"inputs":[{"benchmark":"aime","score_raw":82.33,"score_norm":77.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking-preview","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: preview, source value 0.823333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":86.1,"score_norm":86.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","config":"artificial analysis independent run, no tools"},{"benchmark":"hle","score_raw":28,"score_norm":38.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","config":"artificial analysis independent run, no tools"},{"benchmark":"mmlu_pro","score_raw":85.7,"score_norm":76.17,"trust":1,"source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","config":"qwen3-max-thinking, vendor-reported"},{"benchmark":"swe_bench_verified","score_raw":75.3,"score_norm":72.56,"trust":1,"source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","config":"reported as 'swe verified' (agentic coding)"},{"benchmark":"tau_bench","score_raw":83.63,"score_norm":84.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max thinking, source value 0.836257309941521 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":24.24,"score_norm":22.64,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max thinking, source value 0.242424242424242 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7223,"subset_bias":{"offset":4.12,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.78,"completion_usd_per_m":3.9,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-max-thinking","as_of":"2026-10-06","blended_usd_per_m":1.56,"endpoint_expires_at":"2026-10-09"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":21.3,"score_norm":21.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":32.5,"score_norm":32.5,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max thinking","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: max thinking, source value 0.703333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":82.33,"score_norm":77.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: preview, source value 0.823333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: max thinking, source value 0.0171428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.06,"score_norm":86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: max thinking, source value 0.860606060606061 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.1,"score_norm":86.06,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.4,"score_norm":87.89,"config":"reported as 'gpqa' in qwen vendor table","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","measured_date":"2026-01-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.99,"score_norm":38.22,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: max thinking, source value 0.279888785912882 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28,"score_norm":38.24,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.2,"score_norm":41.47,"config":"no tools, text subset only","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","measured_date":"2026-01-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":70.75,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: max thinking, source value 0.707482993197279 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":53.54,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: preview, source value 0.535449735449735 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.7,"score_norm":76.17,"config":"qwen3-max-thinking, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","measured_date":"2026-01-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":43.06,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max thinking, scicode, source value 0.430555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75.3,"score_norm":72.56,"config":"reported as 'swe verified' (agentic coding)","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","measured_date":"2026-01-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":83.63,"score_norm":84.84,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max thinking, source value 0.836257309941521 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":82.1,"score_norm":82.8,"config":"no custom scaffolding, official tau-squared bench setting, tau2","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3-max-thinking","measured_date":"2026-01-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.24,"score_norm":22.64,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max thinking, source value 0.242424242424242 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.2,"score_norm":22.59,"config":"artificial analysis intelligence index, terminal-bench hard","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-thinking","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max-thinking","publisher":"OpenRouter (listed 2026-02-09)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-next-80b-a3b-2509-instruct","display_name":"Qwen3 Next 80B A3B Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"80b-a3b","params_total_b":81.32,"params_active_b":3,"params_exact":81324862720,"params_source":"hf_safetensors","vram_est":{"bf16_gb":163,"int8_gb":86.2,"int4_gb":48.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.05,"expected_int8_gb":176,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-Next-80B-A3B-Instruct","openrouter_id":"qwen/qwen3-next-80b-a3b-instruct","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-09-17","release_status":"released","knowledge_cutoff":"2025-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-3-next-80b-a3b","qwen-3-next-80b-a3b-instruct","qwen3-next-80b-a3b","qwen3-next-80b-a3b-instruct","qwen3-next-80b-a3b-instruct-2509"],"intelligence":{"index":50.45,"inputs":[{"benchmark":"aime","score_raw":66.33,"score_norm":57.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.663333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1399,"score_norm":53.2,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":73.84,"score_norm":68.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.738383838383838 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":7.65,"score_norm":8.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0764596848934198 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":75.8,"score_norm":79.56,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","config":"global average. official qwen model card, livebench 2024-11-25 release, reasoning section. non-thinking instruct mode. [raw: livebench 20241125]"},{"benchmark":"mmlu_pro","score_raw":80.6,"score_norm":67.67,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","config":"knowledge section. non-thinking instruct mode, no tools. [raw: mmlu-pro], official qwen model card"},{"benchmark":"tau_bench","score_raw":21.64,"score_norm":2.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.216374269005848 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":7.58,"score_norm":3.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7619,"subset_bias":{"offset":2.81,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":1.1,"cache_read_usd_per_m":0.07,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-next-80b-a3b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.35000000000000003,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":162.64,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":170.973,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":162.659,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":170.993,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/tree/main"},{"quant":"fp8","weights_gb":86.199,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":90.71,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":48.792,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":51.433,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":86.199,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":90.71,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":572.65,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":2233.33,"bw_limited_tok_s":1340,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.6,"score_norm":9.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.75,"score_norm":13.75,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":52,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.52 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":66.33,"score_norm":57.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.663333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":69.5,"score_norm":61.88,"config":"aime 2025. official qwen model card, no tools. [raw: aime25], reasoning section. non-thinking instruct mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1399,"score_norm":53.2,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1400.6,"score_norm":53.92,"config":"23000 votes, arena entry qwen3-next-80b-a3b-instruct, leaderboard text-overall-style_control, rank 143, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.84,"score_norm":68.79,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.738383838383838 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":72.9,"score_norm":67.46,"config":"knowledge section. row label is plainly \"gpqa\", official qwen model card, the hf model-index evaluation-results block on the same page labels the identical 72.9 as \"gpqa diamond\". non-thinking instruc [raw: gpqa]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.65,"score_norm":8.31,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0764596848934198 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":39.66,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.396598639455782 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":75.8,"score_norm":79.56,"config":"global average. official qwen model card, livebench 2024-11-25 release, reasoning section. non-thinking instruct mode. [raw: livebench 20241125]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":68.36,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.683597883597884 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.6,"score_norm":67.67,"config":"knowledge section. non-thinking instruct mode, no tools. [raw: mmlu-pro], official qwen model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":30.67,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.306712962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":21.64,"score_norm":2.19,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.216374269005848 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":57.3,"score_norm":49.73,"config":"retail domain only — qwen publishes no cross-domain average. sibling domain rows on the same card: tau2-airline 45.5, tau1-airline 44.0. [raw: tau2-retail], tau1-retail 60.9, tau2-bench (τ2-bench), tau2-telecom 13.2","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":7.58,"score_norm":3.04,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-next-80b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-instruct","publisher":"OpenRouter (listed 2025-09-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-next-80b-a3b-2509-reasoning","display_name":"Qwen3 Next 80B A3B Thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"80b-a3b","params_total_b":81.32,"params_active_b":3,"params_exact":81324862720,"params_source":"hf_safetensors","vram_est":{"bf16_gb":163,"int8_gb":86.2,"int4_gb":48.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.89,"expected_int8_gb":336,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-Next-80B-A3B-Thinking","openrouter_id":"qwen/qwen3-next-80b-a3b-thinking","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-09-11","release_status":"released","knowledge_cutoff":"2025-09-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-next-80b-a3b-thinking","qwen3-next-80b-a3b-thinking-2509"],"intelligence":{"index":70.89,"inputs":[{"benchmark":"aime","score_raw":87.8,"score_norm":84.75,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","config":"aime 2025, no tools, standard thinking mode [raw: aime25]"},{"benchmark":"arena_elo","score_raw":1369.4,"score_norm":40.01,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.2,"score_norm":73.52,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","config":"labeled simply 'gpqa' (qwen's convention is the gpqa-diamond subset), no tools, official qwen model card 'knowledge' row, standard thinking mode [raw: gpqa]"},{"benchmark":"livebench","score_raw":76.6,"score_norm":81.33,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","config":"global average, livebench 2024-11-25 release, vendor self-reported (not the live livebench.ai leaderboard) [raw: livebench 241125]"},{"benchmark":"mmlu_pro","score_raw":82.7,"score_norm":71.17,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","config":"no tools, official qwen model card 'knowledge' row, standard thinking mode [raw: mmlu-pro]"},{"benchmark":"tau_bench","score_raw":67.8,"score_norm":63.73,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","config":"card also lists tau2-airline 60.5 and tau2-telecom 43.9 (and tau1-retail 69.6 / tau1-airline 49.0) [raw: tau2-retail], retail domain only — no average is published, tau2-bench"}],"coverage":0.6032,"subset_bias":{"offset":9.59,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":1.2,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-next-80b-a3b-thinking","as_of":"2026-10-06","blended_usd_per_m":0.4125,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":162.64,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":170.973,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":162.659,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":170.993,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking/tree/main"},{"quant":"fp8","weights_gb":86.199,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":90.71,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":48.792,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":51.433,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":86.199,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":90.71,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":572.65,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":2233.33,"bw_limited_tok_s":1340,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":2.1,"score_norm":2.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-thinking","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":17.4,"score_norm":17.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-thinking","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.2,"score_norm":11.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87.8,"score_norm":84.75,"config":"aime 2025, no tools, standard thinking mode [raw: aime25]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1369.4,"score_norm":40.01,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1368.9,"score_norm":39.79,"config":"13749 votes, arena entry qwen3-next-80b-a3b-thinking, leaderboard text-overall-style_control, rank 180, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.2,"score_norm":73.52,"config":"labeled simply 'gpqa' (qwen's convention is the gpqa-diamond subset), no tools, official qwen model card 'knowledge' row, standard thinking mode [raw: gpqa]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":76.6,"score_norm":81.33,"config":"global average, livebench 2024-11-25 release, vendor self-reported (not the live livebench.ai leaderboard) [raw: livebench 241125]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.7,"score_norm":71.17,"config":"no tools, official qwen model card 'knowledge' row, standard thinking mode [raw: mmlu-pro]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":67.8,"score_norm":63.73,"config":"card also lists tau2-airline 60.5 and tau2-telecom 43.9 (and tau1-retail 69.6 / tau1-airline 49.0) [raw: tau2-retail], retail domain only — no average is published, tau2-bench","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-next-80b-a3b-thinking","publisher":"OpenRouter (listed 2025-09-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-next-code","display_name":"Qwen3 Coder Next","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"frontier","params_total_b":79.67,"params_active_b":null,"params_exact":79674391296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":159,"int8_gb":84.5,"int4_gb":47.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.36,"expected_int8_gb":199,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3-Coder-Next","openrouter_id":"qwen/qwen3-coder-next","is_moe":true,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-02-04","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-coder-next","qwen3-coder-next-2025-02-03"],"intelligence":{"index":54.53,"inputs":[{"benchmark":"gpqa_diamond","score_raw":73.74,"score_norm":68.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.737373737373737 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":10.15,"score_norm":11.99,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.101482854494903 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":70.6,"score_norm":67.33,"trust":2,"source_url":"https://arxiv.org/abs/2603.00729","config":"swe-agent as harness"},{"benchmark":"tau_bench","score_raw":79.53,"score_norm":79.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.795321637426901 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":38.2,"score_norm":39.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.382022471910112 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5239,"subset_bias":{"offset":-3.11,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.12,"completion_usd_per_m":0.8,"cache_read_usd_per_m":0.07,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-coder-next","as_of":"2026-10-06","blended_usd_per_m":0.29000000000000004,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":159.34,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":167.508,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":159.358,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":167.527,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-Coder-Next/tree/main"},{"quant":"fp8","weights_gb":84.45,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":88.874,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":47.802,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":50.393,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":84.45,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":88.874,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":1.93,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":79.67,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.43,"bw_limited_tok_s":1.94,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":48.03,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":79.67,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":84.1,"bw_limited_tok_s":50.46,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.9,"score_norm":0.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-next","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":8.86,"score_norm":8.86,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":36.2,"score_norm":36.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-next","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.2,"score_norm":9.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-next","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.29,"score_norm":21.29,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":42.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.423333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":10.83,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0.10825499999999999 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.74,"score_norm":68.65,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.737373737373737 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.15,"score_norm":11.99,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.101482854494903 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":35.24,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.352380952380952 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":32.29,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.322916666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":44.3,"score_norm":65.5,"config":"swe-agent as harness","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2603.00729","measured_date":"2026-03-16","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70.6,"score_norm":67.33,"config":"swe-agent as harness","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2603.00729","measured_date":"2026-03-16","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":79.53,"score_norm":79.37,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.795321637426901 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.36,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0536082474226804 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":18.18,"score_norm":15.51,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.181818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38.2,"score_norm":39.06,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.382022471910112 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-coder-next","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":36.2,"score_norm":36.71,"config":"2.0, agent: terminus 2","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-Coder-Next","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-next","publisher":"OpenRouter (listed 2026-02-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-plus-code","display_name":"Qwen3 Coder Plus","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"frontier-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3-coder-plus","is_moe":false,"variant_axis":"code","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-09-23","release_status":"released","knowledge_cutoff":"2025-06-30","status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3-coder-plus"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.65,"completion_usd_per_m":3.25,"cache_read_usd_per_m":0.13,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3-coder-plus","as_of":"2026-10-06","blended_usd_per_m":1.3,"endpoint_expires_at":"2026-10-09"},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-coder-plus","publisher":"OpenRouter (listed 2025-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-122b-a10b","display_name":"Qwen3.5-122B-A10B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"122b-a10b","params_total_b":125.09,"params_active_b":10,"params_exact":125086497008,"params_source":"hf_safetensors","vram_est":{"bf16_gb":250,"int8_gb":133,"int4_gb":75.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.33,"expected_int8_gb":308,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.5-122B-A10B","openrouter_id":"qwen/qwen3.5-122b-a10b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-02-25","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.5-122b-a10b"],"intelligence":{"index":68.46,"inputs":[{"benchmark":"arena_elo","score_raw":1416.1,"score_norm":60.83,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":25.21,"score_norm":34.13,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.252085264133457 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.7,"score_norm":77.83,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","config":"official model card language table, thinking mode [raw: mmlu-pro]"},{"benchmark":"swe_bench_verified","score_raw":72,"score_norm":68.89,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","config":"official model card language table [raw: swe-bench verified], resolved rate"},{"benchmark":"tau_bench","score_raw":93.57,"score_norm":98.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.935672514619883 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":47.57,"score_norm":50.08,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.47565543071161 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7223,"subset_bias":{"offset":1.05,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.26,"completion_usd_per_m":2.08,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.5-122b-a10b","as_of":"2026-10-06","blended_usd_per_m":0.7150000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":250.18,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":262.89,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":250.173,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":262.883,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B/tree/main"},{"quant":"fp8","weights_gb":132.595,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":139.426,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":75.054,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":79.008,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":132.595,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":139.426,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":29.97,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":10,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":54.6,"bw_limited_tok_s":30.89,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":286.73,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":10,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":670,"bw_limited_tok_s":402,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":7.6,"score_norm":7.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-122b-a10b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.27,"score_norm":21.27,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":43.3,"score_norm":43.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-122b-a10b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":17.7,"score_norm":17.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-122b-a10b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":32.85,"score_norm":32.85,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.703333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1416.1,"score_norm":60.83,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417,"score_norm":61.23,"config":"28413 votes, arena entry qwen3.5-122b-a10b, leaderboard text-overall-style_control, rank 119, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1226.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1227.1,"score_norm":null,"config":"12453 votes, arena entry qwen3.5-122b-a10b, leaderboard vision-overall-style_control, rank 56, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1359.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1357.7,"score_norm":null,"config":"7862 votes, arena entry qwen3.5-122b-a10b-code, code/overall, leaderboard webdev-overall-raw, rank 84, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0.00857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":24.39,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.24387 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.6,"score_norm":86.76,"config":"no tools [raw: gpqa diamond], official model card language table, thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":25.21,"score_norm":34.13,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.252085264133457 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":47.5,"score_norm":66.91,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":25.3,"score_norm":34.26,"config":"chain-of-thought, humanity's last exam, no tools (card also lists a separate 'hle w/ tool' search-agent score of 47.5, not used) [raw: hle w/ cot]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.71,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.757142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.7,"score_norm":77.83,"config":"official model card language table, thinking mode [raw: mmlu-pro]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.97,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.74971098265896 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.01,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.420138888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72,"score_norm":68.89,"config":"official model card language table [raw: swe-bench verified], resolved rate","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":93.57,"score_norm":98.09,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.935672514619883 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":79.5,"score_norm":79.33,"config":"single published number (no per-domain breakdown given on card) [raw: tau2-bench], tau2-bench","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":15.26,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.152577319587629 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":47.57,"score_norm":50.08,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.47565543071161 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-122b-a10b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":49.4,"score_norm":52.24,"config":"terminal-bench 2.0 [raw: terminal bench 2]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-122B-A10B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-122b-a10b","publisher":"OpenRouter (listed 2026-02-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-397b-a17b","display_name":"Qwen3.5 397B A17B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"397b-a17b","params_total_b":403.4,"params_active_b":17,"params_exact":403397928944,"params_source":"hf_safetensors","vram_est":{"bf16_gb":807,"int8_gb":428,"int4_gb":242,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.92,"expected_int8_gb":392,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.5-397B-A17B","openrouter_id":"qwen/qwen3.5-397b-a17b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-02-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.5-397b-a17b"],"intelligence":{"index":75.03,"inputs":[{"benchmark":"aime","score_raw":91.3,"score_norm":89.13,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","config":"aime 2026, no tools, reasoning mode. model-card row \"aime26\". note: the hf model-index eval-results block separately lists a matharena aime 2026 figure of 93.33 for the same model. [raw: aime26]"},{"benchmark":"arena_elo","score_raw":1441.8,"score_norm":72.28,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.29,"score_norm":90.55,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.892929292929293 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":28.96,"score_norm":39.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.289620018535681 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":87.8,"score_norm":79.67,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","config":"reasoning mode. official qwen model card language-benchmark table. [raw: mmlu-pro]"},{"benchmark":"swe_bench_verified","score_raw":76.4,"score_norm":73.78,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","config":"hf eval-results section repeats it as \"swe bench resolved\" 76.4. [raw: swe-bench verified], qwen internal bash + file-edit agent scaffold, resolved rate on the full verified set. model-card row \"swe-bench verified\""},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":51.31,"score_norm":54.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.51310861423221 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.45,"completion_usd_per_m":3,"cache_read_usd_per_m":0.22,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.5-397b-a17b","as_of":"2026-10-06","blended_usd_per_m":1.0875,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":806.8,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":847.392,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":806.796,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":847.388,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B/tree/main"},{"quant":"fp8","weights_gb":427.604,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":449.236,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":242.04,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":254.394,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":427.604,"kv_cache_gb_per_8k":0.2517,"min_serve_gb":449.236,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":8.3,"score_norm":8.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":19.85,"score_norm":19.85,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":48.2,"score_norm":48.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.4,"score_norm":21.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.26,"score_norm":34.26,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.726666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.3,"score_norm":89.13,"config":"aime 2026, no tools, reasoning mode. model-card row \"aime26\". note: the hf model-index eval-results block separately lists a matharena aime 2026 figure of 93.33 for the same model. [raw: aime26]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":15.34,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.153392330383481 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.8,"score_norm":72.28,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.8,"score_norm":72.28,"config":"70207 votes, arena entry qwen3.5-397b-a17b-text, leaderboard text-overall-style_control, rank 78, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1246.9,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1247.6,"score_norm":null,"config":"25396 votes, arena entry qwen3.5-397b-a17b-vision, leaderboard vision-overall-style_control, rank 43, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1399.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1399.2,"score_norm":null,"config":"18612 votes, arena entry qwen3.5-397b-a17b-webdev, code/overall, leaderboard webdev-overall-raw, rank 66, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":12.2,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.12195605269463661 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0170612244897959 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1186,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1184,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1153,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1197,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":30.89,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.3088630259623993 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":23.21,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.23212 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.29,"score_norm":90.55,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.892929292929293 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.4,"score_norm":89.3,"config":"no tools. official qwen model card comparison table, reasoning mode, row labeled \"gpqa\" (hf model-index eval-results section labels the dataset \"diamond\"). confirmed in 3 separate fetches of the model [raw: gpqa diamond]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":72.38,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.723838471558836 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":28.96,"score_norm":39.65,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.289620018535681 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48.3,"score_norm":68.09,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":28.7,"score_norm":39.26,"config":"no tools. model card separately reports hle-verified 37.6 and \"hle w/ tool\" 48.3 (search-agent setting) — this claim is the plain no-tool number. [raw: humanity's last exam (hle)], text-only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":78.78,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.787755102040816 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":34.09,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.340866290018832 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.8,"score_norm":79.67,"config":"reasoning mode. official qwen model card language-benchmark table. [raw: mmlu-pro]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":77.28,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.772832369942197 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.01,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.420138888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.4,"score_norm":73.78,"config":"hf eval-results section repeats it as \"swe bench resolved\" 76.4. [raw: swe-bench verified], qwen internal bash + file-edit agent scaffold, resolved rate on the full verified set. model-card row \"swe-bench verified\"","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":86.7,"score_norm":88.93,"config":"tau2-bench (τ2-bench) overall/aggregate score across domains. model-card footnote: official setup followed except the airline domain, which applies the fixes proposed in the claude opus 4.5 system car [raw: tau2-bench]","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":13.4,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.134020618556701 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":40.91,"score_norm":42.25,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.409090909090909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":51.31,"score_norm":54.48,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.51310861423221 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-397b-a17b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.5,"score_norm":55.88,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.5,"score_norm":55.88,"config":"reasoning mode. not the terminal-bench hard / v2.1 variant artificial analysis reports. [raw: terminal bench 2], terminal-bench 2.0 (model-card row \"terminal bench 2\")","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-397b-a17b","publisher":"OpenRouter (listed 2026-02-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-9b","display_name":"Qwen3.5-9B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"9b","params_total_b":9.65,"params_active_b":9.65,"params_exact":9653104368,"params_source":"hf_safetensors","vram_est":{"bf16_gb":19.3,"int8_gb":10.2,"int4_gb":5.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":5.03,"expected_int8_gb":51.4,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.5-9B","openrouter_id":"qwen/qwen3.5-9b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-03-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.5-9b"],"intelligence":{"index":59.04,"inputs":[{"benchmark":"gpqa_diamond","score_raw":80.61,"score_norm":78.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.806060606060606 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":14.92,"score_norm":19,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.149212233549583 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-9B","config":null},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":29.21,"score_norm":28.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.292134831460674 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.4842,"subset_bias":{"offset":1.77,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.15,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.5-9b","as_of":"2026-10-06","blended_usd_per_m":0.11250000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":19.3,"kv_cache_gb_per_8k":0.2684,"min_serve_gb":20.568,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":19.306,"kv_cache_gb_per_8k":0.2684,"min_serve_gb":20.574,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.5-9B/tree/main"},{"quant":"fp8","weights_gb":10.229,"kv_cache_gb_per_8k":0.2684,"min_serve_gb":11.497,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.79,"kv_cache_gb_per_8k":0.2684,"min_serve_gb":7.058,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":10.229,"kv_cache_gb_per_8k":0.2684,"min_serve_gb":11.497,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":36.43,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":9.65,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":63.01,"bw_limited_tok_s":37.8,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":16.69,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":9.65,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":28.29,"bw_limited_tok_s":16.97,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":172.38,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":9.65,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":347.15,"bw_limited_tok_s":208.29,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.2,"score_norm":1.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-9b","measured_date":null,"observed_at":"2026-09-29","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":7,"score_norm":7,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":23.5,"score_norm":23.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-9b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.3,"score_norm":13.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-9b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.79,"score_norm":21.79,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.653333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0.00571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":7.19,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.07190499999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.61,"score_norm":78.32,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.806060606060606 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.7,"score_norm":79.86,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-9B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.92,"score_norm":19,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.149212233549583 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":66.73,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.66734693877551 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-9B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.25,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.692485549132948 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":27.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, scicode, source value 0.276821658615137 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":7.01,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0701030927835052 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.24,"score_norm":22.64,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.242424242424242 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":29.21,"score_norm":28.48,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.292134831460674 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-9b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-9b","publisher":"OpenRouter (listed 2026-03-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.6-35b-a3b","display_name":"Qwen3.6 35B A3B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.6","size_class":"35b-a3b","params_total_b":35.95,"params_active_b":3,"params_exact":35951822704,"params_source":"hf_safetensors","vram_est":{"bf16_gb":71.9,"int8_gb":38.1,"int4_gb":21.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":8.68,"expected_int8_gb":331,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.6-35B-A3B","openrouter_id":"qwen/qwen3.6-35b-a3b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["qwen-3.5-35b-a3b"],"superseded_by":null,"aliases":["qwen3.6-35b-a3b"],"intelligence":{"index":70.47,"inputs":[{"benchmark":"aime","score_raw":92.7,"score_norm":90.88,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","config":"2026"},{"benchmark":"gpqa_diamond","score_raw":84.14,"score_norm":83.3,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.841414141414142 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":22.24,"score_norm":29.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.222428174235403 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","config":null},{"benchmark":"swe_bench_verified","score_raw":73.4,"score_norm":70.44,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","config":null},{"benchmark":"tau_bench","score_raw":95.32,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.953216374269006 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":44.94,"score_norm":46.99,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7223,"subset_bias":{"offset":3.02,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":1,"cache_read_usd_per_m":0.05,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.6-35b-a3b","as_of":"2026-10-06","blended_usd_per_m":0.3625,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":71.9,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":75.663,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":71.904,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":75.667,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B/tree/main"},{"quant":"fp8","weights_gb":38.107,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":40.18,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":21.57,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":22.816,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":38.107,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":40.18,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":195.62,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"bw_limited_tok_s":243.2,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":48.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"bw_limited_tok_s":51.49,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":401.2,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"bw_limited_tok_s":670,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":13.1,"score_norm":13.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-35b-a3b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.62,"score_norm":21.62,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":41.9,"score_norm":41.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-35b-a3b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":18.2,"score_norm":18.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-35b-a3b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":32.13,"score_norm":32.13,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":66.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.666666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":92.7,"score_norm":90.88,"config":"2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":27.76,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.27763 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.14,"score_norm":83.3,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.841414141414142 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86,"score_norm":85.92,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.24,"score_norm":29.76,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.222428174235403 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":21.4,"score_norm":28.53,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":64.35,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.643537414965986 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.2,"score_norm":75.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.03,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.75028901734104 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":35.76,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.357638888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":49.5,"score_norm":74.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.4,"score_norm":70.44,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.32,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.953216374269006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.28,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0927835051546392 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.94,"score_norm":46.99,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":51.5,"score_norm":54.71,"config":"2.0, 256k ctx, 32 cpu/48 gb ram, 3h timeout, avg of 5 runs., harbor/terminus-2 harness, max_tokens=80k, temp=1.0, top_k=20, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-35b-a3b","publisher":"OpenRouter (listed 2026-04-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.7-max","display_name":"Qwen3.7-Max","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.7","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.7-max","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-05-16","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["qwen-3-max","qwen-3.6-max"],"superseded_by":null,"aliases":["qwen3.7-max"],"intelligence":{"index":75.68,"inputs":[{"benchmark":"gpqa_diamond","score_raw":90.15,"score_norm":91.76,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=1.497, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":40.5,"score_norm":56.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-7-max","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: max, source value 0.405004633920297 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":73.14,"score_norm":73.64,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: max"},{"benchmark":"mmlu_pro","score_raw":89.31,"score_norm":82.18,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.305, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":68.8,"score_norm":65.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, run: max, runner=external, stderr=2.074, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":94.74,"score_norm":99.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-7-max","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.947368421052632 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":61.05,"score_norm":65.94,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, run: max, runner=external, stderr=0.375, task=overall, updated=2026-08-19"}],"coverage":0.762,"subset_bias":{"offset":-0.2,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.475,"completion_usd_per_m":4.425,"cache_read_usd_per_m":0.295,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.7-max","as_of":"2026-10-06","blended_usd_per_m":2.2125000000000004,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":22.5,"score_norm":22.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":30.86,"score_norm":30.86,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":66,"score_norm":66,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.5,"score_norm":29.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":46.71,"score_norm":46.71,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: max, source value 0.746666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":13.48,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":18.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.1875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":25.58,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.2557589370521027 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":13.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: max, source value 0.134285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1282,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1161,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1174,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1224,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1281,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1288,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1188,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1280,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1187,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1158,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1147,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1194,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1216,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1202,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1280,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":45,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.4500149209191286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":38.55,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: max, source value 0.3855 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.15,"score_norm":91.76,"config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=1.497, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.32,"score_norm":94.82,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: max, source value 0.923232323232323 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":83.43,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.834274135185989 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":40.5,"score_norm":56.62,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: max, source value 0.405004633920297 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":80.54,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: max, source value 0.805442176870748 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":42.47,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: max, source value 0.424670433145009 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":73.14,"score_norm":73.64,"config":"livebench-2026-06-25 release, run: max","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.31,"score_norm":82.18,"config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.305, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":48.84,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, scicode, source value 0.488425925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":68.8,"score_norm":65.33,"config":"edition=swe-bench verified v1, run: max, runner=external, stderr=2.074, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":94.74,"score_norm":99.65,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.947368421052632 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":11.75,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.117525773195876 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.05,"score_norm":65.94,"config":"edition=terminal-bench 2.1, run: max, runner=external, stderr=0.375, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.76,"score_norm":53.84,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.507575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.53,"score_norm":81.8,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.745318352059925 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwen.ai/apiplatform","publisher":"Alibaba"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-max","publisher":"OpenRouter (listed 2026-05-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.7-plus","display_name":"Qwen3.7-Plus","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.7","size_class":"frontier-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.7-plus","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-06-03","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["qwen-3.5-plus","qwen-3.6-plus-04-02"],"superseded_by":null,"aliases":["qwen-3.7-plus-vision","qwen3.7-plus"],"intelligence":{"index":76.34,"inputs":[{"benchmark":"arena_elo","score_raw":1455.8,"score_norm":78.52,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":90,"score_norm":91.55,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.9 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":35.63,"score_norm":49.46,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.356348470806302 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":88.5,"score_norm":80.83,"trust":1,"source_url":"https://qwen.ai/blog?id=qwen3.7-plus","config":"recommended system prompt sets reasoning effort to xhigh, vendor-reported"},{"benchmark":"swe_bench_verified","score_raw":77.7,"score_norm":75.22,"trust":1,"source_url":"https://qwen.ai/blog?id=qwen3.7-plus","config":"200k context window, internal agent scaffold (bash + file-edit tools), temp=1.0, top_p=0.95, vendor-reported"},{"benchmark":"tau_bench","score_raw":92.98,"score_norm":97.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.929824561403509 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":52.81,"score_norm":56.25,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=0.649, task=overall, updated=2026-08-19"}],"coverage":0.7223,"subset_bias":{"offset":0.24,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.32,"completion_usd_per_m":1.28,"cache_read_usd_per_m":0.064,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.7-plus","as_of":"2026-10-06","blended_usd_per_m":0.56,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":17.5,"score_norm":17.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":20.69,"score_norm":20.69,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":55.9,"score_norm":55.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.2,"score_norm":25.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.37,"score_norm":39.37,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":69,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.69 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":1.08,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":22.42,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.224188790560472 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1455.8,"score_norm":78.52,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1457.5,"score_norm":79.28,"config":"34426 votes, arena entry qwen3.7-plus, leaderboard text-overall-style_control, rank 53, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1444.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1439.6,"score_norm":null,"config":"3088 votes, arena entry qwen3.7-plus, document/overall, leaderboard document-overall-raw, rank 26, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1265.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1265.5,"score_norm":null,"config":"8627 votes, arena entry qwen3.7-plus, leaderboard vision-overall-style_control, rank 31, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":20.43,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.20433540292579966 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":9.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0914285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1252,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1140,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1274,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1284,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1208,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1277,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":40.56,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.40555058191584603 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":22.25,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.22248500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90,"score_norm":91.55,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.9 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90,"score_norm":91.55,"config":"artificial analysis eval, qwen3.7 plus","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.3,"score_norm":91.97,"config":"recommended system prompt sets reasoning effort to xhigh, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":81.79,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.817918656824432 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":35.63,"score_norm":49.46,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.356348470806302 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":35.63,"score_norm":49.46,"config":"artificial analysis eval, qwen3.7 plus","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":34.7,"score_norm":48.09,"config":"recommended system prompt sets reasoning effort to xhigh, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77.96,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.779591836734694 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":9.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0944444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.5,"score_norm":80.83,"config":"recommended system prompt sets reasoning effort to xhigh, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.46,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.804624277456647 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mrcr","score_raw":91.7,"score_norm":93.12,"config":"128k context, v2, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":45.49,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.454861111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.7,"score_norm":75.22,"config":"200k context window, internal agent scaffold (bash + file-edit tools), temp=1.0, top_p=0.95, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.98,"score_norm":97.31,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.929824561403509 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":17.53,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.175257731958763 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.81,"score_norm":56.25,"config":"edition=terminal-bench 2.1, runner=external, stderr=0.649, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.81,"score_norm":56.25,"config":"overall, terminal-bench 2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.97,"score_norm":49.38,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.46969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.05,"score_norm":65.94,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.610486891385768 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.05,"score_norm":65.94,"config":"artificial analysis eval, qwen3.7 plus, terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-7-plus","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":70.3,"score_norm":76.82,"config":"2.0, 256k ctx, 5h timeout, avg of 5 runs, harbor/terminus-2 harness, max_tokens=80k, temp=1.0, top_k=20, top_p=0.95, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://qwen.ai/blog?id=qwen3.7-plus","measured_date":"2026-05-31","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwen.ai/apiplatform","publisher":"Alibaba"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-plus","publisher":"OpenRouter (listed 2026-06-03)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.8-2.4t-a95b","display_name":"Qwen3.8-2.4T-A95B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"2.4t-a95b","params_total_b":2446.18,"params_active_b":95,"params_exact":2446182725504,"params_source":"hf_safetensors","vram_est":{"bf16_gb":4892,"int8_gb":2593,"int4_gb":1468,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.21,"expected_int8_gb":551,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.8-2.4T-A95B","openrouter_id":"qwen/qwen3.8-2.4t-a95b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.8-2.4t-a95b"],"intelligence":{"index":82.69,"inputs":[{"benchmark":"gpqa_diamond","score_raw":93.54,"score_norm":96.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.935353535353535 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":42.45,"score_norm":59.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.424467099165894 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":82.02,"score_norm":90.61,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.820224719101124 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3174,"subset_bias":{"offset":-0.05,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.25,"context_length":1048576,"source":"openrouter","source_model_id":"qwen/qwen3.8-2.4t-a95b","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":4892.36,"kv_cache_gb_per_8k":0.7718,"min_serve_gb":5137.75,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":4892.366,"kv_cache_gb_per_8k":0.7718,"min_serve_gb":5137.756,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/tree/main"},{"quant":"fp8","weights_gb":2592.951,"kv_cache_gb_per_8k":0.7718,"min_serve_gb":2723.37,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1467.708,"kv_cache_gb_per_8k":0.7718,"min_serve_gb":1541.865,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":2592.951,"kv_cache_gb_per_8k":0.7718,"min_serve_gb":2723.37,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":50.1,"score_norm":50.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-2.4t-a95b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":57.1,"score_norm":57.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.9,"score_norm":71.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-2.4t-a95b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.9,"score_norm":39.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-2.4t-a95b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":57.7,"score_norm":57.7,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.753333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":4.32,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"agents_last_exam","score_raw":27,"score_norm":null,"config":"pass metric (score metric 52.4)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":87.78,"score_norm":84.73,"config":"aime 2025, avg@3 (pass@3 = 93.33) — official qwen3.8-2.4t-a95b-fp8 checkpoint, blog notes an increased reasoning budget is required to reproduce [raw: aime25 @3 (avg)], measured in vllm's day-0 quantization-comparison table","trust":0,"source_type":"community","source_url":"https://vllm.ai/blog/2026-08-12-qwen3.8","measured_date":null,"observed_at":"2026-08-19","admissible":false,"excluded":null},{"benchmark":"androidbench","score_raw":75.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"automation_bench","score_raw":27.3,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"coworkbench","score_raw":74.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.2 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":56.6,"score_norm":null,"config":"1.1, best of claude code / mini-swe-agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":73.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":61.02,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.610195 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.54,"score_norm":96.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.935353535353535 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.6,"score_norm":95.21,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"healthbench","score_raw":60.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":42.45,"score_norm":59.49,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.424467099165894 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.6,"score_norm":61.18,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":56.2,"score_norm":79.71,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":82.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":53.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"longbench_v2","score_raw":66.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mls_bench_lite","score_raw":41,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mrcr_v2_8_needle","score_raw":92.9,"score_norm":null,"config":"256k","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"nl2repo_bench","score_raw":55.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"paperbench","score_raw":93,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"plawbench","score_raw":73.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"prbench_finance","score_raw":58.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"prbench_legal","score_raw":57.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"qwenqoderbench","score_raw":58.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"qwenreactbench","score_raw":1724,"score_norm":null,"config":"elo","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"qwensvgbench","score_raw":1713,"score_norm":null,"config":"elo","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"qwenswebench","score_raw":80.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":51.62,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.516203703703704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"skillsbench","score_raw":70.2,"score_norm":null,"config":"opencode harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":67.7,"score_norm":100,"config":"claude code harness; thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":49.07,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.490721649484536 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.02,"score_norm":90.61,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.820224719101124 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-2-4t-a95b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":86.6,"score_norm":96,"config":"claude code harness; thinking mode, reasoning_effort xhigh","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":86.6,"score_norm":96,"config":"1m context by default, 2.1, built-in tools) -- may not be identical to the raw ope, non-thinking support, reported under the \"qwen3.8-max\" product tier (vision input","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":72.5,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"widesearch","score_raw":81.9,"score_norm":null,"config":"qwen-agent harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"workspacebench","score_raw":67.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B","publisher":"Hugging Face"},{"url":"https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/","publisher":"NVIDIA"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-2.4t-a95b","publisher":"OpenRouter (listed 2026-08-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.8-27b","display_name":"Qwen3.8 27B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"27b","params_total_b":27.78,"params_active_b":27.78,"params_exact":27781427952,"params_source":"hf_safetensors","vram_est":{"bf16_gb":55.6,"int8_gb":29.4,"int4_gb":16.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.01,"expected_int8_gb":88.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.8-27B","openrouter_id":"qwen/qwen3.8-27b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":1000000,"release_date":"2026-08-14","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["qwen-3.5-27b","qwen-3.6-27b"],"superseded_by":null,"aliases":["qwen3.8-27b"],"intelligence":{"index":75.12,"inputs":[{"benchmark":"arena_elo","score_raw":1437.7,"score_norm":70.45,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":88.89,"score_norm":89.99,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.579, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":33.92,"score_norm":46.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.33920296570899 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":75.27,"score_norm":78.38,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":84.34,"score_norm":73.9,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.356, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":86,"score_norm":84.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.553, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":58.43,"score_norm":62.86,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=xhigh, runner=external, stderr=5.254, task=overall, updated=2026-08-19"}],"coverage":0.7936,"subset_bias":{"offset":-0.13,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.425,"completion_usd_per_m":2.55,"cache_read_usd_per_m":0.085,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-27b","as_of":"2026-10-06","blended_usd_per_m":0.9562499999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":55.56,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.875,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":55.563,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.878,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.8-27B/tree/main"},{"quant":"fp8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":16.668,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":18.205,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":12.96,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":21.89,"bw_limited_tok_s":13.13,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.86,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9.83,"bw_limited_tok_s":5.9,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":67.47,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":120.59,"bw_limited_tok_s":72.35,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":45.8,"score_norm":45.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-27b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":50.88,"score_norm":50.88,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":68.1,"score_norm":68.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-27b","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.7,"score_norm":33.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-27b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":52.02,"score_norm":52.02,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":77.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.773333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1437.7,"score_norm":70.45,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1439.7,"score_norm":71.35,"config":"3205 votes, arena entry qwen3.8-27b, leaderboard text-overall-style_control, rank 81, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1582.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1242.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1252.9,"score_norm":null,"config":"2446 votes, arena entry qwen3.8-27b-vision, leaderboard vision-overall-style_control, rank 37, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1591.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1594.8,"score_norm":null,"config":"2464 votes, arena entry qwen3.8-27b-code-webdev, code/overall, leaderboard webdev-overall-raw, rank 9, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":5.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.0542857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":52.3,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.5229550000000001 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.89,"score_norm":89.99,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.579, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.51,"score_norm":92.27,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.905050505050505 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.2,"score_norm":90.42,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.92,"score_norm":46.94,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.33920296570899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.8,"score_norm":42.35,"config":"judged by gpt-4o.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":75.27,"score_norm":78.38,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.34,"score_norm":73.9,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.356, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":76.3,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.763005780346821 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":44.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.446759259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":61.7,"score_norm":94.5,"config":"256k context, baseline models re-evaluated on the same refined task set., evaluated with the claude code harness, temp=1.0, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":86,"score_norm":84.44,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.553, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":48.04,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.480412371134021 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":58.43,"score_norm":62.86,"config":"edition=terminal-bench 2.1, reasoning_effort=xhigh, runner=external, stderr=5.254, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":79.78,"score_norm":87.98,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.797752808988764 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":73,"score_norm":80,"config":"2.1, no further hyperparameter footnote given for this row., row labeled \"(terminus)\" as the harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-27b","publisher":"OpenRouter (listed 2026-08-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.8-flash","display_name":"Qwen3.8 Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"frontier-flash","params_total_b":180,"params_active_b":null,"params_exact":179999981459,"params_source":"hf_safetensors","vram_est":{"bf16_gb":360,"int8_gb":191,"int4_gb":108,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":2.24,"expected_int8_gb":428,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"Qwen/Qwen3.8-Flash-Next","openrouter_id":"qwen/qwen3.8-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_large","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":1000000,"release_date":"2026-08-26","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["qwen-3.5-flash","qwen-3.6-flash","qwen-3.7-flash"],"superseded_by":null,"aliases":["qwen3.8-flash","qwen3.8-flash-next"],"intelligence":{"index":77.22,"inputs":[{"benchmark":"gpqa_diamond","score_raw":91.7,"score_norm":93.94,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","config":null},{"benchmark":"hle","score_raw":35.9,"score_norm":49.85,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","config":"judged by gpt-4o"},{"benchmark":"livebench","score_raw":76.2,"score_norm":80.44,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2026-06-25 release, overall"}],"coverage":0.3889,"subset_bias":{"offset":-0.25,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.47,"cache_read_usd_per_m":0.016,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-flash","as_of":"2026-10-06","blended_usd_per_m":0.22999999999999998,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":360,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":378.201,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":360,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":378.201,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next/tree/main"},{"quant":"fp8","weights_gb":190.8,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":200.541,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":108,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":113.601,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":190.8,"kv_cache_gb_per_8k":0.2013,"min_serve_gb":200.541,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":1.71,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":180,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":3.03,"bw_limited_tok_s":1.72,"decode_overhead_ms":1,"calibration_factor":0.5657702021996978,"calibration_anchors":1,"calibration_fallback":false,"calibration_basis":"shrunk","calibration_regime":"moe","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0.5,"calibration_anchor_median":0.42885101099848844,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"agents_last_exam","score_raw":24.3,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"androidworld","score_raw":84.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1637.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"coworkbench","score_raw":73.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":58.7,"score_norm":null,"config":"1.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"erqa","score_raw":72.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.7,"score_norm":93.94,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":35.9,"score_norm":49.85,"config":"judged by gpt-4o","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":81.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"jobbench","score_raw":55.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":76.2,"score_norm":80.44,"config":"livebench-2026-06-25 release, overall","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2026-06-25","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":91.9,"score_norm":null,"config":"6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"lvbench","score_raw":76.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"nl2repo_bench","score_raw":48.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"realworldqa","score_raw":88.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":81,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":62.5,"score_norm":95.83,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":73.5,"score_norm":null,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"vision2web","score_raw":64,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-flash","publisher":"OpenRouter (listed 2026-08-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.8-max-0902-2.4t","display_name":"Qwen3.8-Max-0902","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"frontier-max","params_total_b":2400,"params_active_b":95,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":4800,"int8_gb":2544,"int4_gb":1440,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":"Proprietary","is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-3.8-max-0902","qwen3.8-max-0902"],"intelligence":{"index":74.35,"inputs":[{"benchmark":"gpqa_diamond","score_raw":92,"score_norm":94.37,"trust":2,"source_url":"https://epoch.ai/models/qwen3-8-max-0902","config":null},{"benchmark":"hle","score_raw":43.1,"score_norm":60.44,"trust":2,"source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","config":null},{"benchmark":"terminal_bench","score_raw":39,"score_norm":40,"trust":1,"source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","config":"4.0"}],"coverage":0.3174,"subset_bias":{"offset":-1.53,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.25,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-max-0902","as_of":"2026-09-26","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":56,"score_norm":56,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":56,"score_norm":56,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_briefcase_elo","score_raw":1622,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.2,"score_norm":76.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.2,"score_norm":76.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-17","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45,"score_norm":45,"config":"reasoning, v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45.4,"score_norm":45.4,"config":"v4.3.2","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45.4,"score_norm":45.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-17","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":80.3,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":31.7,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_accuracy","score_raw":32,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_non_hallucination_rate","score_raw":71.2,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1639.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1670,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automationbench","score_raw":50.8,"score_norm":null,"config":"agentic saas workflow","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"automationbench_aa","score_raw":56,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"babyvision","score_raw":93.8,"score_norm":null,"config":"general visual reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"chess_puzzles","score_raw":40,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"claweval_mm","score_raw":80.2,"score_norm":null,"config":"pass@3","trust":0,"source_type":"community","source_url":"https://news.yahoo.co.jp/articles/ace27feb892fcbf0c6d1e31f576dd820dc90bdef","measured_date":"2026-09-03","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"coworkbench","score_raw":76.1,"score_norm":null,"config":"in-house, long-horizon office work","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":17.7,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":18,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":18,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":69.3,"score_norm":null,"config":"1.1, agentic coding","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"eci","score_raw":155,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"erqa","score_raw":78.3,"score_norm":null,"config":"embodied intelligence","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"finance_accounting_index","score_raw":46,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"frontiermath_tier_4","score_raw":34,"score_norm":null,"config":"2","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"frontiermath_tiers_1_3_v2","score_raw":66,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"gdp_pdf","score_raw":23,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2_elo","score_raw":1689,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.8,"score_norm":95.49,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92,"score_norm":94.37,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.1,"score_norm":60.44,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43,"score_norm":60.29,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"jobbench","score_raw":64,"score_norm":null,"config":"professional job tasks","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"mls_bench_lite","score_raw":50.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.datacamp.com/blog/qwen3-8-max","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":82.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.datacamp.com/blog/qwen3-8-max","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"nl2repo_bench","score_raw":64.9,"score_norm":null,"config":"repo-level generation","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"otis_mock_aime_2024_2025","score_raw":100,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"programbench_almost_solved","score_raw":28,"score_norm":null,"config":"black-box replication","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"qwenswebench_v2","score_raw":70,"score_norm":null,"config":"in-house","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.1,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/qwen/qwen3.8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":47,"score_norm":null,"config":"verified","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/qwen3-8-max-0902","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"swe_atlas_qna","score_raw":66.3,"score_norm":null,"config":"repo-level understanding","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"swe_marathon","score_raw":44.8,"score_norm":null,"config":"ultra-long-horizon swe","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":39,"score_norm":40,"config":"4.0","trust":1,"source_type":"aggregator","source_url":"https://www.intelligentliving.co/qwen3-8-max-update-smarter/","measured_date":"2026-09-17","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":29,"score_norm":28.24,"config":"agentic terminal coding","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":true,"excluded":null},{"benchmark":"toolathlon_verified","score_raw":73.3,"score_norm":null,"config":"real-world tool use","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"workarena_elo","score_raw":1468,"score_norm":null,"config":"expert evaluation, in-house","trust":1,"source_type":"vendor_claim","source_url":"https://cellcog.ai/blog/qwen3-8-max-0902/","measured_date":"2026-09-02","observed_at":"2026-09-30","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max","publisher":"Alibaba Cloud"},{"url":"https://www.qwencloud.com/models/qwen3.8-max-0902","publisher":"QwenCloud"}],"last_verified":"2026-10-06","last_enriched_at":"2026-09-30T06:08:22.927Z"},{"id":"qwen-3.8-max-2.4t","display_name":"Qwen3.8-Max","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"2.4t","params_total_b":2400,"params_active_b":95,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":4800,"int8_gb":2544,"int4_gb":1440,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.22,"expected_int8_gb":560,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":null,"openrouter_id":"qwen/qwen3.8-max-0902","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":1000000,"release_date":"2026-08-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen-3.8-max","qwen3.8-max"],"intelligence":{"index":82.98,"inputs":[{"benchmark":"arena_elo","score_raw":1481.6,"score_norm":90.02,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.69,"score_norm":96.75,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=1.221, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":43.05,"score_norm":60.37,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, qwen3.8 max"},{"benchmark":"livebench","score_raw":78.46,"score_norm":85.47,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: max"},{"benchmark":"mmlu_pro","score_raw":88.6,"score_norm":81,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.313, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":85.6,"score_norm":84,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, run: max, runner=external, stderr=1.572, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":67.42,"score_norm":73.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, run: max, runner=external, stderr=0.649, task=overall, updated=2026-08-19"}],"coverage":0.7936,"subset_bias":{"offset":0.4,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.25,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-max-0902","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":4800,"kv_cache_gb_per_8k":null,"min_serve_gb":5043.8,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":2544,"kv_cache_gb_per_8k":null,"min_serve_gb":2675,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1440,"kv_cache_gb_per_8k":null,"min_serve_gb":1515.8,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":2544,"kv_cache_gb_per_8k":null,"min_serve_gb":2675,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":58.4,"score_norm":58.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":58.4,"score_norm":58.4,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.8,"score_norm":71.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":58.1,"score_norm":58.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":58.08,"score_norm":58.08,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: max, source value 0.743333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":3.4,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1481.6,"score_norm":90.02,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1481,"score_norm":89.75,"config":"9955 votes, arena entry kinsley-mrp8, leaderboard text-overall-style_control, rank 19, run: max, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1490.6,"score_norm":94.03,"config":"arena entry qwen3.8-max, rank 8, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1617.6,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1301.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1301.9,"score_norm":null,"config":"6743 votes, arena entry thunbergia-alpha-9e3w, leaderboard vision-overall-style_control, rank 2, run: max, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1671.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1668.9,"score_norm":null,"config":"3212 votes, arena entry kinsley-5jmg, code/overall, leaderboard webdev-overall-raw, rank 3, raw (no style control), run: max, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: max, source value 0.2 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1369,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-08-31","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1317,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1284,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1331,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-08-31","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1339,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-08-22","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1262,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-08-30","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1353,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1334,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-08-29","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1299,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","measured_date":null,"observed_at":"2026-09-01","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":61.76,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: max, source value 0.617595 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.69,"score_norm":96.75,"config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=1.221, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.686,"score_norm":96.74,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.73,"score_norm":95.39,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: max, source value 0.927272727272727 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.73,"score_norm":95.39,"config":"artificial analysis eval, qwen3.8 max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.6,"score_norm":95.21,"config":"alibaba vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":43.05,"score_norm":60.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: max, source value 0.430491195551437 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.05,"score_norm":60.37,"config":"artificial analysis eval, qwen3.8 max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.6,"score_norm":61.18,"config":"alibaba vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":82.8,"score_norm":null,"config":"alibaba vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":78.46,"score_norm":85.47,"config":"livebench-2026-06-25 release, run: max","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":78.46,"score_norm":85.47,"config":"global average, livebench-2026-06-25 release, run qwen3.8-max","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"logicvista","score_raw":91.9,"score_norm":null,"config":"alibaba multimodal vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mathvision","score_raw":95.2,"score_norm":null,"config":"alibaba multimodal vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":19.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.194444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.6,"score_norm":81,"config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.313, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.602,"score_norm":81,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":82.31,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: max, source value 0.823121387283237 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":86.1,"score_norm":null,"config":"alibaba vendor table (computer use), verified","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/qwen3-8-max","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"paperbench","score_raw":93,"score_norm":null,"config":"alibaba vendor table","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.89,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, scicode, source value 0.528935185185185 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":67.7,"score_norm":100,"config":"alibaba vendor table; claude code harness","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":85.6,"score_norm":84,"config":"edition=swe-bench verified v1, run: max, runner=external, stderr=1.572, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":85.6,"score_norm":84,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":51.34,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.51340206185567 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.42,"score_norm":73.44,"config":"edition=terminal-bench 2.1, run: max, runner=external, stderr=0.649, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.416,"score_norm":73.43,"config":"overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":81.27,"score_norm":89.73,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: max, source value 0.812734082397004 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-8-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":81.27,"score_norm":89.73,"config":"artificial analysis eval, qwen3.8 max, terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":86.6,"score_norm":96,"config":"alibaba vendor table; coding benchmarks run in claude code harness, reasoning_effort xhigh","trust":0,"source_type":"community","source_url":"https://apidog.com/blog/qwen-3-8-benchmarks/","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwen.ai/blog?id=qwen3.8","publisher":"Alibaba"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max","publisher":"OpenRouter (listed 2026-08-03)"}],"last_verified":"2026-09-26","last_enriched_at":null},{"id":"qwen-3.8-max-prime","display_name":"Qwen3.8 Max Prime","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.8-max-prime","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.8-max-prime"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":4,"completion_usd_per_m":12,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-max-prime","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-max-prime","publisher":"OpenRouter (listed 2026-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.8-omni-flash","display_name":"Qwen3.8 Omni Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.8","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.8-omni-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-09-21","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["qwen3.8-omni-flash"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.47,"cache_read_usd_per_m":0.016,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.8-omni-flash","as_of":"2026-10-06","blended_usd_per_m":0.22999999999999998,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.8-omni-flash","publisher":"OpenRouter (listed 2026-09-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"reka-1-edge-2603","display_name":"Reka Edge","family":{"id":"reka","vendor":"rekaai"},"generation":"1","size_class":"frontier-edge","params_total_b":7.13,"params_active_b":7.13,"params_exact":7128509568,"params_source":"hf_safetensors","vram_est":{"bf16_gb":14.3,"int8_gb":7.6,"int4_gb":4.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"RekaAI/reka-edge-2603","openrouter_id":"rekaai/reka-edge","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","video"],"segment":"open_edge","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":16384,"release_date":"2026-03-20","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["reka-edge","reka-edge-2603"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.1,"cache_read_usd_per_m":null,"context_length":16384,"source":"openrouter","source_model_id":"rekaai/reka-edge","as_of":"2026-10-06","blended_usd_per_m":0.1,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":14.26,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":16.334,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":14.257,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":16.331,"method":"vendor","source_url":"https://huggingface.co/RekaAI/reka-edge-2603/tree/main"},{"quant":"fp8","weights_gb":7.558,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.632,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.278,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":6.352,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":7.558,"kv_cache_gb_per_8k":1.0737,"min_serve_gb":9.632,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":48.67,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":7.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":85.27,"bw_limited_tok_s":51.16,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":22.46,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":7.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":38.29,"bw_limited_tok_s":22.97,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":219.91,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":7.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":469.85,"bw_limited_tok_s":281.91,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"mlvu","score_raw":74.3,"score_norm":null,"config":"video understanding","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmvu","score_raw":71.68,"score_norm":null,"config":"multimodal video understanding","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mobile_actions","score_raw":88.4,"score_norm":null,"config":"tool use","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"refcoco_a","score_raw":93.13,"score_norm":null,"config":"object detection","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"refcoco_b","score_raw":86.7,"score_norm":null,"config":"object detection","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"videohallucer","score_raw":59.57,"score_norm":null,"config":"hallucination","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"vqa_v2","score_raw":88.4,"score_norm":null,"config":"visual question answering","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-edge-2603","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#rekaai/reka-edge","publisher":"OpenRouter (listed 2026-03-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"reka-3-flash","display_name":"Reka Flash 3","family":{"id":"reka","vendor":"rekaai"},"generation":"3","size_class":"frontier-flash","params_total_b":20.91,"params_active_b":20.91,"params_exact":20905482240,"params_source":"hf_safetensors","vram_est":{"bf16_gb":41.8,"int8_gb":22.2,"int4_gb":12.5,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.79,"expected_int8_gb":39.7,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"RekaAI/reka-flash-3","openrouter_id":"rekaai/reka-flash-3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2025-03-12","release_status":"released","knowledge_cutoff":"2025-01-31","status":"current","supersedes":[],"superseded_by":null,"aliases":["reka-flash-3"],"intelligence":{"index":50.38,"inputs":[{"benchmark":"aime","score_raw":70,"score_norm":62.5,"trust":1,"source_url":"https://huggingface.co/RekaAI/reka-flash-3","config":"aime 2024, cons@64"},{"benchmark":"gpqa_diamond","score_raw":61.1,"score_norm":50.85,"trust":1,"source_url":"https://huggingface.co/RekaAI/reka-flash-3","config":"subset not stated, vendor chart labelled gpqa only"},{"benchmark":"mmlu_pro","score_raw":65,"score_norm":41.67,"trust":1,"source_url":"https://huggingface.co/RekaAI/reka-flash-3","config":"general knowledge panel, vendor model-card chart"}],"coverage":0.3572,"subset_bias":{"offset":15.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.2,"cache_read_usd_per_m":null,"context_length":65536,"source":"openrouter","source_model_id":"rekaai/reka-flash-3","as_of":"2026-10-06","blended_usd_per_m":0.125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":41.82,"kv_cache_gb_per_8k":1.1073,"min_serve_gb":45.018,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":41.811,"kv_cache_gb_per_8k":1.1073,"min_serve_gb":45.009,"method":"vendor","source_url":"https://huggingface.co/RekaAI/reka-flash-3/tree/main"},{"quant":"fp8","weights_gb":22.165,"kv_cache_gb_per_8k":1.1073,"min_serve_gb":24.381,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":12.546,"kv_cache_gb_per_8k":1.1073,"min_serve_gb":14.653,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":22.165,"kv_cache_gb_per_8k":1.1073,"min_serve_gb":24.381,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":17.15,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":20.91,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":29.08,"bw_limited_tok_s":17.45,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":7.77,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":20.91,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":13.06,"bw_limited_tok_s":7.83,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":87.7,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":20.91,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":160.21,"bw_limited_tok_s":96.13,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.4,"score_norm":6.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#rekaai/reka-flash-3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":50,"score_norm":37.5,"config":"aime 2025, cons@64","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-flash-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":70,"score_norm":62.5,"config":"aime 2024, cons@64","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-flash-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.1,"score_norm":50.85,"config":"subset not stated, vendor chart labelled gpqa only","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-flash-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":65,"score_norm":41.67,"config":"general knowledge panel, vendor model-card chart","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/RekaAI/reka-flash-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#rekaai/reka-flash-3","publisher":"OpenRouter (listed 2025-03-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"relace-apply-3","display_name":"Relace Apply 3","family":{"id":"relace","vendor":"relace"},"generation":"3","size_class":"frontier-apply","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"relace/relace-apply-3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":256000,"release_date":"2025-09-26","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.85,"completion_usd_per_m":1.25,"cache_read_usd_per_m":null,"context_length":256000,"source":"openrouter","source_model_id":"relace/relace-apply-3","as_of":"2026-10-06","blended_usd_per_m":0.95,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#relace/relace-apply-3","publisher":"OpenRouter (listed 2025-09-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"relace-search-1","display_name":"Relace Search","family":{"id":"relace","vendor":"relace"},"generation":"1","size_class":"frontier-search","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"relace/relace-search","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":256000,"release_date":"2025-12-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["relace-search"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":3,"cache_read_usd_per_m":null,"context_length":256000,"source":"openrouter","source_model_id":"relace/relace-search","as_of":"2026-10-06","blended_usd_per_m":1.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#relace/relace-search","publisher":"OpenRouter (listed 2025-12-08)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ring-2.6-1t","display_name":"ring-2.6-1t","family":{"id":"ring","vendor":"inclusionai"},"generation":"2.6","size_class":"frontier","params_total_b":1000,"params_active_b":1000,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":2000,"int8_gb":1060,"int4_gb":600,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ring-2.6-1t","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-05-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":64.36,"inputs":[{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":21.59,"score_norm":28.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, source value 0.215940685820204 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":92.4,"score_norm":96.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.923976608187135 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":43.07,"score_norm":44.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.430711610486891 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-1.35,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":20.1,"score_norm":20.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":20.1,"score_norm":20.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ring-2.6-1t","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":42.8,"score_norm":42.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ring-2.6-1t","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.66,"score_norm":31.66,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.7,"score_norm":31.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ring-2.6-1t","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":67.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, source value 0.673333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":2.1,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.021013617465460616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, source value 0.0371428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":21.12,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, source value 0.21116500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.66,"score_norm":85.44,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, source value 0.856565656565657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":21.59,"score_norm":28.81,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, source value 0.215940685820204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":44.56,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, source value 0.445578231292517 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, scicode, source value 0.423611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":92.4,"score_norm":96.53,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.923976608187135 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":17.94,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.179381443298969 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.79,"score_norm":27.99,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.287878787878788 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.07,"score_norm":44.79,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, source value 0.430711610486891 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ring-2-6-1t","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ring-2.6-1t","publisher":"OpenRouter (listed 2026-05-08)"}],"last_verified":"2026-08-24","last_enriched_at":null},{"id":"seed-1.6","display_name":"Seed 1.6","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"1.6","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-1.6","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2025-12-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":2,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-1.6","as_of":"2026-10-06","blended_usd_per_m":0.6875,"endpoint_expires_at":"2026-11-11"},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-1.6","publisher":"OpenRouter (listed 2025-12-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"seed-1.6-flash","display_name":"Seed 1.6 Flash","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"1.6","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-1.6-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2025-12-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.075,"completion_usd_per_m":0.3,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-1.6-flash","as_of":"2026-10-06","blended_usd_per_m":0.13124999999999998,"endpoint_expires_at":"2026-11-11"},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-1.6-flash","publisher":"OpenRouter (listed 2025-12-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"seed-2-1-turbo","display_name":"Seed 2.1 Turbo","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"2","size_class":"frontier-turbo","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-2-1-turbo","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-08-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":2.5,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-2-1-turbo","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"agent_startup_bench","score_raw":54,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":61.3,"score_norm":62.56,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/glm-5.2/seed-2.1-turbo","measured_date":"2026-09-18","observed_at":"2026-10-02","admissible":true,"excluded":null},{"benchmark":"babyvision","score_raw":62.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"beyondaime","score_raw":88,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":84.9,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/qwen3.7-max/seed-2.1-turbo","measured_date":"2026-09-28","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"charxiv_rq","score_raw":83.6,"score_norm":null,"config":"w. tool, with tool","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"charxiv_rq","score_raw":82.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"creativework","score_raw":34.5,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":23,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/seed-2.1-turbo","measured_date":"2026-09-27","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"erqa","score_raw":71.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"gameworld","score_raw":25.9,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"gdpval","score_raw":82.7,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/glm-5.2/seed-2.1-turbo","measured_date":"2026-09-18","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval","score_raw":82.7,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":54.6,"score_norm":77.35,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/seed-2.1-turbo","measured_date":"2026-09-27","observed_at":"2026-10-02","admissible":true,"excluded":null},{"benchmark":"kina","score_raw":46.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":92.7,"score_norm":null,"config":"w. tool, with tool","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"math_vision","score_raw":90.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":80.3,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/qwen3.7-max/seed-2.1-turbo","measured_date":"2026-09-28","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"minerva","score_raw":65.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"mmlongbench_128k","score_raw":76.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":82.2,"score_norm":null,"config":"w. tool, with tool","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.1,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"mobileworld","score_raw":70,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"nl2repo_bench","score_raw":43.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"officeqa_pro","score_raw":62.8,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":76.4,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/qwen3.7-max/seed-2.1-turbo","measured_date":"2026-09-28","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":76.4,"score_norm":null,"config":null,"trust":0,"source_type":"community","source_url":"https://howaiworks.ai/models/doubao-pro","measured_date":"2026-06-23","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"ovobench","score_raw":79.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":49.4,"score_norm":null,"config":"0/0/49.4 as published","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":57.8,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/seed-2.1-turbo","measured_date":"2026-09-27","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":67.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"swe_atlas","score_raw":30.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":57,"score_norm":86.67,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/seed-2.1-turbo","measured_date":"2026-09-27","observed_at":"2026-10-02","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.6,"score_norm":73.65,"config":"2.1","trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/qwen3.7-max/seed-2.1-turbo","measured_date":"2026-09-28","observed_at":"2026-10-02","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.6,"score_norm":73.65,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.6,"score_norm":73.65,"config":"2.1","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/terminalbench21","measured_date":"2026-09-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"tomato","score_raw":56.8,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"toolathlon","score_raw":49.1,"score_norm":null,"config":null,"trust":1,"source_type":"aggregator","source_url":"https://anotherwrapper.com/tools/llm-pricing/qwen3.7-max/seed-2.1-turbo","measured_date":"2026-09-28","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"video_mme","score_raw":89,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"videomme","score_raw":89,"score_norm":null,"config":"long video understanding","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"videosimpleqa","score_raw":71.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"workspace_bench","score_raw":54.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"worldvqa","score_raw":48.6,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"xdailybench","score_raw":56.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"zerobench","score_raw":20,"score_norm":null,"config":"w. tool, with tool","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"zerobench","score_raw":11,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2_1","measured_date":"2026-06-24","observed_at":"2026-10-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-2-1-turbo","publisher":"OpenRouter (listed 2026-08-12)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-02T06:07:19.438Z"},{"id":"seed-2.0-code","display_name":"Seed-2.0-Code","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"2.0","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-2.0-code","is_moe":false,"variant_axis":"code","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-08-12","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":3,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-2.0-code","as_of":"2026-10-06","blended_usd_per_m":1.125,"endpoint_expires_at":"2026-11-11"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"codeforces_elo","score_raw":3020,"score_norm":100,"config":null,"trust":0,"source_type":"community","source_url":"https://www.digitalapplied.com/blog/bytedance-seed-2-doubao-ai-model-benchmarks-guide","measured_date":"2026-02-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":87.8,"score_norm":null,"config":"v6","trust":1,"source_type":"aggregator","source_url":"https://www.llmreference.com/model/bytedance-doubao-seed-2-0-code","measured_date":"2026-04-27","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":87.8,"score_norm":null,"config":"6","trust":0,"source_type":"community","source_url":"https://www.digitalapplied.com/blog/bytedance-seed-2-doubao-ai-model-benchmarks-guide","measured_date":"2026-02-16","observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.5,"score_norm":73.89,"config":"shares pro swe-bench score per launch materials (resolved)","trust":1,"source_type":"aggregator","source_url":"https://www.llmreference.com/model/bytedance-doubao-seed-2-0-code","measured_date":"2026-06-07","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.5,"score_norm":73.89,"config":"% resolved","trust":1,"source_type":"aggregator","source_url":"https://evals.report/labs/bytedance","measured_date":"2026-02-14","observed_at":"2026-09-28","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.5,"score_norm":73.89,"config":null,"trust":0,"source_type":"community","source_url":"https://www.digitalapplied.com/blog/bytedance-seed-2-doubao-ai-model-benchmarks-guide","measured_date":"2026-02-16","observed_at":"2026-09-28","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-2.0-code","publisher":"OpenRouter (listed 2026-08-12)"}],"last_verified":"2026-10-06","last_enriched_at":"2026-10-02T06:06:48.175Z"},{"id":"seed-2.0-lite","display_name":"Seed-2.0-Lite","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"2.0","size_class":"frontier-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-2.0-lite","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-03-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":63.62,"inputs":[{"benchmark":"gpqa_diamond","score_raw":88.4,"score_norm":89.3,"trust":1,"source_url":"https://seed.bytedance.com/en/seed2","config":"seed2.0 lite 0428 snapshot"},{"benchmark":"hle","score_raw":25.7,"score_norm":34.85,"trust":1,"source_url":"https://seed.bytedance.com/en/seed2","config":"no tool, seed2.0 lite 0428 snapshot, text only"},{"benchmark":"terminal_bench","score_raw":43.3,"score_norm":45.06,"trust":1,"source_url":"https://seed.bytedance.com/en/seed2","config":"seed2.0 lite 0428 snapshot"}],"coverage":0.3174,"subset_bias":{"offset":-4.48,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":2,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-2.0-lite","as_of":"2026-10-06","blended_usd_per_m":0.6875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"beyondaime","score_raw":79,"score_norm":null,"config":"seed2.0 lite 0428 snapshot","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.4,"score_norm":89.3,"config":"seed2.0 lite 0428 snapshot","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":25.7,"score_norm":34.85,"config":"no tool, seed2.0 lite 0428 snapshot, text only","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":46.6,"score_norm":69.33,"config":"seed2.0 lite 0428 snapshot","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_multilingual","score_raw":66.6,"score_norm":null,"config":"seed2.0 lite 0428 snapshot","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.3,"score_norm":45.06,"config":"seed2.0 lite 0428 snapshot","trust":1,"source_type":"vendor_claim","source_url":"https://seed.bytedance.com/en/seed2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-2.0-lite","publisher":"OpenRouter (listed 2026-03-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"seed-2.0-mini","display_name":"Seed-2.0-Mini","family":{"id":"seed","vendor":"bytedance-seed"},"generation":"2.0","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"bytedance-seed/seed-2.0-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-02-26","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":53.36,"inputs":[{"benchmark":"aime","score_raw":87,"score_norm":83.75,"trust":2,"source_url":"https://arxiv.org/pdf/2607.00248","config":"2025, seed2.0 mini column, seed2.0 model card table 4 (efficient models)"},{"benchmark":"arc_agi_2","score_raw":2.3,"score_norm":0,"trust":2,"source_url":"https://arxiv.org/pdf/2607.00248","config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)"},{"benchmark":"gpqa_diamond","score_raw":79,"score_norm":76.06,"trust":2,"source_url":"https://arxiv.org/pdf/2607.00248","config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)"},{"benchmark":"hle","score_raw":13.3,"score_norm":16.62,"trust":2,"source_url":"https://arxiv.org/pdf/2607.00248","config":"no tools, seed2.0 mini column, seed2.0 model card table 4 (efficient models), text only"},{"benchmark":"mmlu_pro","score_raw":83.6,"score_norm":72.67,"trust":2,"source_url":"https://arxiv.org/pdf/2607.00248","config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)"}],"coverage":0.5475,"subset_bias":{"offset":-1.53,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.4,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"bytedance-seed/seed-2.0-mini","as_of":"2026-10-06","blended_usd_per_m":0.17500000000000002,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":86.7,"score_norm":83.38,"config":"2026, seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87,"score_norm":83.75,"config":"2025, seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":87,"score_norm":83.75,"config":"aime 2025","trust":2,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/bytedance-seed/seed-2.0-mini/","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":2.3,"score_norm":0,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"codeforces_elo","score_raw":1644,"score_norm":49.65,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79,"score_norm":76.06,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":13.3,"score_norm":16.62,"config":"no tools, seed2.0 mini column, seed2.0 model card table 4 (efficient models), text only","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":64.1,"score_norm":null,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models), v6","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.6,"score_norm":72.67,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"supergpqa","score_raw":61.6,"score_norm":null,"config":"seed2.0 mini column, seed2.0 model card table 4 (efficient models)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2607.00248","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"swe_bench","score_raw":67.9,"score_norm":null,"config":"source labels it only 'swe-bench', split unspecified","trust":2,"source_type":"aggregator","source_url":"https://developer.puter.com/ai/bytedance-seed/seed-2.0-mini/","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#bytedance-seed/seed-2.0-mini","publisher":"OpenRouter (listed 2026-02-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"smollm-2-1.7b","display_name":"SmolLM2-1.7B","family":{"id":"smollm","vendor":"Hugging Face"},"generation":"2","size_class":"1.7b","params_total_b":1.71,"params_active_b":1.71,"params_exact":1711376384,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3.4,"int8_gb":1.8,"int4_gb":1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"HuggingFaceTB/SmolLM2-1.7B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2024-11-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["smollm2-1.7b"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":3.42,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":6.031,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":3.423,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":6.034,"method":"vendor","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/tree/main"},{"quant":"fp8","weights_gb":1.813,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":4.424,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.026,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":3.637,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1.813,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":4.424,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":175.82,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":1.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":355.56,"bw_limited_tok_s":213.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":87.42,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":1.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":159.65,"bw_limited_tok_s":95.79,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":540.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":1.71,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1959.06,"bw_limited_tok_s":1175.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc","score_raw":60.5,"score_norm":null,"config":"average, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"commonsenseqa","score_raw":43.6,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm8k","score_raw":31,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":68.7,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":19.4,"score_norm":0,"config":"base model, mcf","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"openbookqa","score_raw":42.2,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"piqa","score_raw":77.6,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"triviaqa","score_raw":36.7,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":59.4,"score_norm":null,"config":"base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"smollm-2-135m","display_name":"SmolLM2-135M","family":{"id":"smollm","vendor":"Hugging Face"},"generation":"2","size_class":"135m","params_total_b":0.13,"params_active_b":0.13,"params_exact":134515008,"params_source":"hf_safetensors","vram_est":{"bf16_gb":0.3,"int8_gb":0.1,"int4_gb":0.1,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"HuggingFaceTB/SmolLM2-135M","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2024-11-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["smollm2-135m"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":0.26,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":1.449,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":0.269,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":1.458,"method":"vendor","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M/tree/main"},{"quant":"fp8","weights_gb":0.138,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":1.327,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.078,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":1.267,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.138,"kv_cache_gb_per_8k":0.1887,"min_serve_gb":1.327,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":737.27,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":4676.92,"bw_limited_tok_s":2806.15,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":557.52,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":2100,"bw_limited_tok_s":1260,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":939.25,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.13,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":25769.23,"bw_limited_tok_s":15461.54,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc","score_raw":43.9,"score_norm":null,"config":"0-shot, average","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"commonsenseqa","score_raw":33.9,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm8k","score_raw":1.4,"score_norm":null,"config":"5-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":42.1,"score_norm":null,"config":"0-shot, lighteval","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":31.5,"score_norm":null,"config":"0-shot, cloze","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":10.85,"score_norm":0,"config":"tiger-lab mmlu-pro leaderboard entry","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"openbookqa","score_raw":34.6,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"piqa","score_raw":68.4,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"triviaqa","score_raw":4.1,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":51.3,"score_norm":null,"config":"0-shot","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"smollm-2-360m","display_name":"SmolLM2-360M","family":{"id":"smollm","vendor":"Hugging Face"},"generation":"2","size_class":"360m","params_total_b":0.36,"params_active_b":0.36,"params_exact":361821120,"params_source":"hf_safetensors","vram_est":{"bf16_gb":0.7,"int8_gb":0.4,"int4_gb":0.2,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"HuggingFaceTB/SmolLM2-360M","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":8192,"release_date":"2024-11-02","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["smollm2-360m"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":0.72,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":2.056,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":0.724,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":2.06,"method":"vendor","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M/tree/main"},{"quant":"fp8","weights_gb":0.382,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":1.718,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":0.216,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":1.551,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":0.382,"kv_cache_gb_per_8k":0.3355,"min_serve_gb":1.718,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":503.31,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":0.36,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1688.89,"bw_limited_tok_s":1013.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":312.71,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":0.36,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":758.33,"bw_limited_tok_s":455,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":848.1,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":0.36,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9305.56,"bw_limited_tok_s":5583.33,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"arc","score_raw":53,"score_norm":null,"config":"average, base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"commonsenseqa","score_raw":38,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm8k","score_raw":3.2,"score_norm":null,"config":"5-shot, base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":54.5,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":35.8,"score_norm":null,"config":"base pre-trained model column, cloze","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"openbookqa","score_raw":37.4,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"piqa","score_raw":71.7,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"triviaqa","score_raw":16.9,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":52.5,"score_norm":null,"config":"base pre-trained model column","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM2-360M","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/HuggingFaceTB/SmolLM2-135M","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"smollm-3-3b","display_name":"SmolLM3-3B","family":{"id":"smollm","vendor":"Hugging Face"},"generation":"3","size_class":"3b","params_total_b":3.08,"params_active_b":3.08,"params_exact":3075098624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":6.2,"int8_gb":3.3,"int4_gb":1.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":3.37,"expected_int8_gb":11,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"HuggingFaceTB/SmolLM3-3B","openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_edge","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-07-08","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["smollm3-3b"],"intelligence":{"index":15.09,"inputs":[{"benchmark":"aime","score_raw":36.7,"score_norm":20.88,"trust":1,"source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","config":"aime 2025, extended thinking"},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"trust":1,"source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","config":"extended thinking"},{"benchmark":"mmlu_pro","score_raw":32.7,"score_norm":0,"trust":1,"source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","config":"0-shot, base model, lighteval, mmlu pro mcf"}],"coverage":0.3572,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":6.16,"kv_cache_gb_per_8k":0.604,"min_serve_gb":7.764,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":6.15,"kv_cache_gb_per_8k":0.604,"min_serve_gb":7.754,"method":"vendor","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B/tree/main"},{"quant":"fp8","weights_gb":3.265,"kv_cache_gb_per_8k":0.604,"min_serve_gb":4.869,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":1.848,"kv_cache_gb_per_8k":0.604,"min_serve_gb":3.452,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":3.265,"kv_cache_gb_per_8k":0.604,"min_serve_gb":4.869,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":105.9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":608,"params_active_b":3.08,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":197.4,"bw_limited_tok_s":118.44,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":50.5,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":273,"params_active_b":3.08,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":88.64,"bw_limited_tok_s":53.18,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":394.89,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_overhead_v2","inputs":{"memory_bw_gbps":3350,"params_active_b":3.08,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1087.66,"bw_limited_tok_s":652.6,"decode_overhead_ms":1,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"calibration_basis":"prior","calibration_regime":"prior","calibration_prior":0.6,"calibration_prior_weight":2,"calibration_anchor_weight":0,"calibration_anchor_median":null,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aime","score_raw":36.7,"score_norm":20.88,"config":"aime 2025, extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"bfcl","score_raw":88.8,"score_norm":null,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"global_mmlu","score_raw":64.1,"score_norm":null,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.7,"score_norm":23.52,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":67.63,"score_norm":null,"config":"5-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gsm_plus","score_raw":83.4,"score_norm":null,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hellaswag","score_raw":76.15,"score_norm":null,"config":"0-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":71.2,"score_norm":null,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":30,"score_norm":null,"config":"extended thinking, v4","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":46.1,"score_norm":null,"config":"4-shot, base model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mixeval_hard","score_raw":30.8,"score_norm":null,"config":"extended thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":32.7,"score_norm":0,"config":"0-shot, base model, lighteval, mmlu pro mcf","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/HuggingFaceTB/SmolLM3-3B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/smollm3","publisher":"Hugging Face"}],"last_verified":"2026-09-05","last_enriched_at":null},{"id":"solar-mini-4","display_name":"Solar Mini 4","family":{"id":"solar","vendor":"upstage"},"generation":"4","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"upstage/solar-mini4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":524288,"release_date":"2026-09-23","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["solar-mini4"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.2,"cache_read_usd_per_m":0.005,"context_length":524288,"source":"openrouter","source_model_id":"upstage/solar-mini4","as_of":"2026-10-06","blended_usd_per_m":0.08750000000000001,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":24.1,"score_norm":24.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-mini4","measured_date":null,"observed_at":"2026-10-01","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#upstage/solar-mini4","publisher":"OpenRouter (listed 2026-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"solar-pro-4","display_name":"Solar Pro 4","family":{"id":"solar","vendor":"upstage"},"generation":"4","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"upstage/solar-pro4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":524288,"release_date":"2026-08-10","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":["solar-pro-3"],"superseded_by":null,"aliases":["solar-pro4"],"intelligence":{"index":75.19,"inputs":[{"benchmark":"aime","score_raw":95.3,"score_norm":94.12,"trust":1,"source_url":"https://www.upstage.ai/blog/en/solar-pro-4","config":"aime 2026, in-house evaluation"},{"benchmark":"arena_elo","score_raw":1386.2,"score_norm":47.5,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89,"score_norm":90.14,"trust":1,"source_url":"https://www.upstage.ai/blog/en/solar-pro-4","config":null},{"benchmark":"mmlu_pro","score_raw":86.3,"score_norm":77.17,"trust":1,"source_url":"https://www.upstage.ai/blog/en/solar-pro-4","config":"in-house internal evaluation environment"},{"benchmark":"swe_bench_verified","score_raw":70.6,"score_norm":67.33,"trust":1,"source_url":"https://www.upstage.ai/blog/en/solar-pro-4","config":"in-house evaluation, openhands harness"},{"benchmark":"terminal_bench","score_raw":57,"score_norm":61.18,"trust":1,"source_url":"https://www.upstage.ai/blog/en/solar-pro-4","config":"v2.1"}],"coverage":0.6428,"subset_bias":{"offset":5.92,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.09,"completion_usd_per_m":0.36,"cache_read_usd_per_m":0.018,"context_length":524288,"source":"openrouter","source_model_id":"upstage/solar-pro4","as_of":"2026-10-06","blended_usd_per_m":0.1575,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":52.7,"score_norm":52.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":28.2,"score_norm":28.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.3,"score_norm":94.12,"config":"aime 2026, in-house evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":18.7,"score_norm":null,"config":"in-house evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1386.2,"score_norm":47.5,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1370.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":49.2,"score_norm":null,"config":"in-house evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1194,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1188,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1187,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1183,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1156,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1098,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1183,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89,"score_norm":90.14,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":87.8,"score_norm":null,"config":"in-house evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":61.4,"score_norm":null,"config":"in-house evaluation","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.3,"score_norm":77.17,"config":"in-house internal evaluation environment","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70.6,"score_norm":67.33,"config":"in-house evaluation, openhands harness","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau3_bench_banking","score_raw":23,"score_norm":null,"config":"tau3/banking","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":57,"score_norm":61.18,"config":"v2.1","trust":1,"source_type":"vendor_claim","source_url":"https://www.upstage.ai/blog/en/solar-pro-4","measured_date":"2026-08-11","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#upstage/solar-pro4","publisher":"OpenRouter (listed 2026-08-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"sonar-1","display_name":"Sonar","family":{"id":"sonar","vendor":"perplexity"},"generation":"1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perplexity/sonar","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":127072,"release_date":"2025-01-27","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["sonar"],"intelligence":{"index":28.67,"inputs":[{"benchmark":"gpqa_diamond","score_raw":47.1,"score_norm":31.13,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=sonar","config":"artificial analysis harness, gpqa diamond"},{"benchmark":"hle","score_raw":4.9,"score_norm":4.26,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=sonar","config":"artificial analysis harness, humanity's last exam"},{"benchmark":"mmlu_pro","score_raw":68.9,"score_norm":48.17,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=sonar","config":"artificial analysis harness, mmlu-pro"}],"coverage":0.3889,"subset_bias":null},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":1,"cache_read_usd_per_m":null,"context_length":127072,"source":"openrouter","source_model_id":"perplexity/sonar","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.7,"score_norm":7.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#perplexity/sonar","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":47.1,"score_norm":31.13,"config":"artificial analysis harness, gpqa diamond","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond?models=sonar","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.9,"score_norm":4.26,"config":"artificial analysis harness, humanity's last exam","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam?models=sonar","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":68.9,"score_norm":48.17,"config":"artificial analysis harness, mmlu-pro","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/mmlu-pro?models=sonar","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perplexity/sonar","publisher":"OpenRouter (listed 2025-01-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"sonar-1-pro","display_name":"Sonar Pro","family":{"id":"sonar","vendor":"perplexity"},"generation":"1","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perplexity/sonar-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-03-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["sonar-pro"],"intelligence":{"index":38.88,"inputs":[{"benchmark":"gpqa_diamond","score_raw":57.78,"score_norm":46.17,"trust":2,"source_url":"https://artificialanalysis.ai/models/sonar-pro","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":6.55,"score_norm":6.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/sonar-pro","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":75.52,"score_norm":59.2,"trust":2,"source_url":"https://artificialanalysis.ai/models/sonar-pro","config":"artificial analysis independent evaluation"}],"coverage":0.3889,"subset_bias":{"offset":3.96,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":null,"context_length":200000,"source":"openrouter","source_model_id":"perplexity/sonar-pro","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.6,"score_norm":7.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#perplexity/sonar-pro","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.78,"score_norm":46.17,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/sonar-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.55,"score_norm":6.69,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/sonar-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":27.51,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/sonar-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":74.47,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/sonar-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":75.52,"score_norm":59.2,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/sonar-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perplexity/sonar-pro","publisher":"OpenRouter (listed 2025-03-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"sonar-1-pro-reasoning","display_name":"Sonar Reasoning Pro","family":{"id":"sonar","vendor":"perplexity"},"generation":"1","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perplexity/sonar-reasoning-pro","is_moe":false,"variant_axis":"reasoning","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2025-03-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["sonar-reasoning-pro"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":8,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"perplexity/sonar-reasoning-pro","as_of":"2026-10-06","blended_usd_per_m":3.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":11.8,"score_norm":11.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#perplexity/sonar-reasoning-pro","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":79,"score_norm":73.75,"config":"aime 2024), artificial analysis independent run (legacy aa 'aime' field, read from the 2025-07-07 wayback capture of aa's now-deprecated model page","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20250707232933/https://artificialanalysis.ai/models/sonar-reasoning-pro","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"frames","score_raw":44.4,"score_norm":null,"config":"open deep search paper table 1 baseline run, with web access","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2503.20201v1","measured_date":"2025-03-26","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":95.7,"score_norm":null,"config":"2025-07-07 wayback capture of aa's deprecated model page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20250707232933/https://artificialanalysis.ai/models/sonar-reasoning-pro","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":85.8,"score_norm":null,"config":"open deep search paper table 1 baseline run, with web access","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2503.20201v1","measured_date":"2025-03-26","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perplexity/sonar-reasoning-pro","publisher":"OpenRouter (listed 2025-03-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"sonar-deep-research-1","display_name":"Sonar Deep Research","family":{"id":"sonar","vendor":"perplexity"},"generation":"1","size_class":"frontier-deep-research","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perplexity/sonar-deep-research","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2025-03-07","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["sonar-deep-research"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":8,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"perplexity/sonar-deep-research","as_of":"2026-10-06","blended_usd_per_m":3.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"hle","score_raw":21.1,"score_norm":28.09,"config":"accuracy, figure published for perplexity deep research mode on the 2025-02-14 product launch page — perplexity does not restate it for the sonar-deep-research api model","trust":1,"source_type":"vendor_claim","source_url":"https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research","measured_date":"2025-02-14","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"simpleqa","score_raw":93.9,"score_norm":null,"config":"accuracy, figure published for perplexity deep research mode on the 2025-02-14 product launch page","trust":1,"source_type":"vendor_claim","source_url":"https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research","measured_date":"2025-02-14","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perplexity/sonar-deep-research","publisher":"OpenRouter (listed 2025-03-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"sonar-pro-search-1","display_name":"Sonar Pro Search","family":{"id":"sonar","vendor":"perplexity"},"generation":"1","size_class":"frontier-pro-search","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"perplexity/sonar-pro-search","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-10-30","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["sonar-pro-search"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":null,"context_length":200000,"source":"openrouter","source_model_id":"perplexity/sonar-pro-search","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#perplexity/sonar-pro-search","publisher":"OpenRouter (listed 2025-10-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"step-3.5-flash","display_name":"Step 3.5 Flash","family":{"id":"step","vendor":"stepfun"},"generation":"3.5","size_class":"frontier-flash","params_total_b":199.38,"params_active_b":null,"params_exact":199384301376,"params_source":"hf_safetensors","vram_est":{"bf16_gb":399,"int8_gb":211,"int4_gb":120,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.45,"expected_int8_gb":307,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"stepfun-ai/Step-3.5-Flash","openrouter_id":"stepfun/step-3.5-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-01-29","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":68.37,"inputs":[{"benchmark":"aime","score_raw":97.3,"score_norm":96.63,"trust":2,"source_url":"https://arxiv.org/pdf/2602.10604","config":"aime 2025, no tools, repeat@64 [raw: aime 2025], table 5 step 3.5 flash 'vanilla' column (non-pacore)"},{"benchmark":"arena_elo","score_raw":1393.3,"score_norm":50.66,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":83.13,"score_norm":81.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: step 3.5 flash, source value 0.831313131313131 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":24.47,"score_norm":33.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-5-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: step 3.5 flash 2603, source value 0.244670991658943 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"trust":2,"source_url":"https://arxiv.org/abs/2602.10604","config":null},{"benchmark":"swe_bench_verified","score_raw":74.4,"score_norm":71.56,"trust":2,"source_url":"https://arxiv.org/abs/2602.10604","config":null},{"benchmark":"tau_bench","score_raw":94.44,"score_norm":99.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: step 3.5 flash, source value 0.944444444444444 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-5-flash","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: step 3.5 flash 2603, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.3,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"stepfun/step-3.5-flash","as_of":"2026-10-06","blended_usd_per_m":0.15000000000000002,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":398.76,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":422.523,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":398.769,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":422.533,"method":"vendor","source_url":"https://huggingface.co/stepfun-ai/Step-3.5-Flash/tree/main"},{"quant":"fp8","weights_gb":211.343,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":225.735,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":119.628,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":129.435,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":211.343,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":225.735,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":17,"score_norm":17,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26.52,"score_norm":26.52,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: step 3.5 flash 2603","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":60.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: step 3.5 flash 2603, source value 0.603333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":97.3,"score_norm":96.63,"config":"aime 2025, no tools, repeat@64 [raw: aime 2025], table 5 step 3.5 flash 'vanilla' column (non-pacore)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1393.3,"score_norm":50.66,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1394.4,"score_norm":51.15,"config":"57417 votes, arena entry step-3.5-flash, leaderboard text-overall-style_control, rank 152, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":2.47,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: step 3.5 flash, source value 0.0246938775510204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.5,"score_norm":82.39,"config":null,"trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2602.10604","measured_date":"2026-02-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.13,"score_norm":81.87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: step 3.5 flash, source value 0.831313131313131 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.5,"score_norm":82.39,"config":"avg of 8 runs [raw: gpqa-diamond], no tools, official gpqa-diamond metric, step 3.5 flash 'vanilla' column (non-pacore), table 5","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.1,"score_norm":31.03,"config":"text only","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2602.10604","measured_date":"2026-02-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":24.47,"score_norm":33.04,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: step 3.5 flash 2603, source value 0.244670991658943 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":66.53,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: step 3.5 flash 2603, source value 0.66530612244898 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"config":null,"trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2602.10604","measured_date":"2026-02-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"config":"avg of 4 runs [raw: mmlu-pro], official mmlu-pro metric, step 3.5 flash 'vanilla' column, table 5","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":40.37,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: step 3.5 flash, scicode, source value 0.403747266639333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74.4,"score_norm":71.56,"config":null,"trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2602.10604","measured_date":"2026-02-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74.4,"score_norm":71.56,"config":"avg of 4 runs [raw: swe-bench verified], full 500-instance verified set, openhands codeact scaffold, step 3.5 flash column, table 5 'swe verified' row","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":94.44,"score_norm":99.25,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: step 3.5 flash, source value 0.944444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash-0202","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":88.2,"score_norm":90.93,"config":"avg of 8 runs [raw: tau2-bench (τ2-bench)], gpt-4.1 user model, official tau2-bench codebase across airline/retail/telecom, step 3.5 flash column, table 5","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":51,"score_norm":54.12,"config":"2.0","trust":2,"source_type":"paper","source_url":"https://arxiv.org/abs/2602.10604","measured_date":"2026-02-11","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: step 3.5 flash 2603, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":51,"score_norm":54.12,"config":"avg@8 (appendix reports 50.98%) [raw: terminal-bench 2.0], modified terminus 2 harness, table 5 step 3.5 flash column, terminal-bench 2.0","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2602.10604","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#stepfun/step-3.5-flash","publisher":"OpenRouter (listed 2026-01-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"step-3.7-flash","display_name":"Step 3.7 Flash","family":{"id":"step","vendor":"stepfun"},"generation":"3.7","size_class":"frontier-flash","params_total_b":201.37,"params_active_b":null,"params_exact":201365316160,"params_source":"hf_safetensors","vram_est":{"bf16_gb":403,"int8_gb":213,"int4_gb":121,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":1.14,"expected_int8_gb":244,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"stepfun-ai/Step-3.7-Flash","openrouter_id":"stepfun/step-3.7-flash","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-05-28","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":61.25,"inputs":[{"benchmark":"gpqa_diamond","score_raw":80.91,"score_norm":78.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-7-flash","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.809090909090909 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":21.41,"score_norm":28.54,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-7-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.214087117701576 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-7-flash","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":39.33,"score_norm":40.39,"trust":2,"source_url":"https://artificialanalysis.ai/models/step-3-7-flash","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.393258426966292 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-0.94,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2,"completion_usd_per_m":1.15,"cache_read_usd_per_m":0.04,"context_length":262144,"source":"openrouter","source_model_id":"stepfun/step-3.7-flash","as_of":"2026-10-06","blended_usd_per_m":0.4375,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":402.74,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":426.702,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":402.731,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":426.693,"method":"vendor","source_url":"https://huggingface.co/stepfun-ai/Step-3.7-Flash/tree/main"},{"quant":"fp8","weights_gb":213.452,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":227.95,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":120.822,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":130.688,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":213.452,"kv_cache_gb_per_8k":3.8252,"min_serve_gb":227.95,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":21.7,"score_norm":21.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.74,"score_norm":21.74,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":39.6,"score_norm":39.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.5,"score_norm":19.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":30.9,"score_norm":30.9,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":69.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, source value 0.696666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":14.82,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.148230088495575 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":7,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0700460712463557 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, source value 0.0228571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1161,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1187,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1179,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1080,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1185,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1201,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":29.07,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.29065950462548495 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":25.88,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, source value 0.25883999999999996 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.91,"score_norm":78.75,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.809090909090909 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":72.73,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.727312201476335 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":21.41,"score_norm":28.54,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.214087117701576 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48.1,"score_norm":67.79,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/stepfun-ai/Step-3.7-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":67.28,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, source value 0.672789115646258 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":30.27,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=high, it-bench sre, reasoning=true, source value 0.302730696798493 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.32,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, source value 0.753179190751445 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.05,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, scicode, source value 0.400462962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":56.3,"score_norm":85.5,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/stepfun-ai/Step-3.7-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":11.96,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.119587628865979 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":35.61,"score_norm":36.01,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.356060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":39.33,"score_norm":40.39,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.393258426966292 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/step-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.5,"score_norm":64.12,"config":"2.1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/stepfun-ai/Step-3.7-Flash","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#stepfun/step-3.7-flash","publisher":"OpenRouter (listed 2026-05-28)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"trinity-1-large-reasoning","display_name":"Trinity Large Thinking","family":{"id":"trinity","vendor":"arcee-ai"},"generation":"1","size_class":"frontier-large","params_total_b":398.64,"params_active_b":null,"params_exact":398635286016,"params_source":"hf_safetensors","vram_est":{"bf16_gb":797,"int8_gb":423,"int4_gb":239,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":{"efficiency":0.76,"expected_int8_gb":323,"method":"frontier-residual-v1","approximate":true},"hugging_face_id":"arcee-ai/Trinity-Large-Thinking","openrouter_id":"arcee-ai/trinity-large-thinking","is_moe":true,"variant_axis":"reasoning","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04-01","release_status":"released","knowledge_cutoff":null,"status":"current","supersedes":[],"superseded_by":null,"aliases":["trinity-large-thinking"],"intelligence":{"index":69.81,"inputs":[{"benchmark":"aime","score_raw":96.3,"score_norm":95.38,"trust":1,"source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","config":"aime25"},{"benchmark":"arena_elo","score_raw":1367.3,"score_norm":39.08,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":76.3,"score_norm":72.25,"trust":1,"source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","config":null},{"benchmark":"mmlu_pro","score_raw":83.4,"score_norm":72.33,"trust":1,"source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","config":null},{"benchmark":"swe_bench_verified","score_raw":63.2,"score_norm":59.11,"trust":1,"source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","config":"mini-swe-agent-v2 scaffold"},{"benchmark":"tau_bench","score_raw":94.7,"score_norm":99.6,"trust":1,"source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","config":"tau2-telecom"}],"coverage":0.643,"subset_bias":{"offset":11.26,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":0.8,"cache_read_usd_per_m":0.06,"context_length":262144,"source":"openrouter","source_model_id":"arcee-ai/trinity-large-thinking","as_of":"2026-10-06","blended_usd_per_m":0.3875,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":797.28,"kv_cache_gb_per_8k":1.2583,"min_serve_gb":838.402,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":797.276,"kv_cache_gb_per_8k":1.2583,"min_serve_gb":838.398,"method":"vendor","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking/tree/main"},{"quant":"fp8","weights_gb":422.558,"kv_cache_gb_per_8k":1.2583,"min_serve_gb":444.944,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":239.184,"kv_cache_gb_per_8k":1.2583,"min_serve_gb":252.402,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":422.558,"kv_cache_gb_per_8k":1.2583,"min_serve_gb":444.944,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":25.8,"score_norm":25.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.8,"score_norm":10.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":96.3,"score_norm":95.38,"config":"aime25","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1367.3,"score_norm":39.08,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1237.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1097,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1052,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1123,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1106,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1092,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1015,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1052,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.3,"score_norm":72.25,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.4,"score_norm":72.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":63.2,"score_norm":59.11,"config":"mini-swe-agent-v2 scaffold","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":94.7,"score_norm":99.6,"config":"tau2-telecom","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":88,"score_norm":90.67,"config":"tau2-airline","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/arcee-ai/Trinity-Large-Thinking","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#arcee-ai/trinity-large-thinking","publisher":"OpenRouter (listed 2026-04-01)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"wizardlm-2-8x22b","display_name":"WizardLM-2 8x22B","family":{"id":"wizardlm","vendor":"microsoft"},"generation":"2","size_class":"8x22b","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":"microsoft/WizardLM-2-8x22B","openrouter_id":"microsoft/wizardlm-2-8x22b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":65535,"release_date":"2024-04-16","release_status":"released","knowledge_cutoff":"2024-04-30","status":"current","supersedes":[],"superseded_by":null,"aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.62,"completion_usd_per_m":0.62,"cache_read_usd_per_m":null,"context_length":65535,"source":"openrouter","source_model_id":"microsoft/wizardlm-2-8x22b","as_of":"2026-10-06","blended_usd_per_m":0.62,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"bbh","score_raw":48.58,"score_norm":null,"config":"3-shot, open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":17.56,"score_norm":0,"config":"0-shot, open llm leaderboard v2 normalized accuracy","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":52.72,"score_norm":null,"config":"0-shot, open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_lvl_5","score_raw":22.28,"score_norm":null,"config":"4-shot, open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":39.96,"score_norm":0,"config":"5-shot, open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":14.54,"score_norm":null,"config":"0-shot, open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/alpindale/WizardLM-2-8x22B","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#microsoft/wizardlm-2-8x22b","publisher":"OpenRouter (listed 2024-04-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hermes-3-llama-3.1-405b","display_name":"Hermes 3 405B Instruct","family":{"id":"hermes","vendor":"nousresearch"},"generation":"3","size_class":"405b","params_total_b":405.85,"params_active_b":405.85,"params_exact":405853388800,"params_source":"hf_safetensors","vram_est":{"bf16_gb":812,"int8_gb":430,"int4_gb":244,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"NousResearch/Hermes-3-Llama-3.1-405B","openrouter_id":"nousresearch/hermes-3-llama-3.1-405b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-08-16","release_status":"released","knowledge_cutoff":"2023-12-31","status":"superseded","supersedes":[],"superseded_by":"hermes-4-405b","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1,"completion_usd_per_m":1,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"nousresearch/hermes-3-llama-3.1-405b","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":811.7,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":856.513,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":811.707,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":856.52,"method":"vendor","source_url":"https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-405B/tree/main"},{"quant":"fp8","weights_gb":430.201,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":455.939,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":243.51,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":259.913,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":430.201,"kv_cache_gb_per_8k":4.2279,"min_serve_gb":455.939,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.4,"score_norm":7.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-3-llama-3.1-405b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"agieval","score_raw":61.84,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_c","score_raw":69.45,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_e","score_raw":86.24,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":75.37,"score_norm":null,"config":"3-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"boolq","score_raw":88.93,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":44.84,"score_norm":27.94,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hellaswag","score_raw":90.19,"score_norm":null,"config":"10-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":84.87,"score_norm":null,"config":"strict — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_lvl_5","score_raw":30.85,"score_norm":null,"config":"4-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":85.02,"score_norm":null,"config":"5-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":54.14,"score_norm":23.57,"config":"5-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":48.26,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"openbookqa","score_raw":48.8,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"piqa","score_raw":85.96,"score_norm":null,"config":"0-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":65.57,"score_norm":null,"config":"0-shot — hermes 3 technical report, mc2, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":86.27,"score_norm":null,"config":"5-shot — hermes 3 technical report, table 5 (final downstream task evaluations)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2408.11857v1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-3-llama-3.1-405b","publisher":"OpenRouter (listed 2024-08-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.2","display_name":"GLM-5.2","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.2","size_class":"frontier","params_total_b":753.33,"params_active_b":39,"params_exact":753329940480,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1507,"int8_gb":799,"int4_gb":452,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-5.2","openrouter_id":"z-ai/glm-5.2","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-06-17","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"glm-5.3","aliases":["glm-5.2-744b","glm-5.2-753b","glm-5.2-reasoning"],"intelligence":{"index":74.45,"inputs":[{"benchmark":"aime","score_raw":99.2,"score_norm":99,"trust":1,"source_url":"https://huggingface.co/zai-org/GLM-5.2","config":"judge model gpt-5.5 (medium), answer-format system prompt"},{"benchmark":"arc_agi_2","score_raw":22.78,"score_norm":19.76,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.2501, eval=arc-agi-2 semi-private (v2_semi_private), org=z.ai, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":85.61,"score_norm":85.37,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.774, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":41.15,"score_norm":57.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-2","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.411492122335496 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":73.16,"score_norm":73.69,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":86.71,"score_norm":77.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.334, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":82.8,"score_norm":80.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=1.689, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":99.12,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.991228070175439 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=0.991, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.152,"completion_usd_per_m":12,"cache_read_usd_per_m":0.15,"context_length":1048576,"source":"openrouter","source_model_id":"z-ai/glm-5.2","as_of":"2026-10-06","blended_usd_per_m":3.114,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1506.66,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1582.729,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1506.667,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1582.736,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-5.2/tree/main"},{"quant":"fp8","weights_gb":798.53,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.193,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":451.998,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":475.334,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":798.53,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.193,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":38.4,"score_norm":38.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":45.67,"score_norm":45.67,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":68.8,"score_norm":68.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.7,"score_norm":33.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":52.64,"score_norm":52.64,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, reasoning=true, run: max, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":4.43,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":99.2,"score_norm":99,"config":"judge model gpt-5.5 (medium), answer-format system prompt","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":33.7,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.337020648967552 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":77,"score_norm":null,"config":"cost_per_task_usd=0.1916, eval=arc-agi-1 semi-private (v1_semi_private), org=z.ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":22.78,"score_norm":19.76,"config":"cost_per_task_usd=0.2501, eval=arc-agi-2 semi-private (v2_semi_private), org=z.ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":22.775,"score_norm":19.75,"config":"arc prize verified, glm-5.2","trust":2,"source_type":"aggregator","source_url":"https://benchmarklist.com/benchmarks/arc_agi_2/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":27.84,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.27844625079401897 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, reasoning=true, run: max, source value 0.208571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20.9,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":46.2,"score_norm":null,"config":"pier framework + mini-swe-agent harness, 400k context, isolated container","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1304,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1205,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1193,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1206,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1295,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1295,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1239,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1281,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1142,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1175,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1156,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1194,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1218,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1298,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1215,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1294,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":42.73,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.42733512384362876 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"frontierswe","score_raw":74.4,"score_norm":null,"config":"dominance; evaluated by proximal, 1m context, max effort, 128k max output; as of 2026-06-16","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":50.28,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, reasoning=true, run: max, source value 0.50276 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.61,"score_norm":85.37,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.774, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.49,"score_norm":90.83,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, reasoning=true, run: max, source value 0.894949494949495 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.2,"score_norm":93.24,"config":"idavidrein/gpqa dataset","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":90.97,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.909683022145028 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":41.15,"score_norm":57.57,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, reasoning=true, run: max, source value 0.411492122335496 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":54.7,"score_norm":77.5,"config":"with tools; 300k max context, no context management","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":40.5,"score_norm":56.62,"config":"text-only subset, no tools; temperature=1.0, top_p=0.95, max gen 163840 tok","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hmmt_feb_2026","score_raw":92.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hmmt_nov_2025","score_raw":94.4,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":73.33,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, reasoning=true, run: max, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"imoanswerbench","score_raw":91,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":42.66,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, reasoning=true, run: max, source value 0.426553672316384 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":73.16,"score_norm":73.69,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"lmarena_text_arena_elo","score_raw":1471,"score_norm":null,"config":"+/-5, 26, 973 votes, glm-5.2-max, rank 33, text arena overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":76.8,"score_norm":null,"config":"public 500-task subset, think mode, judged by gemini-3.0-pro","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":7.22,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.0722222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.71,"score_norm":77.85,"config":"edition=mmlu pro v1, runner=platform, stderr=0.334, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.714,"score_norm":77.86,"config":"thinking mode: max, vals ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-07-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"nl2repo","score_raw":48.9,"score_norm":null,"config":"temperature=1.0, top_p=1.0, max_new_tokens=48k, 400k context","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"posttrainbench","score_raw":34.3,"score_norm":null,"config":"1m context, max effort, 128k max output","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"programbench","score_raw":63.7,"score_norm":null,"config":"claude-code 2.1.156, reasoning_effort=max, max_tokens=64000, 400k context","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":50.46,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, scicode, source value 0.50462962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":62.1,"score_norm":95.17,"config":"openhands harness; temperature=1, top_p=1, max_new_tokens=32k, 400k context","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":82.8,"score_norm":80.89,"config":"edition=swe-bench verified v1, reasoning_effort=max, runner=external, stderr=1.689, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":82.8,"score_norm":80.89,"config":"no harness specified, vals ai independent run","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-07-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_marathon","score_raw":13,"score_norm":null,"config":"evaluated by abundant ai; 1m context, max effort","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":99.12,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.991228070175439 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":99.12,"score_norm":100,"config":"artificial analysis run, glm-5.2 (max effort), tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":34.64,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.34639175257732 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"config":"edition=terminal-bench 2.1, reasoning_effort=max, runner=external, stderr=0.991, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":50.76,"score_norm":53.84,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.507575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.9,"score_norm":85.76,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, reasoning=true, run: max, source value 0.779026217228464 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":82.7,"score_norm":91.41,"config":"best reported harness","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":81,"score_norm":89.41,"config":"terminus-2 harness, 256k context, temperature=1.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tool_decathlon","score_raw":48.2,"score_norm":null,"config":"official evaluation service, max_token 128k","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"wildclawbench","score_raw":54.2,"score_norm":null,"config":"overall","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.2","measured_date":"2026-06-17","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/zai-org/GLM-5.2","publisher":"Hugging Face"},{"url":"https://z.ai/model-api","publisher":"Zhipu AI"},{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.2","publisher":"OpenRouter (listed 2026-06-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.6-27b","display_name":"Qwen3.6 27B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.6","size_class":"27b","params_total_b":27.78,"params_active_b":27.78,"params_exact":27781427952,"params_source":"hf_safetensors","vram_est":{"bf16_gb":55.6,"int8_gb":29.4,"int4_gb":16.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3.6-27B","openrouter_id":"qwen/qwen3.6-27b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.8-27b","aliases":["qwen3.6-27b"],"intelligence":{"index":68.92,"inputs":[{"benchmark":"aime","score_raw":94.1,"score_norm":92.62,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","config":"2026"},{"benchmark":"gpqa_diamond","score_raw":84.24,"score_norm":83.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.842424242424242 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":23.08,"score_norm":31,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.230769230769231 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":64.03,"score_norm":53.4,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":86.2,"score_norm":77,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","config":null},{"benchmark":"swe_bench_verified","score_raw":70,"score_norm":66.67,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.051, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":94.15,"score_norm":98.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.941520467836257 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":60.67,"score_norm":65.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.606741573033708 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8413,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.32,"completion_usd_per_m":3.25,"cache_read_usd_per_m":0.03,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.6-27b","as_of":"2026-10-06","blended_usd_per_m":1.0525,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":55.56,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.875,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":55.563,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.878,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B/tree/main"},{"quant":"fp8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":16.668,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":18.205,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":13.13,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":608,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":21.89,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":5.9,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":9.83,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":13216.93,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"deepseek-v4-flash-reasoning","value":1262,"params_active_b":290.94,"quant":"fp8","batch_size":32,"context_tokens":8192,"trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash"},"scaling":"active_param_ratio","quant_scale":1,"quant":"fp8","params_active_b":27.78,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":72.35,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":27.78,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":120.59,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":18.5,"score_norm":18.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-27b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":27.51,"score_norm":27.51,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":53.7,"score_norm":53.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-27b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":21.4,"score_norm":21.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-27b","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.7,"score_norm":37.7,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":94.1,"score_norm":92.62,"config":"2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":32,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.31998000000000004 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.24,"score_norm":83.44,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.842424242424242 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.8,"score_norm":88.45,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":82.3,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.822984512954118 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":23.08,"score_norm":31,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.230769230769231 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":24,"score_norm":32.35,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.55,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.675510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":64.03,"score_norm":53.4,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.2,"score_norm":77,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.62,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.746242774566474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.81,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.398148148148148 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":53.5,"score_norm":80.83,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70,"score_norm":66.67,"config":"edition=swe-bench verified v1, runner=external, stderr=2.051, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.2,"score_norm":74.67,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":94.15,"score_norm":98.87,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.941520467836257 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":16.7,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.167010309278351 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":60.67,"score_norm":65.49,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.606741573033708 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.3,"score_norm":63.88,"config":"2.0, 256k ctx, 32 cpu/48 gb ram, 3h timeout, avg of 5 runs., harbor/terminus-2 harness, max_tokens=80k, temp=1.0, top_k=20, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.6-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-27b","publisher":"OpenRouter (listed 2026-04-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v4-pro-0813","display_name":"DeepSeek V4 Pro 0813","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"4","size_class":"frontier-v-pro","params_total_b":1650.5,"params_active_b":null,"params_exact":1650497936906,"params_source":"hf_safetensors","vram_est":{"bf16_gb":3301,"int8_gb":1750,"int4_gb":990,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-V4-Pro-0813","openrouter_id":"deepseek/deepseek-v4-pro-0813","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":1048576,"release_date":"2026-08-12","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"deepseek-v4-pro","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.66,"completion_usd_per_m":1.98,"cache_read_usd_per_m":0.022,"context_length":1048576,"source":"openrouter","source_model_id":"deepseek/deepseek-v4-pro-0813","as_of":"2026-10-06","blended_usd_per_m":0.99,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":3301,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":3467.073,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1749.53,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1838.03,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":990.3,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1040.838,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":892.744,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":938.405,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/tree/main"},{"quant":"int8","weights_gb":1749.53,"kv_cache_gb_per_8k":1.0234,"min_serve_gb":1838.03,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":41.3,"score_norm":41.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro-0813","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":68.8,"score_norm":68.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro-0813","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":36,"score_norm":36,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro-0813","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":87.9,"score_norm":97.53,"config":"2.1, deepseek harness (minimal mode), max reasoning effort, temperature=1.0, top_p=0.95.","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813","measured_date":null,"observed_at":"2026-09-28","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v4-pro-0813","publisher":"OpenRouter (listed 2026-08-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-plus","display_name":"Qwen3.5 Plus 2026-04-20","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"frontier-plus","params_total_b":397,"params_active_b":17,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":794,"int8_gb":421,"int4_gb":238,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.5-plus-02-15","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-03-04","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.7-plus","aliases":["qwen-3.5-plus-02-15","qwen-3.5-plus-397b-a17b","qwen3.5-plus","qwen3.5-plus-02-15","qwen3.5-plus-20260420"],"intelligence":{"index":83.6,"inputs":[{"benchmark":"aime","score_raw":86.04,"score_norm":82.55,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"+/-1.11, vals ai independent eval"},{"benchmark":"gpqa_diamond","score_raw":87.37,"score_norm":87.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"+/-1.67, vals ai independent eval"},{"benchmark":"mmlu_pro","score_raw":87.18,"score_norm":78.63,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"+/-0.33, overall task, vals ai independent eval"}],"coverage":0.3572,"subset_bias":{"offset":14.46,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.26,"completion_usd_per_m":1.56,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.5-plus-02-15","as_of":"2026-10-06","blended_usd_per_m":0.585,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":86.04,"score_norm":82.55,"config":"+/-1.11, vals ai independent eval","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-04-16","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1134,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1105,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1173,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1118,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1107,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1184,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1194,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.37,"score_norm":87.85,"config":"+/-1.67, vals ai independent eval","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.18,"score_norm":78.63,"config":"+/-0.33, overall task, vals ai independent eval","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-09-01","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-plus-02-15","publisher":"OpenRouter (listed 2026-02-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-fable-5","display_name":"Claude Fable 5","family":{"id":"claude","vendor":"Anthropic"},"generation":"5","size_class":"frontier-fable","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-fable-5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-06-09","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-fable-5.1","aliases":["claude-5-fable","claude-fable-5-reasoning"],"intelligence":{"index":91.24,"inputs":[{"benchmark":"arc_agi_2","score_raw":89.17,"score_norm":93.52,"trust":3,"source_url":"https://arcprize.org/results/anthropic-claude-fable-5","config":"cost_per_task_usd=5.449076000000001, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-claude-fable-5, org=anthropic, run: max, setting=evaluation conducted with 'max' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1504.3,"score_norm":100,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.18,"score_norm":96.03,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"compute_effort=max, edition=gpqa diamond v1, runner=platform, stderr=1.939, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":53,"score_norm":75,"trust":2,"source_url":"https://artificialanalysis.ai/articles/claude-fable-5-mythos-intelligence-index","config":"artificial analysis harness; adaptive reasoning max effort, opus 4.8 fallback on ~9% of hle tasks"},{"benchmark":"mmlu_pro","score_raw":91.5,"score_norm":85.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.278, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":95,"score_norm":94.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"compute_effort=max, edition=swe-bench verified v1, runner=external, stderr=0.976, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-fable-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":80.52,"score_norm":88.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=1.35, task=overall, updated=2026-08-19"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":10,"completion_usd_per_m":50,"cache_read_usd_per_m":1,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-fable-5","as_of":"2026-10-06","blended_usd_per_m":20,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":50.7,"score_norm":50.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":56.59,"score_norm":56.59,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.5,"score_norm":76.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":49.6,"score_norm":49.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":62.07,"score_norm":62.07,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience","score_raw":40,"score_norm":null,"config":"artificial analysis; adaptive reasoning max effort, opus 4.8 fallback","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/claude-fable-5-mythos-intelligence-index","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":43.3,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":48.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.4875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":98.5,"score_norm":null,"config":"cost_per_task_usd=2.1101577000000007, eval=arc-agi-1 semi-private (v1_semi_private), model_group=anthropic-claude-fable-5, org=anthropic, run: max, setting=evaluation conducted with 'max' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-fable-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":89.17,"score_norm":93.52,"config":"cost_per_task_usd=5.449076000000001, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-claude-fable-5, org=anthropic, run: max, setting=evaluation conducted with 'max' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-fable-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":89.2,"score_norm":93.56,"config":"$5.45/task, arc prize verified semi-private, max reasoning","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-fable-5","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1504.3,"score_norm":100,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1507.8,"score_norm":100,"config":"24331 votes, arena entry claude-fable-5-text, leaderboard text-overall-style_control, rank 1, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1496.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1504.3,"score_norm":null,"config":"5792 votes, arena entry claude-fable-5-document, document/overall, leaderboard document-overall-raw, rank 4, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1622.8,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1230.2,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1236.8,"score_norm":null,"config":"20630 votes, arena entry claude-fable-5-search, leaderboard search-overall-raw, rank 3, raw (no style control), search/overall, vote cutoff 2026-07-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1308.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1311.9,"score_norm":null,"config":"9562 votes, arena entry claude-fable-5-vision, leaderboard vision-overall-style_control, rank 1, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1625.3,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1626.3,"score_norm":null,"config":"8382 votes, arena entry claude-fable-5-webdev, code/overall, leaderboard webdev-overall-raw, rank 6, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":48.58,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.48581851262754977 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"blueprint_bench_2","score_raw":38.6,"score_norm":null,"config":"spatial reasoning; anthropic launch table","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":28.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.2857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1334,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1285,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1254,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1297,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1338,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1316,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1313,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1278,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1348,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1292,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1277,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1259,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1295,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1303,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1325,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1254,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1302,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":51.12,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.5111906893464637 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1932,"score_norm":null,"config":"elo; artificial analysis","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/claude-fable-5-mythos-intelligence-index","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":61.89,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.618855 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.18,"score_norm":96.03,"config":"compute_effort=max, edition=gpqa diamond v1, runner=platform, stderr=1.939, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.63,"score_norm":95.25,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.926262626262626 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.6,"score_norm":95.21,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://www.requesty.ai/models/anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":93.56,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.935591257779708 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":53,"score_norm":75,"config":"artificial analysis harness; adaptive reasoning max effort, opus 4.8 fallback on ~9% of hle tasks","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/claude-fable-5-mythos-intelligence-index","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":55.47,"score_norm":78.63,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.554680259499537 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":64.5,"score_norm":91.91,"config":"with tools; anthropic launch table","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":63.47,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.63469387755102 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":83,"score_norm":95.56,"config":"claude-fable-5-max-effort, global average (mean of 7 category averages), livebench 2026-06-25 release","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/livebench","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":64.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.644444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":91.5,"score_norm":85.83,"config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.278, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":91.5,"score_norm":85.83,"config":"compute effort max, opus 4.8 fallback used for refused tasks, temperature 1.0","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"osworld","score_raw":85,"score_norm":null,"config":"anthropic launch table, computer use, verified","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/qwen3-8-max","measured_date":"2026-08-03","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":60.19,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, scicode, source value 0.601851851851852 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":80.3,"score_norm":100,"config":"anthropic launch benchmark table (vendor-reported)","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95,"score_norm":94.44,"config":"compute_effort=max, edition=swe-bench verified v1, runner=external, stderr=0.976, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95,"score_norm":94.44,"config":"compute effort max, opus 4.8 fallback used for refused tasks, temperature 1.0","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/models/anthropic_claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.54,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.985380116959064 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.5,"score_norm":100,"config":"tau2","trust":2,"source_type":"aggregator","source_url":"https://www.requesty.ai/models/anthropic/claude-fable-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":38.14,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.381443298969072 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.52,"score_norm":88.85,"config":"edition=terminal-bench 2.1, runner=external, stderr=1.35, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":62.88,"score_norm":68.09,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.628787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.64,"score_norm":93.69,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, opus 4.8 fallback, reasoning=true, run: adaptive reasoning, source value 0.846441947565543 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-fable-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88,"score_norm":97.65,"config":"anthropic launch benchmark table (vendor-reported)","trust":0,"source_type":"community","source_url":"https://www.datacamp.com/blog/claude-fable-5","measured_date":"2026-06-09","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.claude.com/docs/en/about-claude/models/overview","publisher":"Anthropic"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.1","display_name":"Claude Opus 4.1","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.1","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-08-05","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4.1-opus"],"intelligence":{"index":56.48,"inputs":[{"benchmark":"aime","score_raw":80.33,"score_norm":75.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.803333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":80.9,"score_norm":78.73,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.809 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":12.47,"score_norm":15.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.124652455977757 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":71.37,"score_norm":68.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.713660933660934 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":34.34,"score_norm":34.52,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.343352330715909 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":2.06,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":15,"completion_usd_per_m":75,"cache_read_usd_per_m":1.5,"context_length":200000,"source":"openrouter","source_model_id":"anthropic/claude-opus-4.1","as_of":"2026-10-06","blended_usd_per_m":30,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":22.8,"score_norm":22.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.54,"score_norm":34.54,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":80.33,"score_norm":75.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.803333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1168,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1178,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1184,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1183,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.9,"score_norm":78.73,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.809 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.47,"score_norm":15.4,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.124652455977757 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":55.44,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.554421768707483 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":65.35,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.6535448276 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":67.92,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.679190751445087 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.9,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.4089793103 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":71.37,"score_norm":68.49,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.713660933660934 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.34,"score_norm":34.52,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.343352330715909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-1-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.1","publisher":"OpenRouter (listed 2025-08-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.5","display_name":"Claude Opus 4.5","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.5","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-11-24","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4.5-opus"],"intelligence":{"index":67.98,"inputs":[{"benchmark":"aime","score_raw":91.33,"score_norm":89.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.913333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":37.64,"score_norm":36.27,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"64k, cost_per_task_usd=2.4, eval=arc-agi-2 semi-private (v2_semi_private), model_group=opus-4-5-thinking, org=anthropic, run: thinking, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":86.57,"score_norm":86.72,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.865656565656566 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":30.12,"score_norm":41.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.301204819277108 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":72.58,"score_norm":72.4,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: high effort"},{"benchmark":"tau_bench","score_raw":89.47,"score_norm":92.63,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.894736842105263 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":46.97,"score_norm":49.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.46969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":-1.8,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":25,"cache_read_usd_per_m":0.5,"context_length":200000,"source":"openrouter","source_model_id":"anthropic/claude-opus-4.5","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":29.1,"score_norm":29.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":41.87,"score_norm":41.87,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.76 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":14,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.33,"score_norm":89.16,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.913333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":80,"score_norm":null,"config":"64k, cost_per_task_usd=1.47, eval=arc-agi-1 semi-private (v1_semi_private), model_group=opus-4-5-thinking, org=anthropic, run: thinking, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":37.64,"score_norm":36.27,"config":"64k, cost_per_task_usd=2.4, eval=arc-agi-2 semi-private (v2_semi_private), model_group=opus-4-5-thinking, org=anthropic, run: thinking, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":4.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0457142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1229,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1191,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1201,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1248,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1251,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1155,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1186,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1180,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1245,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1152,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1253,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.57,"score_norm":86.72,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.865656565656566 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.12,"score_norm":41.35,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.301204819277108 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":57.96,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.579591836734694 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":72.58,"score_norm":72.4,"config":"livebench-2026-06-25 release, run: high effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":87.09,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.870899470899471 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.05,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.740462427745665 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":49.54,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.49537037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":89.47,"score_norm":92.63,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.894736842105263 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.97,"score_norm":49.38,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.46969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-5-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.5","publisher":"OpenRouter (listed 2025-11-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.6","display_name":"Claude Opus 4.6","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.6","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.6","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-02-04","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4.6-opus"],"intelligence":{"index":78.13,"inputs":[{"benchmark":"arc_agi_2","score_raw":69.17,"score_norm":71.3,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=3.4719, eval=arc-agi-2 semi-private (v2_semi_private), high, model_group=claude-opus-4-6-thinking, org=anthropic, run: 120k, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1497.3,"score_norm":97.02,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.646,"score_norm":91.05,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"overall"},{"benchmark":"hle","score_raw":39.94,"score_norm":55.79,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, claude opus 4.6 (adaptive reasoning, max effort)"},{"benchmark":"livebench","score_raw":74.52,"score_norm":76.71,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run claude-opus-4-6-thinking-auto-high-effort"},{"benchmark":"mmlu_pro","score_raw":89.107,"score_norm":81.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"overall"},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"overall"},{"benchmark":"tau_bench","score_raw":92.11,"score_norm":96.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.921052631578947 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":48.48,"score_norm":51.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-6","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.484848484848485 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":25,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-opus-4.6","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":31.9,"score_norm":31.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":44.93,"score_norm":44.93,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=true, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.743333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":13.67,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":33.04,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.3303834808259587 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":94,"score_norm":null,"config":"cost_per_task_usd=1.3948, eval=arc-agi-1 semi-private (v1_semi_private), high, model_group=claude-opus-4-6-thinking, org=anthropic, run: 120k, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":69.17,"score_norm":71.3,"config":"cost_per_task_usd=3.4719, eval=arc-agi-2 semi-private (v2_semi_private), high, model_group=claude-opus-4-6-thinking, org=anthropic, run: 120k, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":68.75,"score_norm":70.83,"config":"run claude-opus-4-6-thinking-120k-max, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1497.3,"score_norm":97.02,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1504.6,"score_norm":100,"config":"arena entry claude-opus-4-6-high, rank 2, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1507.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1540,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1294.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1537,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":12.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.125714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1213,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1215,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1266,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1295,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1218,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1290,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1204,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1227,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1291,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1199,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1298,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.646,"score_norm":91.05,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.6,"score_norm":90.99,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.895959595959596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.6,"score_norm":90.99,"config":"artificial analysis eval, claude opus 4.6 (adaptive reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":39.94,"score_norm":55.79,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.399443929564411 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":39.94,"score_norm":55.79,"config":"artificial analysis eval, claude opus 4.6 (adaptive reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":53.13,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: adaptive reasoning, source value 0.531292517006803 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":74.52,"score_norm":76.71,"config":"livebench-2026-06-25 release, run: high effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":74.52,"score_norm":76.71,"config":"global average, livebench-2026-06-25 release, run claude-opus-4-6-thinking-auto-high-effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.107,"score_norm":81.85,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.43,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, mmmu-pro, reasoning=true, run: adaptive reasoning, source value 0.754335260115607 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":51.85,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, scicode, source value 0.518518518518518 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.11,"score_norm":96.15,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.921052631578947 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":48.48,"score_norm":51.15,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.484848484848485 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.6","publisher":"OpenRouter (listed 2026-02-04)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.7","display_name":"Claude Opus 4.7","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.7","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.7","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-04-16","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4.7-opus"],"intelligence":{"index":80.92,"inputs":[{"benchmark":"arc_agi_2","score_raw":68.33,"score_norm":70.37,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=3.17, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-4-7, org=anthropic, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1494.1,"score_norm":95.59,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":90.152,"score_norm":91.76,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"overall"},{"benchmark":"hle","score_raw":42.31,"score_norm":59.28,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, claude opus 4.7 (adaptive reasoning, max effort)"},{"benchmark":"livebench","score_raw":76.53,"score_norm":81.18,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh effort"},{"benchmark":"mmlu_pro","score_raw":89.871,"score_norm":83.12,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"overall"},{"benchmark":"swe_bench_verified","score_raw":82,"score_norm":80,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"overall"},{"benchmark":"tau_bench","score_raw":88.6,"score_norm":91.47,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.885964912280702 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":68.539,"score_norm":74.75,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"overall, v2.1"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":25,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-opus-4.7","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":38.6,"score_norm":38.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":46.31,"score_norm":46.31,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":73.6,"score_norm":73.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.7,"score_norm":40.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":54.96,"score_norm":54.96,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":75.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.753333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":27.27,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":43.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.4375 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":93.5,"score_norm":null,"config":"cost_per_task_usd=1.41, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-4-7, org=anthropic, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":68.33,"score_norm":70.37,"config":"cost_per_task_usd=3.17, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-4-7, org=anthropic, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":75.83,"score_norm":78.7,"config":"run claude-4-7-max, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.18,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=anthropic-opus-4-7, org=anthropic, run: high, system_type=cot, total_cost_usd=10000","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1494.1,"score_norm":95.59,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1502.4,"score_norm":99.29,"config":"arena entry claude-opus-4-7-high, rank 3, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1494.7,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1564.5,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1233.3,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1297.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1555.3,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":12,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.12 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1290,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1246,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-08-30","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1243,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1334,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1247,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1503,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1320,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1270,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1222,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1333,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1336,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1263,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1329,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1279,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-08-29","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1309,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":49.48,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: adaptive reasoning, source value 0.49475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.152,"score_norm":91.76,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.41,"score_norm":93.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.914141414141414 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.41,"score_norm":93.54,"config":"artificial analysis eval, claude opus 4.7 (adaptive reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.31,"score_norm":59.28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.423076923076923 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.31,"score_norm":59.28,"config":"artificial analysis eval, claude opus 4.7 (adaptive reasoning, max effort)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":58.64,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: adaptive reasoning, source value 0.586394557823129 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":46.66,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, max effort, reasoning=true, run: adaptive reasoning, source value 0.466572504708098 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":76.53,"score_norm":81.18,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":76.53,"score_norm":81.18,"config":"global average, livebench-2026-06-25 release, run claude-opus-4-7-xhigh-effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":28.89,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.288888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.871,"score_norm":83.12,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.84,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, mmmu-pro, reasoning=true, run: adaptive reasoning, source value 0.788439306358382 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":54.51,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, scicode, source value 0.545138888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":82,"score_norm":80,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":88.6,"score_norm":91.47,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.885964912280702 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":34.64,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.34639175257732 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":68.539,"score_norm":74.75,"config":"overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":54.55,"score_norm":58.29,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.545454545454545 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":83.15,"score_norm":91.94,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.831460674157303 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":83.15,"score_norm":91.94,"config":"artificial analysis eval, claude opus 4.7 (adaptive reasoning, max effort), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.7","publisher":"OpenRouter (listed 2026-04-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4.8","display_name":"Claude Opus 4.8","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.8","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4.8","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-05-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4.8-opus"],"intelligence":{"index":85.66,"inputs":[{"benchmark":"arc_agi_2","score_raw":72.08,"score_norm":74.53,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=2.7425, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-opus-4-8, org=anthropic, run: high, setting=evaluation conducted with 'high' output effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1474.5,"score_norm":86.86,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":92.02,"score_norm":94.39,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.92020202020202 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":48.66,"score_norm":68.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.486561631139944 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":88.6,"score_norm":87.33,"trust":2,"source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","config":"anthropic-reported; default (high) effort"},{"benchmark":"tau_bench","score_raw":94.44,"score_norm":99.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.944444444444445 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":84.64,"score_norm":93.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.846441947565543 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.6826,"subset_bias":{"offset":0.1,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":25,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-opus-4.8","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":41.9,"score_norm":41.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":49.37,"score_norm":49.37,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":74.3,"score_norm":74.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":41.8,"score_norm":41.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":57.33,"score_norm":57.33,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":28.75,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":45,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.45 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":92.5,"score_norm":null,"config":"cost_per_task_usd=2.332, eval=arc-agi-1 semi-private (v1_semi_private), model_group=anthropic-opus-4-8, org=anthropic, run: max, setting=evaluation conducted with 'max' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":72.08,"score_norm":74.53,"config":"cost_per_task_usd=2.7425, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-opus-4-8, org=anthropic, run: high, setting=evaluation conducted with 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":1.52,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=anthropic-opus-4-8, org=anthropic, run: high, setting=evaluation conducted with 'high' output effort., system_type=cot, total_cost_usd=10000","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1474.5,"score_norm":86.86,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1464.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1204.3,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1279.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1534.1,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":48.53,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.48529386173175704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":20.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.208571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1238,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1238,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1227,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1294,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1247,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1290,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1259,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1247,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1248,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1231,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1220,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1220,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1306,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1298,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1193,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1224,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1263,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4.8","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":43.96,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.4395702775290958 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"finance_agent_v2","score_raw":53.9,"score_norm":null,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa","score_raw":1890,"score_norm":null,"config":"opus 4.8 system card table 8.1.a; non-percentage scale","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":54.19,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: adaptive reasoning, source value 0.54187 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.6,"score_norm":96.62,"config":"anthropic-reported; default effort","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.02,"score_norm":94.39,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.92020202020202 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":91.08,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.910840932117528 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":57.9,"score_norm":82.21,"config":"with tools","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":49.8,"score_norm":70.29,"config":"no tools","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48.66,"score_norm":68.62,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.486561631139944 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":62.24,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: adaptive reasoning, source value 0.622448979591837 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":45.56,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.455555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"online_mind2web","score_raw":84,"score_norm":null,"config":"reported by browserbase tester, not system-card table","trust":0,"source_type":"community","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":83.4,"score_norm":null,"config":"opus 4.8 system card table 8.1.a, verified","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.47,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, scicode, source value 0.534722222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":69.2,"score_norm":100,"config":"anthropic-reported; default (high) effort","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":88.6,"score_norm":87.33,"config":"anthropic-reported; default (high) effort","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":94.44,"score_norm":99.25,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.944444444444445 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":34.23,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.342268041237113 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.6,"score_norm":81.88,"config":"terminus-2 public harness","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-4-8-benchmarks-explained","measured_date":"2026-05-28","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":58.33,"score_norm":62.74,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.583333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.64,"score_norm":93.69,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.846441947565543 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-4-8","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/anthropics/skills/blob/main/skills/claude-api/shared/models.md","publisher":"Anthropic"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-5","display_name":"Claude Opus 5","family":{"id":"claude","vendor":"Anthropic"},"generation":"5","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-07-24","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":[],"intelligence":{"index":90.38,"inputs":[{"benchmark":"arc_agi_2","score_raw":90.42,"score_norm":94.91,"trust":3,"source_url":"https://arcprize.org/results/anthropic-claude-opus-5","config":"cost_per_task_usd=2.06, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-claude-opus-5, org=anthropic, run: max, setting=evaluation conducted with 'max' output effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1489.7,"score_norm":93.63,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.43,"score_norm":96.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.244, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":53,"score_norm":75,"trust":2,"source_url":"https://artificialanalysis.ai/articles/opus-5","config":"effort level unstated"},{"benchmark":"mmlu_pro","score_raw":91.59,"score_norm":85.98,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.276, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":97,"score_norm":96.67,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=0.764, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":80,"score_norm":80,"trust":2,"source_url":"https://openrouter.ai/anthropic/claude-opus-5","config":"domain (airline/retail/telecom) not stated on source page. other provider endpoints range 78.9-80.9 [raw: tau-bench], first-party anthropic provider endpoint row, openrouter \"benchmarks\" panel"},{"benchmark":"terminal_bench","score_raw":84.64,"score_norm":93.69,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"compute_effort=high, edition=terminal-bench 2.1, runner=external, stderr=0.991, task=overall, updated=2026-08-19"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":5,"completion_usd_per_m":25,"cache_read_usd_per_m":0.5,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-opus-5","as_of":"2026-10-06","blended_usd_per_m":10,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":56.5,"score_norm":56.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":59.17,"score_norm":59.17,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_briefcase","score_raw":1720,"score_norm":null,"config":"elo, max effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":78,"score_norm":78,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":50.8,"score_norm":50.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":63.05,"score_norm":63.05,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":78.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=medium, medium effort, reasoning=true, run: adaptive reasoning, source value 0.786666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_hallucination_rate","score_raw":50,"score_norm":null,"config":"max effort; lower is better","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":37.07,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":53.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.5375 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":97.5,"score_norm":null,"config":"cost_per_task_usd=0.47, eval=arc-agi-1 semi-private (v1_semi_private), model_group=anthropic-claude-opus-5, org=anthropic, run: high, setting=evaluation conducted with 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":97.5,"score_norm":null,"config":"max effort","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":90.42,"score_norm":94.91,"config":"cost_per_task_usd=2.06, eval=arc-agi-2 semi-private (v2_semi_private), model_group=anthropic-claude-opus-5, org=anthropic, run: max, setting=evaluation conducted with 'max' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":88.3,"score_norm":92.56,"config":"high effort","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":90.4,"score_norm":94.89,"config":"max effort; semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":30.16,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=anthropic-claude-opus-5, org=anthropic, run: high, setting=evaluation conducted with 'high' output effort., system_type=cot, total_cost_usd=20657.37","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":30.16,"score_norm":null,"config":"high effort; public demo, 25 environments","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/anthropic-claude-opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_3","score_raw":30.2,"score_norm":null,"config":"relative human action efficiency","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-5-benchmarks-explained","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1489.7,"score_norm":93.63,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1492.6,"score_norm":94.92,"config":"27610 votes, arena entry claude-opus-5-high-text, leaderboard text-overall-style_control, rank 7, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1489,"score_norm":93.32,"config":"text arena; claude-opus-5-max; rank 10, 9679 votes","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1493,"score_norm":95.1,"config":"text arena; claude-opus-5-high; rank 7, 20030 votes","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1516.3,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1520.1,"score_norm":null,"config":"1663 votes, arena entry claude-opus-5-high-document, document/overall, leaderboard document-overall-raw, rank 1, raw (no style control), run: high, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1289,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1292,"score_norm":null,"config":"7049 votes, arena entry claude-opus-5-high-vision, leaderboard vision-overall-style_control, rank 9, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1660,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1662.6,"score_norm":null,"config":"8659 votes, arena entry claude-opus-5-high-webdev, code/overall, leaderboard webdev-overall-raw, rank 4, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"autobench","score_raw":26,"score_norm":null,"config":"vellum llm leaderboard (work automations), rank 2","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/llm-leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":90.8,"score_norm":null,"config":"vellum llm leaderboard (browsing), rank 3","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/llm-leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":29.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.291428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1350,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1261,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1269,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1360,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1330,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1340,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1305,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1351,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1279,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1348,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1267,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1316,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1350,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1259,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1316,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":47.48,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.474783646672635 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"frontier_bench_v0_1","score_raw":43.3,"score_norm":null,"config":"mini-swe-agent harness, mean reward over 5 attempts","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/blog/claude-opus-5-benchmarks-explained","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":67.23,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: adaptive reasoning, source value 0.672335 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1861,"score_norm":null,"config":"elo, max effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.43,"score_norm":96.38,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.244, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.74,"score_norm":96.82,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, high effort, reasoning=true, run: adaptive reasoning, source value 0.937373737373737 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-high","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.1,"score_norm":90.28,"config":"first-party anthropic provider endpoint row (other provider endpoints on the same page range 88.3-90.1) [raw: gpqa diamond], openrouter \"benchmarks\" panel","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":53,"score_norm":75,"config":"effort level unstated","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":54.87,"score_norm":77.75,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.548656163113994 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":64.7,"score_norm":92.21,"config":"vellum llm leaderboard, rank 1","trust":2,"source_type":"aggregator","source_url":"https://www.vellum.ai/llm-leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":59.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=true, run: adaptive reasoning, source value 0.594444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":91.59,"score_norm":85.98,"config":"compute_effort=max, edition=mmlu pro v1, runner=platform, stderr=0.276, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":91.59,"score_norm":85.98,"config":"compute effort \"max\", max output 128k. refusal-fallback-adjusted variant on same page: 91.58 [raw: mmlu pro], rank #1 of 133, temperature 1, vals ai independent run","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/anthropic_claude-opus-5","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":84.74,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, mmmu-pro, reasoning=true, run: adaptive reasoning, source value 0.847398843930636 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":55.67,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, scicode, source value 0.556712962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":97,"score_norm":96.67,"config":"edition=swe-bench verified v1, runner=external, stderr=0.764, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":97,"score_norm":96.67,"config":"vals.ai independent run; minimal bash-tool-only agent harness, 500 tasks","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":80,"score_norm":80,"config":"domain (airline/retail/telecom) not stated on source page. other provider endpoints range 78.9-80.9 [raw: tau-bench], first-party anthropic provider endpoint row, openrouter \"benchmarks\" panel","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/anthropic/claude-opus-5","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":44.74,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=true, run: adaptive reasoning, source value 0.447422680412371 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5-high","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":84.64,"score_norm":93.69,"config":"compute_effort=high, edition=terminal-bench 2.1, runner=external, stderr=0.991, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":89,"score_norm":98.82,"config":"max effort","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/articles/opus-5","measured_date":"2026-07-24","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":89.14,"score_norm":98.99,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.891385767790262 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-opus-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.claude.com/docs/en/about-claude/models/overview","publisher":"Anthropic"},{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-5","publisher":"OpenRouter (listed 2026-07-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-sonnet-4","display_name":"Claude Sonnet 4","family":{"id":"claude","vendor":"Anthropic"},"generation":"4","size_class":"frontier-sonnet","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-sonnet-4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-05-22","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"claude-sonnet-5","aliases":["claude-4-sonnet"],"intelligence":{"index":49.58,"inputs":[{"benchmark":"aime","score_raw":76.25,"score_norm":70.31,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=1.435, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":5.93,"score_norm":1.03,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.4857, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot"},{"benchmark":"arena_elo","score_raw":1399.8,"score_norm":53.56,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"35220 votes, arena entry claude-sonnet-4-20250514-thinking-32k, leaderboard text-overall-style_control, rank 144, run: claude-sonnet-4-20250514-thinking-32k, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":75,"score_norm":70.42,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=2.191, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":10.7,"score_norm":12.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.107043558850788 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":83.86,"score_norm":73.1,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=0.361, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":64.62,"score_norm":59.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.646198830409357 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":36.33,"score_norm":36.86,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.363295880149813 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.42,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":0.3,"context_length":200000,"source":"openrouter","source_model_id":"anthropic/claude-sonnet-4","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":17.6,"score_norm":17.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":17.6,"score_norm":17.6,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":37.6,"score_norm":37.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":18.9,"score_norm":18.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.76,"score_norm":29.76,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":69.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.696666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.17,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":76.25,"score_norm":70.31,"config":"archived=true, edition=aime v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=1.435, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":74.33,"score_norm":67.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.743333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":40,"score_norm":null,"config":"cost_per_task_usd=0.3658, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-sonnet-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":5.93,"score_norm":1.03,"config":"cost_per_task_usd=0.4857, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1399.8,"score_norm":53.56,"config":"35220 votes, arena entry claude-sonnet-4-20250514-thinking-32k, leaderboard text-overall-style_control, rank 144, run: claude-sonnet-4-20250514-thinking-32k, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1207.8,"score_norm":null,"config":"1159 votes, arena entry claude-sonnet-4-20250514-thinking-32k, leaderboard vision-overall-style_control, rank 68, run: claude-sonnet-4-20250514-thinking-32k, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1154,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1147,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1166,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1077,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1135,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1152,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":18.7,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.18696 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75,"score_norm":70.42,"config":"edition=gpqa diamond v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=2.191, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.68,"score_norm":74.2,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.776767676767677 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.7,"score_norm":12.79,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.107043558850788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":54.69,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.546938775510204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":65.5,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.655026455026455 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.86,"score_norm":73.1,"config":"edition=mmlu pro v1, run: anthropic/claude-sonnet-4-20250514-thinking, runner=platform, stderr=0.361, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":62.37,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, run: non-reasoning, source value 0.623699421965318 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.05,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.400462962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":64.62,"score_norm":59.49,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.646198830409357 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":16.7,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.167010309278351 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":36.33,"score_norm":36.86,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.363295880149813 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4","publisher":"OpenRouter (listed 2025-05-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-sonnet-4.5","display_name":"Claude Sonnet 4.5","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.5","size_class":"frontier-sonnet","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-sonnet-4.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2025-09-29","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"claude-sonnet-5","aliases":["claude-4.5-sonnet"],"intelligence":{"index":56.68,"inputs":[{"benchmark":"aime","score_raw":88,"score_norm":85,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.88 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":13.61,"score_norm":9.57,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.7589, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-5-20250929, org=anthropic, run: thinking 32k, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":83.43,"score_norm":82.3,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.834343434343434 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":17.79,"score_norm":23.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.177942539388323 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":78.07,"score_norm":77.43,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.780701754385965 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":55.81,"score_norm":59.78,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.558052434456929 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-4.46,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":0.3,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-sonnet-4.5","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":15.8,"score_norm":15.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":26.44,"score_norm":26.44,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":52.1,"score_norm":52.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.7,"score_norm":20.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.39,"score_norm":37.39,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":88,"score_norm":85,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.88 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":63.67,"score_norm":null,"config":"cost_per_task_usd=0.5193, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-sonnet-4-5-20250929, org=anthropic, run: thinking 32k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":13.61,"score_norm":9.57,"config":"cost_per_task_usd=0.7589, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-5-20250929, org=anthropic, run: thinking 32k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1173,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1216,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1189,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1049,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1179,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1134,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1114,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1179,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1053,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1196,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":27.78,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.277775 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.43,"score_norm":82.3,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.834343434343434 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":17.79,"score_norm":23.22,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.177942539388323 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":57.28,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.572789115646258 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":71.43,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.714285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":68.73,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.68728323699422 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":44.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.446759259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":78.07,"score_norm":77.43,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.780701754385965 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":24.54,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.245360824742268 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":55.81,"score_norm":59.78,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.558052434456929 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":35.61,"score_norm":36.01,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.356060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-5-sonnet-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.5","publisher":"OpenRouter (listed 2025-09-29)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-sonnet-4.6","display_name":"Claude Sonnet 4.6","family":{"id":"claude","vendor":"Anthropic"},"generation":"4.6","size_class":"frontier-sonnet","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-sonnet-4.6","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-02-17","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-sonnet-5","aliases":["claude-4.6-sonnet"],"intelligence":{"index":75.85,"inputs":[{"benchmark":"aime","score_raw":95.6,"score_norm":94.5,"trust":1,"source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","config":"adaptive thinking, aime 2025, avg of 10 trials [raw: aime 2025], max effort, no tools"},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=2.6966, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-6, org=anthropic, run: high, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1472.2,"score_norm":85.83,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":87.47,"score_norm":87.99,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.874747474747475 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":33.6,"score_norm":46.47,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.33595922150139 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":73,"score_norm":73.33,"trust":3,"source_url":"https://livebench.ai/","config":"2026-06-25 release, global average (overall), row 'claude 4.6 sonnet thinking medium effort' [raw: livebench]"},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"trust":1,"source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","config":"500 problems, adaptive thinking, averaged over 10 trials (table footnote says 25), max effort, no prompt modification [raw: swe-bench verified]"},{"benchmark":"tau_bench","score_raw":79.53,"score_norm":79.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.795321637426901 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":71.16,"score_norm":77.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.711610486891386 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":3,"completion_usd_per_m":15,"cache_read_usd_per_m":0.3,"context_length":1000000,"source":"openrouter","source_model_id":"anthropic/claude-sonnet-4.6","as_of":"2026-10-06","blended_usd_per_m":6,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":31.8,"score_norm":31.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":42.08,"score_norm":42.08,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=max, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":63,"score_norm":63,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":30.1,"score_norm":30.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":48.37,"score_norm":48.37,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=max, estimated=false, index unit 0-100 emitted as-is (no rescale), max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.74 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":12.22,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=max, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, max effort, reasoning=true, run: adaptive reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.6,"score_norm":94.5,"config":"adaptive thinking, aime 2025, avg of 10 trials [raw: aime 2025], max effort, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"analyst_agent","score_raw":20,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.2 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":28.02,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.28023598820059 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":86.5,"score_norm":null,"config":"cost_per_task_usd=1.1908, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-sonnet-4-6, org=anthropic, run: high, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"cost_per_task_usd=2.6966, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-sonnet-4-6, org=anthropic, run: high, setting=evaluation conducted with a 120k thinking budget and a maximum context window of 128k tokens and 'high' output effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"120k thinking budget ('claude sonnet 4.6 (high)') [raw: arc-agi-2], arc-agi-2 semi-private eval, high effort","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/media/data/leaderboard/v2.json","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1472.2,"score_norm":85.83,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1482.2,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1530,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1275.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1521.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":23.82,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.23823599456385894 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.0314285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1250,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1177,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1211,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1246,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1282,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1287,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1211,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1226,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1209,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1196,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1278,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1183,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1291,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":43.72,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=max, gdpval-aa, max effort, reasoning=true, run: adaptive reasoning, source value 0.43716499999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.47,"score_norm":87.99,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=max, gpqa diamond, max effort, reasoning=true, run: adaptive reasoning, source value 0.874747474747475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"config":"adaptive thinking, avg of 10 trials [raw: gpqa diamond], diamond subset (198 q), max effort, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":85.97,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.859748154580981 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":33.6,"score_norm":46.47,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=max, humanity's last exam, max effort, reasoning=true, run: adaptive reasoning, source value 0.33595922150139 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":33.2,"score_norm":45.88,"config":"adaptive thinking, avg of 10 trials, claude sonnet 4.5 as grader [raw: humanity's last exam (hle)], max effort, no tools, reasoning-only","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":56.6,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=max, ifbench, max effort, reasoning=true, run: adaptive reasoning, source value 0.565986394557823 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":39.8,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=max, it-bench sre, max effort, reasoning=true, run: adaptive reasoning, source value 0.398022598870057 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":73,"score_norm":73.33,"config":"2026-06-25 release, global average (overall), row 'claude 4.6 sonnet thinking medium effort' [raw: livebench]","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":24.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.244444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.29,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, mmmu-pro, reasoning=true, run: adaptive reasoning, source value 0.732947976878613 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, scicode, source value 0.46875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"config":"500 problems, adaptive thinking, averaged over 10 trials (table footnote says 25), max effort, no prompt modification [raw: swe-bench verified]","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":79.53,"score_norm":79.37,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, high effort, reasoning=false, run: non-reasoning, source value 0.795321637426901 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":34.43,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.344329896907216 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":71.16,"score_norm":77.84,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.711610486891386 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.03,"score_norm":56.51,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=max, max effort, reasoning=true, run: adaptive reasoning, source value 0.53030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-sonnet-4-6-adaptive","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":59.1,"score_norm":63.65,"config":"89 tasks x 5 runs [raw: terminal-bench 2.0], max effort, no thinking, terminus-2 harness / harbor scaffold","trust":1,"source_type":"vendor_claim","source_url":"https://www.anthropic.com/claude-sonnet-4-6-system-card","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-sonnet-4.6","publisher":"OpenRouter (listed 2026-02-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v3","display_name":"DeepSeek-V3","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3","size_class":"frontier-v","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-V3","openrouter_id":"deepseek/deepseek-chat","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"MIT","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2024-12","release_status":"released","knowledge_cutoff":"2024-07-31","status":"superseded","supersedes":["deepseek-3-instruct"],"superseded_by":"deepseek-v3.2","aliases":["deepseek-3-instruct","deepseek-chat","deepseek-chat-v3","deepseek-v3-671b"],"intelligence":{"index":27.75,"inputs":[{"benchmark":"aime","score_raw":27.5,"score_norm":9.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.73, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1358.4,"score_norm":35.11,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":54.55,"score_norm":41.62,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.504, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":2.9,"score_norm":1.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: dec '24, source value 0.029 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":73.82,"score_norm":56.37,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.424, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":22.81,"score_norm":3.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.228070175438596 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":16.85,"score_norm":13.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.168539325842697 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":1.78,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.2574,"completion_usd_per_m":1.0287,"cache_read_usd_per_m":null,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-chat","as_of":"2026-10-06","blended_usd_per_m":0.450225,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1369.06,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.089,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.587,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.592,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3/tree/main"},{"quant":"int4","weights_gb":410.718,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.58,"score_norm":1.58,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: dec '24","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.22,"score_norm":14.22,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: dec '24","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":31.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: dec '24, source value 0.316666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":27.5,"score_norm":9.38,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.73, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":26,"score_norm":7.5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.26 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1358.4,"score_norm":35.11,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1358.5,"score_norm":35.15,"config":"21770 votes, arena entry deepseek-v3, leaderboard text-overall-style_control, rank 189, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: dec '24, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1102,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1114,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1097,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1066,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":975,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1100,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1126,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, run: dec '24, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":54.55,"score_norm":41.62,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.504, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":55.66,"score_norm":43.18,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: dec '24, source value 0.556565656565657 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":2.9,"score_norm":1.32,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: dec '24, source value 0.029 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":34.76,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, run: dec '24, source value 0.347619047619048 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":35.87,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, run: dec '24, source value 0.358730158730159 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":73.82,"score_norm":56.37,"config":"edition=mmlu pro v1, runner=platform, stderr=0.424, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":64.4,"score_norm":40.67,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":35.42,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, scicode, source value 0.354166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":22.81,"score_norm":3.75,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.228070175438596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":4.74,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.0474226804123711 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":16.85,"score_norm":13.94,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: dec '24, source value 0.168539325842697 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://github.com/deepseek-ai/DeepSeek-V3","publisher":"DeepSeek"},{"url":"https://arxiv.org/pdf/2412.19437","publisher":"arXiv"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-v3.1-terminus","display_name":"DeepSeek V3.1 Terminus","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3.1","size_class":"frontier-v","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-V3.1-Terminus","openrouter_id":"deepseek/deepseek-v3.1-terminus","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-09-22","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":[],"superseded_by":"deepseek-v3.2","aliases":[],"intelligence":{"index":56.28,"inputs":[{"benchmark":"aime","score_raw":89.67,"score_norm":87.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.896666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1414.8,"score_norm":60.25,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":79.19,"score_norm":76.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.791919191919192 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":16.36,"score_norm":21.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.163577386468953 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":37.13,"score_norm":22.84,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.371345029239766 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":44.94,"score_norm":46.99,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":0.06,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.27,"completion_usd_per_m":1,"cache_read_usd_per_m":null,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-v3.1-terminus","as_of":"2026-10-06","blended_usd_per_m":0.4525,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1369.06,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.089,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.589,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.594,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus/tree/main"},{"quant":"int4","weights_gb":410.718,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":6.8,"score_norm":6.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":19.85,"score_norm":19.85,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":43.5,"score_norm":43.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.8,"score_norm":14.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.36,"score_norm":31.36,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.68 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":89.67,"score_norm":87.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.896666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1414.8,"score_norm":60.25,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1417,"score_norm":61.23,"config":"3470 votes, arena entry deepseek-v3.1-terminus-thinking, leaderboard text-overall-style_control, rank 118, run: deepseek-v3.1-terminus-thinking, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":1.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0171428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1156,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1181,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1143,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1064,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1189,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1193,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":19.16,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.19159500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.19,"score_norm":76.32,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.791919191919192 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":16.36,"score_norm":21.12,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.163577386468953 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":57.01,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.570068027210884 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":79.79,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.797883597883598 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.62,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.40625 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":37.13,"score_norm":22.84,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.371345029239766 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":21.03,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.210309278350515 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.94,"score_norm":46.99,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.449438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.82,"score_norm":31.55,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.318181818181818 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-1-terminus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-v3.1-terminus","publisher":"OpenRouter (listed 2025-09-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-2.5-flash","display_name":"Gemini 2.5 Flash","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-04","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"gemini-3.8-flash","aliases":[],"intelligence":{"index":41.16,"inputs":[{"benchmark":"aime","score_raw":73.33,"score_norm":66.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":1.69,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.057, eval=arc-agi-2 semi-private (v2_semi_private), org=google, run: preview, system_type=cot"},{"benchmark":"arena_elo","score_raw":1409.4,"score_norm":57.84,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":78.99,"score_norm":76.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.78989898989899 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":12.14,"score_norm":14.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.121408711770158 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":31.58,"score_norm":15.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.315789473684211 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6031,"subset_bias":{"offset":-5.07,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.03,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-2.5-flash","as_of":"2026-10-06","blended_usd_per_m":0.85,"endpoint_expires_at":"2026-10-20"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":15.5,"score_norm":15.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.34,"score_norm":20.34,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.656666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":73.33,"score_norm":66.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":33.33,"score_norm":null,"config":"cost_per_task_usd=0.0371, eval=arc-agi-1 semi-private (v1_semi_private), org=google, run: preview, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":1.69,"score_norm":0,"config":"cost_per_task_usd=0.057, eval=arc-agi-2 semi-private (v2_semi_private), org=google, run: preview, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1409.4,"score_norm":57.84,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1410,"score_norm":58.11,"config":"124731 votes, arena entry gemini-2.5-flash, leaderboard text-overall-style_control, rank 132, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1215.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1214.3,"score_norm":null,"config":"59372 votes, arena entry gemini-2.5-flash, leaderboard vision-overall-style_control, rank 65, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0.0142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1086,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1110,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1142,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1078,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1019,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1098,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1122,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.99,"score_norm":76.04,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.78989898989899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.14,"score_norm":14.91,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.121408711770158 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":50.27,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.502721088435374 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":69.52,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.695238095238095 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":8.89,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0888888888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":69.08,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.690751445086705 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.35,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.393518518518519 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":31.58,"score_norm":15.44,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.315789473684211 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash","publisher":"OpenRouter (listed 2025-06-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3-flash","display_name":"Gemini 3 Flash Preview","family":{"id":"gemini","vendor":"Google"},"generation":"3","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3-flash-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file","audio","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-12-17","release_status":"preview","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gemini-3.8-flash","aliases":["gemini-3-flash-preview"],"intelligence":{"index":72.93,"inputs":[{"benchmark":"aime","score_raw":95.62,"score_norm":94.53,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.516, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":33.61,"score_norm":31.79,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.2314, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-flash-preview-thinking, org=google, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1472.8,"score_norm":86.1,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"30852 votes, arena entry gemini-3-flash, leaderboard text-overall-style_control, rank 30, run: gemini-3-flash, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":87.88,"score_norm":88.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.638, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":36.56,"score_norm":50.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.365616311399444 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":88.59,"score_norm":80.98,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.315, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.938, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":80.41,"score_norm":80.55,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.804093567251462 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":53.93,"score_norm":57.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.297, task=overall, updated=2026-08-19"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.5,"completion_usd_per_m":3,"cache_read_usd_per_m":0.05,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3-flash-preview","as_of":"2026-10-06","blended_usd_per_m":1.125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":26.3,"score_norm":26.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.74,"score_norm":38.74,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":10.13,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.62,"score_norm":94.53,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.516, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":97,"score_norm":96.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.97 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":27.73,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.277286135693215 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":84.67,"score_norm":null,"config":"cost_per_task_usd=0.1743, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gemini-3-flash-preview-thinking, org=google, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":33.61,"score_norm":31.79,"config":"cost_per_task_usd=0.2314, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-flash-preview-thinking, org=google, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1472.8,"score_norm":86.1,"config":"30852 votes, arena entry gemini-3-flash, leaderboard text-overall-style_control, rank 30, run: gemini-3-flash, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1413.3,"score_norm":null,"config":"7173 votes, arena entry gemini-3-flash, document/overall, leaderboard document-overall-raw, rank 35, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1270.5,"score_norm":null,"config":"36437 votes, arena entry gemini-3-flash, leaderboard vision-overall-style_control, rank 28, run: gemini-3-flash, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1437.9,"score_norm":null,"config":"10477 votes, arena entry gemini-3-flash, code/overall, leaderboard webdev-overall-raw, rank 51, raw (no style control), run: gemini-3-flash, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1199,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1073,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1065,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1195,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1061,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1179,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1161,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1087,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1036,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1103,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1203,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.88,"score_norm":88.56,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.638, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.8,"score_norm":91.27,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.897979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":36.56,"score_norm":50.82,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.365616311399444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77.96,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.779591836734694 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":90.79,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.907936507936508 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":11.67,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.116666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.59,"score_norm":80.98,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.315, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":79.94,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.799421965317919 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":50.58,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.505787037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75,"score_norm":72.22,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.938, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":80.41,"score_norm":80.55,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.804093567251462 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":20.82,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.208247422680412 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.93,"score_norm":57.56,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.297, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":38.64,"score_norm":39.58,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.386363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-flash-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3-flash-preview","publisher":"OpenRouter (listed 2025-12-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.1-flash-lite","display_name":"Gemini 3.1 Flash Lite","family":{"id":"gemini","vendor":"Google"},"generation":"3.1","size_class":"frontier-flash-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.1-flash-lite","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file","audio"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-05-07","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gemini-3.5-flash-lite","aliases":["gemini-3.1-flash-lite-preview"],"intelligence":{"index":51.45,"inputs":[{"benchmark":"arena_elo","score_raw":1432.7,"score_norm":68.23,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":82.22,"score_norm":80.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.822222222222222 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":17.19,"score_norm":22.34,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.17191844300278 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":31.29,"score_norm":15.05,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.312865497076023 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":31.09,"score_norm":30.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.310861423220974 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-5.14,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":1.5,"cache_read_usd_per_m":0.025,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.1-flash-lite-preview","as_of":"2026-10-06","blended_usd_per_m":0.5625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.6,"score_norm":1.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":6.52,"score_norm":6.52,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":34.7,"score_norm":34.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.6,"score_norm":15.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":25.61,"score_norm":25.61,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, source value 0.713333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":8.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":12.17,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.121681415929204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1432.7,"score_norm":68.23,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1336.2,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1234.6,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1255.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":10.64,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.10641533320288254 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1063,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1078,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1053,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1042,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1051,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1078,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1089,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite-preview","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":28.02,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.2802148612354521 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":7.41,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, source value 0.07413999999999998 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.22,"score_norm":80.59,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, source value 0.822222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":31.15,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.311477782602403 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":17.19,"score_norm":22.34,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, source value 0.17191844300278 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77.21,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, source value 0.772108843537415 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":6.11,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0611111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.55,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, source value 0.755491329479769 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":41.9,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, scicode, source value 0.418981481481481 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":31.29,"score_norm":15.05,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.312865497076023 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":9.69,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.0969072164948454 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.24,"score_norm":22.64,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.242424242424242 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.09,"score_norm":30.69,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, source value 0.310861423220974 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-1-flash-lite-preview","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.1-flash-lite","publisher":"OpenRouter (listed 2026-05-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.5-flash","display_name":"Gemini 3.5 Flash","family":{"id":"gemini","vendor":"Google"},"generation":"3.5","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.5-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-05","release_status":"released","knowledge_cutoff":"2025-01-01","status":"superseded","supersedes":[],"superseded_by":"gemini-3.8-flash","aliases":[],"intelligence":{"index":80.82,"inputs":[{"benchmark":"arc_agi_2","score_raw":72.08,"score_norm":74.53,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.8503, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-5-flash, org=google, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1476.1,"score_norm":87.57,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":92.68,"score_norm":95.32,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.462, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":42.68,"score_norm":59.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.42678405931418 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":74.64,"score_norm":76.98,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run gemini-3.5-flash-high"},{"benchmark":"mmlu_pro","score_raw":89.52,"score_norm":82.53,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.306, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":78.8,"score_norm":76.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.83, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-medium","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":74.16,"score_norm":81.36,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.124, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.5,"completion_usd_per_m":9,"cache_read_usd_per_m":0.15,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.5-flash","as_of":"2026-10-06","blended_usd_per_m":3.375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":26,"score_norm":26,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":39.71,"score_norm":39.71,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":70.1,"score_norm":70.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.6,"score_norm":33.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":51.96,"score_norm":51.96,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.81 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":21.18,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":45,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.45 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":47.05,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.470501474926254 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":92.5,"score_norm":null,"config":"cost_per_task_usd=0.4278, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gemini-3-5-flash, org=google, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":72.08,"score_norm":74.53,"config":"cost_per_task_usd=0.8503, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-5-flash, org=google, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":72.08,"score_norm":74.53,"config":"run gemini-3-5-flash-high, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.1,"score_norm":87.57,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1478.2,"score_norm":88.51,"config":"30004 votes, arena entry gemini-3.5-flash-high-public, leaderboard text-overall-style_control, rank 21, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1461.4,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1465,"score_norm":null,"config":"1372 votes, arena entry gemini-3.5-flash-high-public, document/overall, leaderboard document-overall-raw, rank 17, raw (no style control), run: high, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1496.9,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1284.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1286.3,"score_norm":null,"config":"7586 votes, arena entry gemini-3.5-flash-high-public, leaderboard vision-overall-style_control, rank 11, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1490,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1498.6,"score_norm":null,"config":"7559 votes, arena entry gemini-3.5-flash-high-public, code/overall, leaderboard webdev-overall-raw, rank 34, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":42.64,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.42637546323234665 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":13.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.131428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1259,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1162,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1244,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1199,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1265,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1270,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1243,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1195,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1274,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1098,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1142,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1173,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1244,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1247,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1254,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1280,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1196,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1272,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":50.07,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.5007460459564309 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":42.17,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.42169000000000006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.68,"score_norm":95.32,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.462, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.676,"score_norm":95.32,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.22,"score_norm":94.68,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.922222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.12,"score_norm":94.54,"config":"artificial analysis eval, gemini 3.5 flash (medium)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":82.08,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.820813431755681 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":42.68,"score_norm":59.82,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.42678405931418 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":41.33,"score_norm":57.84,"config":"artificial analysis eval, gemini 3.5 flash (medium)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":76.33,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.763265306122449 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":40.35,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=high, it-bench sre, reasoning=true, run: high, source value 0.403483992467043 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":74.64,"score_norm":76.98,"config":"livebench-2026-06-25 release, run: high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":74.64,"score_norm":76.98,"config":"global average, livebench-2026-06-25 release, run gemini-3.5-flash-high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":18.33,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.183333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.52,"score_norm":82.53,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.306, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.515,"score_norm":82.53,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":84.28,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.842774566473988 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.12,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.53125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.8,"score_norm":76.44,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.83, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.8,"score_norm":76.44,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.61,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.956140350877193 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":32.16,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.321649484536082 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.16,"score_norm":81.36,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.124, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":74.157,"score_norm":81.36,"config":"high reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.21,"score_norm":48.48,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=minimal, reasoning=false, run: minimal, source value 0.462121212121212 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash-minimal","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.65,"score_norm":86.65,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.786516853932584 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-5-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.5-flash","publisher":"OpenRouter (listed 2026-05-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.6-flash","display_name":"Gemini 3.6 Flash","family":{"id":"gemini","vendor":"Google"},"generation":"3.6","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.6-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-07-21","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gemini-3.8-flash","aliases":[],"intelligence":{"index":78.48,"inputs":[{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"trust":3,"source_url":"https://arcprize.org/results/gemini-3-6-flash","config":"cost_per_task_usd=0.6060796625000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-6-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1483,"score_norm":90.64,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.43,"score_norm":96.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.329, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":40.82,"score_norm":57.09,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gemini 3.6 flash (high)"},{"benchmark":"livebench","score_raw":73.59,"score_norm":74.64,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: high"},{"benchmark":"mmlu_pro","score_raw":89.28,"score_norm":82.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.305, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.804, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":73.78,"score_norm":80.92,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.633, task=overall, updated=2026-08-19"}],"coverage":0.873,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.75,"completion_usd_per_m":3.75,"cache_read_usd_per_m":0.075,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.6-flash","as_of":"2026-10-06","blended_usd_per_m":1.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":29,"score_norm":29,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":40.51,"score_norm":40.51,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":69.2,"score_norm":69.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34,"score_norm":34,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":51.58,"score_norm":51.58,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.79 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":22.13,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":91.17,"score_norm":null,"config":"cost_per_task_usd=0.33563630999999994, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gemini-3-6-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/gemini-3-6-flash","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"cost_per_task_usd=0.6060796625000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gemini-3-6-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/gemini-3-6-flash","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":60.42,"score_norm":61.58,"config":"run gemini-3-6-flash-high, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1483,"score_norm":90.64,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1480.8,"score_norm":89.66,"config":"18018 votes, arena entry gemini-3.6-flash, leaderboard text-overall-style_control, rank 20, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1484,"score_norm":91.09,"config":"arena entry gemini-3.6-flash-high, rank 16, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1456.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1547.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1279.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1285,"score_norm":null,"config":"3119 votes, arena entry gemini-3.6-flash, leaderboard vision-overall-style_control, rank 13, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1536.3,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-30","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1538.9,"score_norm":null,"config":"6464 votes, arena entry gemini-3.6-flash, code/overall, leaderboard webdev-overall-raw, rank 21, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":51.08,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.5107699915638565 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"charxiv_reasoning","score_raw":89.4,"score_norm":null,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"charxiv_reasoning","score_raw":85.2,"score_norm":null,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":10.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.105714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":49,"score_norm":null,"config":"1.1, long-horizon software engineering (datacurve)","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1217,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1288,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1295,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1300,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1179,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1275,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1148,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1195,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1146,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1303,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1187,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdm_mrcr_v2","score_raw":54,"score_norm":null,"config":"8-needle, 1m pointwise","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdm_mrcr_v2","score_raw":91.8,"score_norm":null,"config":"8-needle, 128k average","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":46.11,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.46110500000000004 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1421,"score_norm":null,"config":"elo score, knowledge work","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.43,"score_norm":96.38,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.329, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.434,"score_norm":96.39,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.83,"score_norm":95.54,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.928282828282828 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.83,"score_norm":95.54,"config":"artificial analysis eval, gemini 3.6 flash (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":85.15,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.851498046026921 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":40.82,"score_norm":57.09,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.408248378127896 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":40.82,"score_norm":57.09,"config":"artificial analysis eval, gemini 3.6 flash (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":73.59,"score_norm":74.64,"config":"livebench-2026-06-25 release, run: high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":73.59,"score_norm":74.64,"config":"global average, livebench-2026-06-25 release, run gemini-3.6-flash-high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":14.44,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.144444444444444 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mle_bench","score_raw":63.9,"score_norm":null,"config":"machine learning engineering","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.28,"score_norm":82.13,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.305, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.276,"score_norm":82.13,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":83.24,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.832369942196532 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":83,"score_norm":null,"config":"agentic computer use, verified","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.66,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.52662037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":58.7,"score_norm":89.5,"config":"diverse agentic coding tasks; vendor model card","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.804, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":79.6,"score_norm":77.33,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":29.9,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.298969072164948 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":73.78,"score_norm":80.92,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=1.633, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":73.783,"score_norm":80.92,"config":"high reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.775280898876405 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"config":"artificial analysis eval, gemini 3.6 flash (high), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78,"score_norm":85.88,"config":"terminus-2 harness","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-6-flash/","measured_date":"2026-07-21","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://ai.google.dev/gemini-api/docs/changelog","publisher":"Google"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.6-flash","publisher":"OpenRouter (listed 2026-07-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-3.7-flash","display_name":"Gemini 3.7 Flash","family":{"id":"gemini","vendor":"Google"},"generation":"3.7","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-3.7-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","audio","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-08-13","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gemini-3.8-flash","aliases":[],"intelligence":{"index":81.31,"inputs":[{"benchmark":"arc_agi_2","score_raw":84.58,"score_norm":88.42,"trust":3,"source_url":"https://arcprize.org/results/google-gemini-3-7-flash","config":"cost_per_task_usd=0.24944951875000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=google-gemini-3-7-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1487.7,"score_norm":92.74,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":93.94,"score_norm":97.1,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.488, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":47.87,"score_norm":67.46,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","config":"gemini 3.7 flash (high), no tools"},{"benchmark":"livebench","score_raw":78.83,"score_norm":86.29,"trust":3,"source_url":"https://livebench.ai/table_2026_06_25.csv","config":"2026-06-25 question release, gemini-3.7-flash-high, global average of 7 category averages"},{"benchmark":"mmlu_pro","score_raw":90.12,"score_norm":83.53,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.296, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":80.8,"score_norm":78.67,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.763, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":32.78,"score_norm":17.04,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","config":"gemini 3.7 flash (high), tau^3-banking (artificial analysis edition)"},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.649, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.75,"completion_usd_per_m":3.75,"cache_read_usd_per_m":0.075,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-3.7-flash","as_of":"2026-10-06","blended_usd_per_m":1.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":35.2,"score_norm":35.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":45.1,"score_norm":45.1,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.5,"score_norm":71.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.6,"score_norm":39.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":56.03,"score_norm":56.03,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=medium, reasoning=true, run: medium, source value 0.81 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":26.48,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":60,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.6 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":95.5,"score_norm":null,"config":"cost_per_task_usd=0.12013233, eval=arc-agi-1 semi-private (v1_semi_private), model_group=google-gemini-3-7-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/google-gemini-3-7-flash","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":84.58,"score_norm":88.42,"config":"cost_per_task_usd=0.24944951875000002, eval=arc-agi-2 semi-private (v2_semi_private), model_group=google-gemini-3-7-flash, org=google, run: high, setting=evaluation conducted with 'high' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/google-gemini-3-7-flash","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1487.7,"score_norm":92.74,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1489.9,"score_norm":93.72,"config":"5718 votes, arena entry andwise-evfd, leaderboard text-overall-style_control, rank 9, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1490.4,"score_norm":93.94,"config":"5744 votes, gemini-3.7-flash-high, rank 9, text arena","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1296,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1591.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1587,"score_norm":null,"config":"2552 votes, arena entry andwise-evfd, code/overall, leaderboard webdev-overall-raw, rank 10, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":62.75,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.6274556199710792 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"charxiv_reasoning","score_raw":84.5,"score_norm":null,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"code_arena","score_raw":1588,"score_norm":null,"config":"elo, web development","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":14.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":65.3,"score_norm":null,"config":"1.1, long-horizon software engineering","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1325,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1235,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1255,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1288,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1315,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1322,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1207,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1334,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1264,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1227,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1285,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1214,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1311,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"frontiercode_1_1_main","score_raw":43.6,"score_norm":null,"config":"production code quality","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdm_mrcr_v2_8_needle","score_raw":97,"score_norm":null,"config":"128k average","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":51.58,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.515775 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1525,"score_norm":null,"config":"elo, knowledge work","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.94,"score_norm":97.1,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.488, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.55,"score_norm":97.96,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.945454545454545 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.55,"score_norm":97.96,"config":"gemini 3.7 flash (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.5,"score_norm":97.89,"config":"gemini 3.7 flash (high), artificial analysis harness","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/gpqa-diamond","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":90.66,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.906643508467217 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":47.87,"score_norm":67.46,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.478683966635774 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":47.87,"score_norm":67.46,"config":"gemini 3.7 flash (high), no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle_verified","score_raw":53.6,"score_norm":null,"config":"multidisciplinary expert reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":78.83,"score_norm":86.29,"config":"2026-06-25 question release, gemini-3.7-flash-high, global average of 7 category averages","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/table_2026_06_25.csv","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":78.83,"score_norm":86.29,"config":"livebench-2026-06-25 release, run: high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"lvbench","score_raw":85.4,"score_norm":null,"config":"long video understanding","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":15,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.15 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":90.12,"score_norm":83.53,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.296, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":90.121,"score_norm":83.54,"config":"reasoning effort: high","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":85.49,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.854913294797688 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":47.9,"score_norm":null,"config":"2.0, agentic computer use","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":57.87,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, scicode, source value 0.578703703703704 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.8,"score_norm":78.67,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.763, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.8,"score_norm":78.67,"config":"500 tasks, max_output_tokens 65536, reasoning effort: high","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":32.78,"score_norm":17.04,"config":"gemini 3.7 flash (high), tau^3-banking (artificial analysis edition)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":35.46,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.354639175257732 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.53,"score_norm":85.33,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=0.649, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.857677902621723 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.77,"score_norm":95.02,"config":"2.1, gemini 3.7 flash (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-3-7-flash","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":14.9,"score_norm":11.65,"config":"general agent capabilities","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":85.8,"score_norm":95.06,"config":"agentic terminal coding, as published on model card (aug 2026)","trust":1,"source_type":"vendor_claim","source_url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://ai.google.dev/gemini-api/docs/models","publisher":"Google"},{"url":"https://deepmind.google/models/model-cards/gemini-3-7-flash/","publisher":"Google DeepMind"},{"url":"https://openrouter.ai/api/v1/models#google/gemini-3.7-flash","publisher":"OpenRouter (listed 2026-08-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.6","display_name":"GLM 4.6","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.6","size_class":"frontier","params_total_b":356.79,"params_active_b":32,"params_exact":356785898816,"params_source":"hf_safetensors","vram_est":{"bf16_gb":714,"int8_gb":378,"int4_gb":214,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-4.6","openrouter_id":"z-ai/glm-4.6","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2025-09-30","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":[],"superseded_by":"glm-5.3","aliases":["glm-4.6-355b-a32b"],"intelligence":{"index":59.9,"inputs":[{"benchmark":"aime","score_raw":92.71,"score_norm":90.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.605, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1424.1,"score_norm":64.39,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":74.5,"score_norm":69.72,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.197, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":14.46,"score_norm":18.32,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.144578313253012 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.2,"score_norm":70.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.375, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":76.9,"score_norm":75.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-6","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.769005847953216 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":24.5,"score_norm":22.94,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.6428,"subset_bias":{"offset":2.78,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.43,"completion_usd_per_m":1.75,"cache_read_usd_per_m":0.08,"context_length":204800,"source":"openrouter","source_model_id":"z-ai/glm-4.6","as_of":"2026-10-06","blended_usd_per_m":0.76,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":713.58,"kv_cache_gb_per_8k":3.087,"min_serve_gb":752.346,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":713.577,"kv_cache_gb_per_8k":3.087,"min_serve_gb":752.343,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.6/tree/main"},{"quant":"fp8","weights_gb":378.197,"kv_cache_gb_per_8k":3.087,"min_serve_gb":400.194,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":214.074,"kv_cache_gb_per_8k":3.087,"min_serve_gb":227.865,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":378.197,"kv_cache_gb_per_8k":3.087,"min_serve_gb":400.194,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":18.6,"score_norm":18.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":18.62,"score_norm":18.62,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":45.8,"score_norm":45.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":18.5,"score_norm":18.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.3,"score_norm":29.3,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":55.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.553333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":92.71,"score_norm":90.89,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.605, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":86,"score_norm":82.5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.86 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1424.1,"score_norm":64.39,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1424.3,"score_norm":64.48,"config":"35830 votes, arena entry glm-4.6, leaderboard text-overall-style_control, rank 107, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1339.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1340.2,"score_norm":null,"config":"6606 votes, arena entry glm-4.6, code/overall, leaderboard webdev-overall-raw, rank 90, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1144,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1099,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1171,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1028,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1163,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1093,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1108,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1154,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1181,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":21.69,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.21694 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.5,"score_norm":69.72,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.197, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.98,"score_norm":74.62,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.77979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":14.46,"score_norm":18.32,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.144578313253012 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":43.4,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.434013605442177 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":69.52,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.695238095238095 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.2,"score_norm":70.33,"config":"edition=mmlu pro v1, runner=platform, stderr=0.375, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":38.43,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.384259259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":9.67,"score_norm":7.78,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":76.9,"score_norm":75.87,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.769005847953216 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":13.4,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.134020618556701 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.5,"score_norm":22.94,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2025-11-01","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":49.44,"score_norm":52.28,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.49438202247191 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.79,"score_norm":27.99,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.287878787878788 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.6","publisher":"OpenRouter (listed 2025-09-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-4.7","display_name":"GLM 4.7","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"4.7","size_class":"frontier","params_total_b":358.34,"params_active_b":32,"params_exact":358337791296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":717,"int8_gb":380,"int4_gb":215,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-4.7","openrouter_id":"z-ai/glm-4.7","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2025-12-22","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"glm-5.3","aliases":["glm-4.7-355b","glm-4.7-355b-a32b"],"intelligence":{"index":68.33,"inputs":[{"benchmark":"aime","score_raw":93.33,"score_norm":91.66,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.445, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1441.5,"score_norm":72.15,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":80.05,"score_norm":77.54,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.006, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":27.39,"score_norm":37.34,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.27386468952734 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.74,"score_norm":71.23,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.371, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":69.4,"score_norm":66,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.063, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-4-7","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":33.4,"score_norm":33.41,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":2.2,"cache_read_usd_per_m":0.11,"context_length":204800,"source":"openrouter","source_model_id":"z-ai/glm-4.7","as_of":"2026-10-06","blended_usd_per_m":1,"endpoint_expires_at":"2026-12-31"},"memory":[{"quant":"fp16","weights_gb":716.68,"kv_cache_gb_per_8k":3.087,"min_serve_gb":755.601,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":716.681,"kv_cache_gb_per_8k":3.087,"min_serve_gb":755.602,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-4.7/tree/main"},{"quant":"fp8","weights_gb":379.84,"kv_cache_gb_per_8k":3.087,"min_serve_gb":401.919,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":215.004,"kv_cache_gb_per_8k":3.087,"min_serve_gb":228.841,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":379.84,"kv_cache_gb_per_8k":3.087,"min_serve_gb":401.919,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":26.2,"score_norm":26.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":26.22,"score_norm":26.22,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":45.3,"score_norm":45.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":22.2,"score_norm":22.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.46,"score_norm":34.46,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.68 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":93.33,"score_norm":91.66,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.445, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":95,"score_norm":93.75,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.95 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.5,"score_norm":72.15,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.6,"score_norm":72.19,"config":"12155 votes, arena entry glm-4.7, leaderboard text-overall-style_control, rank 79, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1434.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1434.3,"score_norm":null,"config":"3814 votes, arena entry glm-4.7, code/overall, leaderboard webdev-overall-raw, rank 56, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0171020408163265 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1206,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1128,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1223,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1204,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1048,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1201,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1058,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1102,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1142,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1206,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1232,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":33.45,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.33453499999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.05,"score_norm":77.54,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.006, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.86,"score_norm":85.72,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.858585858585858 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.7,"score_norm":85.49,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.39,"score_norm":37.34,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.27386468952734 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.8,"score_norm":60,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":24.8,"score_norm":33.53,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.89,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.67891156462585 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":89.42,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.894179894179894 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.74,"score_norm":71.23,"config":"edition=mmlu pro v1, runner=platform, stderr=0.371, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":45.14,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.451388888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.4,"score_norm":66,"config":"edition=swe-bench verified v1, runner=external, stderr=2.063, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.8,"score_norm":70.89,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-4.7","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":12.16,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.121649484536082 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.4,"score_norm":33.41,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2026-01-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.82,"score_norm":31.55,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.318181818181818 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":45.32,"score_norm":47.44,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.453183520599251 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-4-7","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-4.7","publisher":"OpenRouter (listed 2025-12-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5","display_name":"GLM-5","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5","size_class":"frontier","params_total_b":753.86,"params_active_b":40,"params_exact":753864139008,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1508,"int8_gb":799,"int4_gb":452,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-5","openrouter_id":"z-ai/glm-5","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2026-02-12","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"glm-5.3","aliases":["glm-5-744b","glm-5-744b-a40b"],"intelligence":{"index":65,"inputs":[{"benchmark":"aime","score_raw":95.83,"score_norm":94.79,"trust":1,"source_url":"https://matharena.ai/?comp=aime--aime_2026","config":"2026"},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.27, eval=arc-agi-2 semi-private (v2_semi_private), org=z.ai, system_type=cot"},{"benchmark":"arena_elo","score_raw":1457.6,"score_norm":79.32,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":82.02,"score_norm":80.31,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.82020202020202 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":29.29,"score_norm":40.13,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.292863762743281 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":72.8,"score_norm":69.78,"trust":2,"source_url":"https://www.swebench.com/","config":"high reasoning, official"},{"benchmark":"tau_bench","score_raw":98.25,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.982456140350877 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":52.4,"score_norm":55.76,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.7619,"subset_bias":{"offset":-1.41,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.6,"completion_usd_per_m":1.92,"cache_read_usd_per_m":0.12,"context_length":204800,"source":"openrouter","source_model_id":"z-ai/glm-5","as_of":"2026-10-06","blended_usd_per_m":0.9299999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1507.72,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1583.842,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1507.736,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1583.859,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-5/tree/main"},{"quant":"fp8","weights_gb":799.092,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.783,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":452.316,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":475.668,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":799.092,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.783,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":27.9,"score_norm":27.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.55,"score_norm":40.55,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":50,"score_norm":50,"config":"v4.0","trust":1,"source_type":"vendor_claim","source_url":"https://arxiv.org/html/2602.15763v1","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":70.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.706666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.27,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.83,"score_norm":94.79,"config":"2026","trust":1,"source_type":"aggregator","source_url":"https://matharena.ai/?comp=aime--aime_2026","measured_date":"2026-02-18","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92.7,"score_norm":90.88,"config":"aime 2026 i","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":14.45,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.14454277286135694 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":44.67,"score_norm":null,"config":"cost_per_task_usd=0.17, eval=arc-agi-1 semi-private (v1_semi_private), org=z.ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"config":"cost_per_task_usd=0.27, eval=arc-agi-2 semi-private (v2_semi_private), org=z.ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1457.6,"score_norm":79.32,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1456.8,"score_norm":78.97,"config":"27850 votes, arena entry glm-5, leaderboard text-overall-style_control, rank 55, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1434,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1435.5,"score_norm":null,"config":"7068 votes, arena entry glm-5-webdev, code/overall, leaderboard webdev-overall-raw, rank 55, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":75.9,"score_norm":null,"config":"with hierarchical context management","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":62,"score_norm":null,"config":"no context management","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"browsecomp_zh","score_raw":72.7,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.02 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"cybergym","score_raw":43.2,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1244,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1161,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1158,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1252,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1121,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1247,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1144,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1147,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1133,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1167,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1254,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.02,"score_norm":80.31,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.82020202020202 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86,"score_norm":85.92,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":29.29,"score_norm":40.13,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.292863762743281 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":50.4,"score_norm":71.18,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":30.5,"score_norm":41.91,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":96.9,"score_norm":null,"config":"hmmt nov. 2025","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":72.31,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.72312925170068 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"imoanswerbench","score_raw":82.5,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"lmarena_text_arena_elo","score_raw":1452,"score_norm":null,"config":"#1 open-weight, #11 overall at time of writing","trust":0,"source_type":"community","source_url":"https://huggingface.co/blog/mlabonne/glm-5","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mcp_atlas","score_raw":67.8,"score_norm":null,"config":"public set","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.18,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.461805555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":73.3,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.8,"score_norm":69.78,"config":"high reasoning, official","trust":2,"source_type":"aggregator","source_url":"https://www.swebench.com/","measured_date":"2026-03-24","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.8,"score_norm":75.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":77.8,"score_norm":75.33,"config":"z.ai reported number","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5/","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":98.25,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.982456140350877 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":89.7,"score_norm":92.93,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.4,"score_norm":55.76,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2026-02-23","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.18,"score_norm":44.92,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.431818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.2,"score_norm":60.24,"config":"terminus 2 harness (standard tasks)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tool_decathlon","score_raw":38,"score_norm":null,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5","measured_date":"2026-02-11","observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://z.ai/blog/glm-5","publisher":"Zhipu AI"},{"url":"https://huggingface.co/zai-org/GLM-5","publisher":"Hugging Face"},{"url":"https://arxiv.org/html/2602.15763v1","publisher":"arXiv"},{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5","publisher":"OpenRouter (listed 2026-02-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"glm-5.1","display_name":"GLM-5.1","family":{"id":"glm","vendor":"Zhipu AI (Z.ai)"},"generation":"5.1","size_class":"frontier","params_total_b":753.86,"params_active_b":39,"params_exact":753864139008,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1508,"int8_gb":799,"int4_gb":452,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"zai-org/GLM-5.1","openrouter_id":"z-ai/glm-5.1","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2026-04-07","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"glm-5.3","aliases":["glm-5.1-reasoning"],"intelligence":{"index":75.17,"inputs":[{"benchmark":"aime","score_raw":91.88,"score_norm":89.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.438, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1464.6,"score_norm":82.44,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":84.52,"score_norm":83.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.821, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":30.07,"score_norm":41.28,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-1","config":"artificial analysis eval, glm-5.1 (reasoning)"},{"benchmark":"mmlu_pro","score_raw":86.9,"score_norm":78.17,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.334, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":76.4,"score_norm":73.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.901, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/glm-5-1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":56.93,"score_norm":61.09,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=1.35, task=overall, updated=2026-08-19"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.4,"completion_usd_per_m":4.4,"cache_read_usd_per_m":0.26,"context_length":204800,"source":"openrouter","source_model_id":"z-ai/glm-5.1","as_of":"2026-10-06","blended_usd_per_m":2.15,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1507.72,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1583.842,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":1507.736,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":1583.859,"method":"vendor","source_url":"https://huggingface.co/zai-org/GLM-5.1/tree/main"},{"quant":"fp8","weights_gb":799.092,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.783,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":452.316,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":475.668,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":799.092,"kv_cache_gb_per_8k":0.7361,"min_serve_gb":839.783,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":23.9,"score_norm":23.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":30.56,"score_norm":30.56,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":55.8,"score_norm":55.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":26.1,"score_norm":26.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.97,"score_norm":40.97,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.68 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.85,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.88,"score_norm":89.85,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.438, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":95.3,"score_norm":94.12,"config":"2026, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://z.ai/blog/glm-5.1","measured_date":"2026-04-07","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":95.3,"score_norm":94.12,"config":"2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1464.6,"score_norm":82.44,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1467.8,"score_norm":83.87,"config":"42267 votes, arena entry glm-5.1-text, leaderboard text-overall-style_control, rank 39, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1508.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1508.9,"score_norm":null,"config":"9296 votes, arena entry glm-5.1-code, code/overall, leaderboard webdev-overall-raw, rank 31, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":4.57,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0457142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1336,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1165,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1205,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1245,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1194,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1141,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1366,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1166,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1275,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-09-28","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1053,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1169,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1158,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1241,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1258,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1217,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1292,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1168,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1290,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":37.89,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.37890999999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.52,"score_norm":83.83,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.821, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.52,"score_norm":83.83,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.77,"score_norm":87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.867676767676768 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.77,"score_norm":87,"config":"artificial analysis eval, glm-5.1 (reasoning)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.2,"score_norm":86.2,"config":"vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://z.ai/blog/glm-5.1","measured_date":"2026-04-07","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.2,"score_norm":86.2,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.07,"score_norm":41.28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.300741427247451 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.07,"score_norm":41.28,"config":"artificial analysis eval, glm-5.1 (reasoning)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":52.3,"score_norm":73.97,"config":"vendor-reported, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://z.ai/blog/glm-5.1","measured_date":"2026-04-07","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":31,"score_norm":42.65,"config":"no tools, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://z.ai/blog/glm-5.1","measured_date":"2026-04-07","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":52.3,"score_norm":73.97,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":31,"score_norm":42.65,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":76.26,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.762585034013605 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":40.25,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.402542372881356 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.9,"score_norm":78.17,"config":"edition=mmlu pro v1, runner=platform, stderr=0.334, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.9,"score_norm":78.17,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":43.75,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.4375 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":58.4,"score_norm":89,"config":"high reasoning","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.4,"score_norm":73.78,"config":"edition=swe-bench verified v1, runner=external, stderr=1.901, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.4,"score_norm":73.78,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":13.61,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.136082474226804 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.93,"score_norm":61.09,"config":"edition=terminal-bench 2.1, runner=external, stderr=1.35, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":56.93,"score_norm":61.09,"config":"overall, terminal-bench 2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.18,"score_norm":44.92,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.431818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.8,"score_norm":66.82,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.617977528089888 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.8,"score_norm":66.82,"config":"artificial analysis eval, glm-5.1 (reasoning), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/glm-5-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.5,"score_norm":68.82,"config":"2.0, terminus-2 harness, vendor-reported","trust":1,"source_type":"vendor_claim","source_url":"https://z.ai/blog/glm-5.1","measured_date":"2026-04-07","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":69,"score_norm":75.29,"config":"2.0, agent: terminus 2(claude code)","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":63.5,"score_norm":68.82,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/zai-org/GLM-5.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://z.ai/model-api","publisher":"Zhipu AI"},{"url":"https://openrouter.ai/api/v1/models#z-ai/glm-5.1","publisher":"OpenRouter (listed 2026-04-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4-o","display_name":"GPT-4o","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4o","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-11-20","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":["gpt-4o","gpt-4o-2024-05-13","gpt-4o-2024-08-06","gpt-4o-2024-11-20"],"intelligence":{"index":20.99,"inputs":[{"benchmark":"aime","score_raw":25.67,"score_norm":7.09,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, reasoning=false, run: march 2025, source value 0.256666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.08, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1335.5,"score_norm":24.9,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":65.45,"score_norm":56.97,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, gpqa diamond, reasoning=false, run: march 2025, source value 0.654545454545455 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.97,"score_norm":2.9,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, humanity's last exam, reasoning=false, run: march 2025, source value 0.0397 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":28.95,"score_norm":11.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-2024-08-06","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: aug '24, source value 0.289473684210526 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: nov '24, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6031,"subset_bias":{"offset":-5.07,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":10,"cache_read_usd_per_m":1.25,"context_length":128000,"source":"openrouter","source_model_id":"openai/gpt-4o-2024-11-20","as_of":"2026-10-06","blended_usd_per_m":4.375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":24.2,"score_norm":24.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-2024-05-13","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.3,"score_norm":7.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-2024-05-13","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.7,"score_norm":7.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-2024-08-06","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":8.4,"score_norm":8.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-2024-11-20","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":12.29,"score_norm":12.29,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, chatgpt-4o-latest, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: march 2025","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":55,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: chatgpt, source value 0.55 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":25.67,"score_norm":7.09,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, reasoning=false, run: march 2025, source value 0.256666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":4.5,"score_norm":null,"config":"cost_per_task_usd=0.05, eval=arc-agi-1 semi-private (v1_semi_private), org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.08, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1335.5,"score_norm":24.9,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1118.3,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: nov '24, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":884,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":866,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":864,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":919,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":892,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":838,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":65.45,"score_norm":56.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, gpqa diamond, reasoning=false, run: march 2025, source value 0.654545454545455 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.97,"score_norm":2.9,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, humanity's last exam, reasoning=false, run: march 2025, source value 0.0397 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":35.99,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, run: aug '24, source value 0.359863945578231 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-2024-08-06","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":42.54,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, livecodebench, reasoning=false, run: march 2025, source value 0.425396825396825 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":56.3,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, run: aug '24, source value 0.563005780346821 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-2024-08-06","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":36.57,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, chatgpt-4o-latest, effort=none, reasoning=false, run: march 2025, scicode, source value 0.365740740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-chatgpt-03-25","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":28.95,"score_norm":11.93,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: aug '24, source value 0.289473684210526 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-2024-08-06","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":8.33,"score_norm":3.92,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: nov '24, source value 0.0833333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-2024-11-20","publisher":"OpenRouter (listed 2024-11-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4.1","display_name":"GPT-4.1","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4.1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1047576,"release_date":"2025-04-14","release_status":"released","knowledge_cutoff":"2024-06-30","status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":["gpt-4.1-2025-04-14"],"intelligence":{"index":36.49,"inputs":[{"benchmark":"aime","score_raw":39.58,"score_norm":24.48,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.386, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":0.42,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0691, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1414.8,"score_norm":60.25,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":65.4,"score_norm":56.9,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.402, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":4.18,"score_norm":3.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0418 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.385, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":47.08,"score_norm":36.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.470760233918129 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":8,"cache_read_usd_per_m":0.5,"context_length":1047576,"source":"openrouter","source_model_id":"openai/gpt-4.1","as_of":"2026-10-06","blended_usd_per_m":3.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.7,"score_norm":12.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.61,"score_norm":19.61,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":64.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.643333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":39.58,"score_norm":24.48,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.386, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":34.67,"score_norm":18.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.346666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":5.5,"score_norm":null,"config":"cost_per_task_usd=0.039, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0.42,"score_norm":0,"config":"cost_per_task_usd=0.0691, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1414.8,"score_norm":60.25,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1414,"score_norm":59.89,"config":"51118 votes, arena entry gpt-4.1-2025-04-14, leaderboard text-overall-style_control, rank 126, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1214,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1213.9,"score_norm":null,"config":"41710 votes, arena entry gpt-4.1-2025-04-14, leaderboard vision-overall-style_control, rank 66, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":865,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1031,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1111,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1092,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1007,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1045,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":65.4,"score_norm":56.9,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.402, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.57,"score_norm":58.55,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.665656565656566 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.18,"score_norm":3.21,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0418 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":42.99,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.429931972789116 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":45.71,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.457142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.5,"score_norm":67.5,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.385, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":61.21,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.612138728323699 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":38.08,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.380787037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":47.08,"score_norm":36.11,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.470760233918129 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":13.64,"score_norm":10.16,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.136363636363636 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1","publisher":"OpenRouter (listed 2025-04-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4.1-mini","display_name":"GPT-4.1 Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4.1","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4.1-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1047576,"release_date":"2025-04-14","release_status":"released","knowledge_cutoff":"2024-06-30","status":"superseded","supersedes":[],"superseded_by":"gpt-5.4-mini","aliases":["gpt-4.1-mini-2025-04-14"],"intelligence":{"index":36.58,"inputs":[{"benchmark":"aime","score_raw":49.38,"score_norm":36.73,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.122, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0139, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1382.7,"score_norm":45.94,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":67.93,"score_norm":60.46,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.351, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":5.02,"score_norm":4.44,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0502 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":77.22,"score_norm":62.03,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.405, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":52.92,"score_norm":43.89,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.529239766081871 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":10.11,"score_norm":6.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.101123595505618 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":1.6,"cache_read_usd_per_m":0.1,"context_length":1047576,"source":"openrouter","source_model_id":"openai/gpt-4.1-mini","as_of":"2026-10-06","blended_usd_per_m":0.7000000000000001,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.8,"score_norm":1.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.79,"score_norm":1.79,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":20.2,"score_norm":20.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":10.2,"score_norm":10.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.82,"score_norm":14.82,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":45.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.453333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":49.38,"score_norm":36.73,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.122, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":46.33,"score_norm":32.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.463333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":3.5,"score_norm":null,"config":"cost_per_task_usd=0.0078, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.0139, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1382.7,"score_norm":45.94,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1382.8,"score_norm":45.98,"config":"39399 votes, arena entry gpt-4.1-mini-2025-04-14, leaderboard text-overall-style_control, rank 167, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1200.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1202.1,"score_norm":null,"config":"40954 votes, arena entry gpt-4.1-mini-2025-04-14, leaderboard vision-overall-style_control, rank 72, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":852,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":999,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1041,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1083,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":968,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1004,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0.24,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0.0023950000000000104 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.93,"score_norm":60.46,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.351, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":66.36,"score_norm":58.25,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.663636363636364 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":5.02,"score_norm":4.44,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0502 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":38.3,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.382993197278912 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":48.25,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.482539682539683 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":77.22,"score_norm":62.03,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.405, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":58.73,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.58728323699422 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.39,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.403935185185185 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":52.92,"score_norm":43.89,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.529239766081871 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":5.36,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0536082474226804 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":7.58,"score_norm":3.04,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":10.11,"score_norm":6.01,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.101123595505618 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-mini","publisher":"OpenRouter (listed 2025-04-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4.1-nano","display_name":"GPT-4.1 Nano","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4.1","size_class":"frontier-nano","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4.1-nano","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1047576,"release_date":"2025-04-14","release_status":"released","knowledge_cutoff":"2024-06-30","status":"superseded","supersedes":[],"superseded_by":"gpt-5.4-nano","aliases":["gpt-4.1-nano-2025-04-14"],"intelligence":{"index":17.8,"inputs":[{"benchmark":"aime","score_raw":26.46,"score_norm":8.08,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.814, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0036, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1322.3,"score_norm":19.02,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":50.76,"score_norm":36.28,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.457, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":3.75,"score_norm":2.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0375347544022243 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":63.48,"score_norm":39.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.462, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":17.25,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.172514619883041 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.7222,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.4,"cache_read_usd_per_m":0.025,"context_length":1047576,"source":"openrouter","source_model_id":"openai/gpt-4.1-nano","as_of":"2026-10-06","blended_usd_per_m":0.17500000000000002,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.2,"score_norm":1.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.17,"score_norm":1.17,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":11.1,"score_norm":11.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.8,"score_norm":7.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.64,"score_norm":9.64,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":19.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.193333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":26.46,"score_norm":8.08,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.814, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":24,"score_norm":5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.24 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":0,"score_norm":null,"config":"cost_per_task_usd=0.0021, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.0036, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-4.1, org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1322.3,"score_norm":19.02,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1322.2,"score_norm":18.97,"config":"6103 votes, arena entry gpt-4.1-nano-2025-04-14, leaderboard text-overall-style_control, rank 238, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1088.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1089.2,"score_norm":null,"config":"1211 votes, arena entry gpt-4.1-nano-2025-04-14, leaderboard vision-overall-style_control, rank 119, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":940,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":969,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":899,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":983,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":921,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":979,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":50.76,"score_norm":36.28,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.457, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":51.21,"score_norm":36.92,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.512121212121212 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.75,"score_norm":2.57,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0375347544022243 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":32.04,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.320408163265306 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":32.59,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.325925925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":63.48,"score_norm":39.13,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.462, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":40.12,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.401156069364162 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":25.93,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.259259259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":17.25,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.172514619883041 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":3.51,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0350515463917526 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4-1-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4.1-nano","publisher":"OpenRouter (listed 2025-04-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5","display_name":"GPT-5","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-08-07","release_status":"released","knowledge_cutoff":"2024-09-30","status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":["gpt-5-2025-08-07"],"intelligence":{"index":64.45,"inputs":[{"benchmark":"aime","score_raw":93.37,"score_norm":91.71,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.305, task=overall, updated=2026-04-16, verbosity=medium"},{"benchmark":"arc_agi_2","score_raw":9.86,"score_norm":5.4,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.7302, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-reasoning, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1434.5,"score_norm":69.03,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":85.61,"score_norm":85.37,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.764, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"hle","score_raw":28.5,"score_norm":38.97,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.28498609823911 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.54,"score_norm":77.57,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.336, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"swe_bench_verified","score_raw":69,"score_norm":65.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.07, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":86.55,"score_norm":88.73,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-medium","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.865497076023392 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":37.88,"score_norm":38.68,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-medium","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.125,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":26.5,"score_norm":26.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":26.53,"score_norm":26.53,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":37.8,"score_norm":37.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23,"score_norm":23,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":35.31,"score_norm":35.31,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.763333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":93.37,"score_norm":91.71,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.305, task=overall, updated=2026-04-16, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":94.33,"score_norm":92.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.943333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":65.67,"score_norm":null,"config":"cost_per_task_usd=0.5087, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-reasoning, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":9.86,"score_norm":5.4,"config":"cost_per_task_usd=0.7302, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-reasoning, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1434.5,"score_norm":69.03,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1434.1,"score_norm":68.85,"config":"32081 votes, arena entry gpt-5-high, leaderboard text-overall-style_control, rank 90, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1210.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1225.7,"score_norm":null,"config":"40317 votes, arena entry gpt-5-chat, leaderboard vision-overall-style_control, rank 57, run: gpt-5-chat, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1417.1,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-30","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":5.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1071,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1230,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1198,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1188,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1186,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1191,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":29.14,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.29142999999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.61,"score_norm":85.37,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.764, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.35,"score_norm":85,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.853535353535353 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28.5,"score_norm":38.97,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.28498609823911 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.06,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.730612244897959 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":84.55,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.845502645502646 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.54,"score_norm":77.57,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.336, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.34,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=medium, mmmu-pro, reasoning=true, run: medium, source value 0.743352601156069 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":42.94,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.429398148148148 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69,"score_norm":65.56,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.07, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":86.55,"score_norm":88.73,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.865497076023392 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":22.06,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.220618556701031 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.88,"score_norm":38.68,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":35.21,"score_norm":35.54,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.352059925093633 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5","publisher":"OpenRouter (listed 2025-08-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5-mini","display_name":"GPT-5 Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-08-07","release_status":"released","knowledge_cutoff":"2024-05-31","status":"superseded","supersedes":[],"superseded_by":"gpt-5.4-mini","aliases":["gpt-5-mini-2025-08-07"],"intelligence":{"index":55.28,"inputs":[{"benchmark":"aime","score_raw":91.46,"score_norm":89.32,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.836, task=overall, updated=2026-04-16, verbosity=medium"},{"benchmark":"arc_agi_2","score_raw":4.44,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.1977, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-mini-reasoning, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1389.7,"score_norm":49.06,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":80.3,"score_norm":77.89,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.996, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"hle","score_raw":21.46,"score_norm":28.62,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-mini","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.214550509731233 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82.23,"score_norm":70.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.373, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"swe_bench_verified","score_raw":60.8,"score_norm":56.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.185, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":71.05,"score_norm":68.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-mini-medium","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.710526315789474 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-mini","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.25,"completion_usd_per_m":2,"cache_read_usd_per_m":0.025,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5-mini","as_of":"2026-10-06","blended_usd_per_m":0.6875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":6.8,"score_norm":6.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":19.64,"score_norm":19.64,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":15.6,"score_norm":15.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.6,"score_norm":20.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":31.63,"score_norm":31.63,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=medium, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: medium","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.71 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":91.46,"score_norm":89.32,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.836, task=overall, updated=2026-04-16, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":90.67,"score_norm":88.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.906666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":54.33,"score_norm":null,"config":"cost_per_task_usd=0.116, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-mini-reasoning, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.44,"score_norm":0,"config":"cost_per_task_usd=0.1977, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-mini-reasoning, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1389.7,"score_norm":49.06,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1389.7,"score_norm":49.06,"config":"27140 votes, arena entry gpt-5-mini-high, leaderboard text-overall-style_control, rank 157, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1183.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1182.7,"score_norm":null,"config":"29292 votes, arena entry gpt-5-mini-high, leaderboard vision-overall-style_control, rank 84, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=medium, reasoning=true, run: medium, source value 0.0142857142857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1052,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1134,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1121,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1142,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1090,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1113,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1132,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":21.75,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.21754500000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":80.3,"score_norm":77.89,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.996, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.83,"score_norm":81.45,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.828282828282828 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":21.46,"score_norm":28.62,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.214550509731233 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.44,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.754421768707483 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":83.81,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.838095238095238 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.23,"score_norm":70.38,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.373, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":70.06,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.700578034682081 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.97,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, scicode, source value 0.409722222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":60.8,"score_norm":56.44,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.185, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":71.05,"score_norm":68.07,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.710526315789474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":15.46,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.154639175257732 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":33.33,"score_norm":33.33,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.333333333333333 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.75,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0374531835205993 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-mini","publisher":"OpenRouter (listed 2025-08-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5-nano","display_name":"GPT-5 Nano","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-nano","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-nano","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-08-07","release_status":"released","knowledge_cutoff":"2024-05-31","status":"superseded","supersedes":[],"superseded_by":"gpt-5.4-nano","aliases":["gpt-5-nano-2025-08-07"],"intelligence":{"index":37.15,"inputs":[{"benchmark":"aime","score_raw":81.18,"score_norm":76.48,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.29, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":2.61,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.0295, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-nano-reasoning, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1337.5,"score_norm":25.79,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":63.38,"score_norm":54.06,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.455, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"hle","score_raw":9.5,"score_norm":11.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-nano","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.0949953660797034 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":76.07,"score_norm":60.12,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.412, task=overall, updated=2026-08-19, verbosity=medium"},{"benchmark":"tau_bench","score_raw":36.55,"score_norm":22.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-nano","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.365497076023392 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":17.42,"score_norm":14.61,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-nano-medium","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.174242424242424 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.4,"cache_read_usd_per_m":0.005,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5-nano","as_of":"2026-10-06","blended_usd_per_m":0.1375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":13,"score_norm":13,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":20.14,"score_norm":20.14,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":43.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.436666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":81.18,"score_norm":76.48,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.29, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":83.67,"score_norm":79.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.836666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":20.71,"score_norm":null,"config":"cost_per_task_usd=0.0124, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-nano-reasoning, org=openai, run: medium, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":2.61,"score_norm":0,"config":"cost_per_task_usd=0.0295, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-nano-reasoning, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1337.5,"score_norm":25.79,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1337.5,"score_norm":25.79,"config":"8292 votes, arena entry gpt-5-nano-high, leaderboard text-overall-style_control, rank 216, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1146.2,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1148.1,"score_norm":null,"config":"3984 votes, arena entry gpt-5-nano-high, leaderboard vision-overall-style_control, rank 100, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":981,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1090,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1064,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1057,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1073,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1108,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":63.38,"score_norm":54.06,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.455, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":67.58,"score_norm":59.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.675757575757576 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":9.5,"score_norm":11.03,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.0949953660797034 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":67.55,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.675510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":78.94,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.789417989417989 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":76.07,"score_norm":60.12,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.412, task=overall, updated=2026-08-19, verbosity=medium","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":60.98,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.609826589595376 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":36.57,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.365740740740741 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":36.55,"score_norm":22.07,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.365497076023392 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":17.42,"score_norm":14.61,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, source value 0.174242424242424 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-nano-medium","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-nano","publisher":"OpenRouter (listed 2025-08-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.1","display_name":"GPT-5.1","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.1","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-11-13","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":[],"intelligence":{"index":66.35,"inputs":[{"benchmark":"aime","score_raw":93.33,"score_norm":91.66,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.62, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":17.64,"score_norm":14.04,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=1.1704, eval=arc-agi-2 semi-private (v2_semi_private), high, model_group=gpt-5-1-reasoning, org=openai, run: thinking, system_type=cot"},{"benchmark":"arena_elo","score_raw":1438.5,"score_norm":70.81,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":86.62,"score_norm":86.79,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.711, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":28.5,"score_norm":38.97,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.28498609823911 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.38,"score_norm":77.3,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.337, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":69.8,"score_norm":66.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.055, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":81.87,"score_norm":82.49,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.818713450292398 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":52.43,"score_norm":55.8,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.524344569288389 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.125,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.1","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":21.6,"score_norm":21.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.63,"score_norm":21.63,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":49.4,"score_norm":49.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.7,"score_norm":24.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.47,"score_norm":37.47,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":5.4,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":93.33,"score_norm":91.66,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=0.62, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":94,"score_norm":92.5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.94 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":72.83,"score_norm":null,"config":"cost_per_task_usd=0.674, eval=arc-agi-1 semi-private (v1_semi_private), high, model_group=gpt-5-1-reasoning, org=openai, run: thinking, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":17.64,"score_norm":14.04,"config":"cost_per_task_usd=1.1704, eval=arc-agi-2 semi-private (v2_semi_private), high, model_group=gpt-5-1-reasoning, org=openai, run: thinking, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1438.5,"score_norm":70.81,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1454.7,"score_norm":78.03,"config":"40945 votes, arena entry gpt-5.1-high, leaderboard text-overall-style_control, rank 61, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1403.2,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1400.8,"score_norm":null,"config":"8220 votes, arena entry gpt-5.1, document/overall, leaderboard document-overall-raw, rank 38, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1344.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1239.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1250.5,"score_norm":null,"config":"9184 votes, arena entry gpt-5.1-high, leaderboard vision-overall-style_control, rank 41, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1341,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1340.6,"score_norm":null,"config":"10343 votes, arena entry gpt-5.1, code/overall, leaderboard webdev-overall-raw, rank 89, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":4.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0485714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1076,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1129,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1213,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1191,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1148,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1173,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1029,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1193,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":24.64,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.24639 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":86.62,"score_norm":86.79,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.711, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.27,"score_norm":87.7,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.872727272727273 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":28.5,"score_norm":38.97,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.28498609823911 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":72.86,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.728571428571429 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":86.77,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.867724867724868 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.38,"score_norm":77.3,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.337, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.49,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.754913294797688 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":43.29,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.43287037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.8,"score_norm":66.44,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.055, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":81.87,"score_norm":82.49,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.818713450292398 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":15.88,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.158762886597938 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":45.45,"score_norm":47.59,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.454545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":52.43,"score_norm":55.8,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.524344569288389 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1","publisher":"OpenRouter (listed 2025-11-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.2","display_name":"GPT-5.2","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.2","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["file","image","text"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-12-10","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":[],"intelligence":{"index":74.51,"inputs":[{"benchmark":"aime","score_raw":96.88,"score_norm":96.1,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.208, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":52.91,"score_norm":53.23,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=1.8982, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-2-2025-12-11-thinking, org=openai, run: xhigh, system_type=cot"},{"benchmark":"arena_elo","score_raw":1435.6,"score_norm":69.52,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.837, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":37.67,"score_norm":52.46,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gpt-5.2 (xhigh)"},{"benchmark":"livebench","score_raw":74.63,"score_norm":76.96,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: high"},{"benchmark":"mmlu_pro","score_raw":86.23,"score_norm":77.05,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.344, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":75.8,"score_norm":73.11,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.917, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":84.8,"score_norm":86.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.847953216374269 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":46.97,"score_norm":49.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.46969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":1,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.75,"completion_usd_per_m":14,"cache_read_usd_per_m":0.175,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.2","as_of":"2026-10-06","blended_usd_per_m":4.8125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":30.4,"score_norm":30.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":43.34,"score_norm":43.34,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.793333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.27,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=medium, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: medium","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":96.88,"score_norm":96.1,"config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.208, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":99,"score_norm":98.75,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.99 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":99,"score_norm":98.75,"config":"aime 2025, artificial analysis eval, gpt-5.2 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":86.17,"score_norm":null,"config":"cost_per_task_usd=0.96, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-2-2025-12-11-thinking, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":52.91,"score_norm":53.23,"config":"cost_per_task_usd=1.8982, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-2-2025-12-11-thinking, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":52.91,"score_norm":53.23,"config":"run gpt-5-2-2025-12-11-thinking-xhigh, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1435.6,"score_norm":69.52,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1437.6,"score_norm":70.41,"config":"47920 votes, arena entry gpt-5.2-high, leaderboard text-overall-style_control, rank 85, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.2,"score_norm":87.61,"config":"arena entry gpt-5.2-chat-latest-20260210, rank 23, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1401.1,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1405,"score_norm":null,"config":"7073 votes, arena entry gpt-5.2-high, document/overall, leaderboard document-overall-raw, rank 36, raw (no style control), run: high, vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1228.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1244.3,"score_norm":null,"config":"17865 votes, arena entry gpt-5.2-high, leaderboard vision-overall-style_control, rank 45, run: high, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1415.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1417.8,"score_norm":null,"config":"1057 votes, arena entry gpt-5.2-code, code/overall, leaderboard webdev-overall-raw, rank 60, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":11.55,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.115510204081633 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1094,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1074,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1163,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1175,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1209,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1041,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1208,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1142,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1091,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1138,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1197,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1079,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1201,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.837, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.666,"score_norm":93.9,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.3,"score_norm":91.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.903030303030303 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.3,"score_norm":91.97,"config":"artificial analysis eval, gpt-5.2 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37.67,"score_norm":52.46,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.376737720111214 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":37.67,"score_norm":52.46,"config":"artificial analysis eval, gpt-5.2 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.44,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.754421768707483 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":74.63,"score_norm":76.96,"config":"livebench-2026-06-25 release, run: high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":74.63,"score_norm":76.96,"config":"global average, livebench-2026-06-25 release, run gpt-5.2-2025-12-11-high","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":89.42,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=medium, livecodebench, reasoning=true, run: medium, source value 0.894179894179894 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.23,"score_norm":77.05,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.344, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.234,"score_norm":77.06,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":74.57,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=medium, mmmu-pro, reasoning=true, run: medium, source value 0.745664739884393 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.08,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.520833333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75.8,"score_norm":73.11,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.917, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75.8,"score_norm":73.11,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":84.8,"score_norm":86.4,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.847953216374269 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":11.13,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.111340206185567 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":46.97,"score_norm":49.38,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.46969696969697 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2","publisher":"OpenRouter (listed 2025-12-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.4","display_name":"GPT-5.4","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.4","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-03-05","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.5","aliases":[],"intelligence":{"index":81.82,"inputs":[{"benchmark":"aime","score_raw":96.67,"score_norm":95.84,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.534, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":73.95,"score_norm":76.61,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=1.52, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4, org=openai, run: xhigh, system_type=cot"},{"benchmark":"arena_elo","score_raw":1464.8,"score_norm":82.53,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.908, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":43.74,"score_norm":61.38,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gpt-5.4 (xhigh)"},{"benchmark":"livebench","score_raw":77.97,"score_norm":84.38,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh effort"},{"benchmark":"mmlu_pro","score_raw":87.48,"score_norm":79.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.421, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.848, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":87.13,"score_norm":89.51,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-4","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.871345029239766 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":78.28,"score_norm":86.21,"trust":2,"source_url":"https://artificialanalysis.ai/models","config":"artificial analysis eval, gpt-5.4 (xhigh), terminal-bench 2.1"}],"coverage":1,"subset_bias":null},"pricing":{"prompt_usd_per_m":2.5,"completion_usd_per_m":15,"cache_read_usd_per_m":0.25,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.4","as_of":"2026-10-06","blended_usd_per_m":5.625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":44.2,"score_norm":44.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":44.17,"score_norm":44.17,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.1,"score_norm":71.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39,"score_norm":39,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":53.12,"score_norm":53.12,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":77.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.776666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":5.78,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=xhigh, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":96.67,"score_norm":95.84,"config":"archived=true, edition=aime v1, reasoning_effort=xhigh, runner=platform, stderr=0.534, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":33.26,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.33259587020649 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":93.67,"score_norm":null,"config":"cost_per_task_usd=0.62, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-4, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":73.95,"score_norm":76.61,"config":"cost_per_task_usd=1.52, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":73.95,"score_norm":76.61,"config":"run gpt-5-4-xhigh, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.21,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=openai-gpt-5-4-2026-03-05-thinking, org=openai, run: high, system_type=cot, total_cost_usd=5187.41","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1464.8,"score_norm":82.53,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.5,"score_norm":87.75,"config":"60708 votes, arena entry gpt-5.4-high-no-system-prompt, leaderboard text-overall-style_control, rank 22, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1476.4,"score_norm":87.7,"config":"arena entry gpt-5.4-high, rank 22, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1471,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1470.1,"score_norm":null,"config":"29809 votes, arena entry gpt-5.4-no-system-prompt, document/overall, leaderboard document-overall-raw, rank 15, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1445.5,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1278.7,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-06","observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1280.3,"score_norm":null,"config":"21227 votes, arena entry gpt-5.4-no-system-prompt-vision, leaderboard vision-overall-style_control, rank 20, run: gpt-5.4, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1395.1,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1390.6,"score_norm":null,"config":"1951 votes, arena entry gpt-5.4-code-arena-harness, code/overall, leaderboard webdev-overall-raw, rank 71, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":23.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.2342857143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1117,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1046,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1210,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1217,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1243,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1016,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1251,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1135,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1084,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1194,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1245,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1052,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1226,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":44.7,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.447015 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.67,"score_norm":93.9,"config":"edition=gpqa diamond v1, reasoning_effort=xhigh, runner=platform, stderr=1.908, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.666,"score_norm":93.9,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.02,"score_norm":94.39,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.92020202020202 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.02,"score_norm":94.39,"config":"artificial analysis eval, gpt-5.4 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.74,"score_norm":61.38,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.437442075996293 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":43.74,"score_norm":61.38,"config":"artificial analysis eval, gpt-5.4 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":73.95,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.739455782312925 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":77.97,"score_norm":84.38,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":77.97,"score_norm":84.38,"config":"global average, livebench-2026-06-25 release, run gpt-5.4-xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.48,"score_norm":79.13,"config":"edition=mmlu pro v1, reasoning_effort=xhigh, runner=platform, stderr=0.421, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.482,"score_norm":79.14,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.44,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.784393063583815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":56.6,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.565972222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"config":"edition=swe-bench verified v1, reasoning_effort=xhigh, runner=external, stderr=1.848, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":78.2,"score_norm":75.78,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":87.13,"score_norm":89.51,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.871345029239766 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":39.59,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.395876288659794 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":57.58,"score_norm":61.86,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.575757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.28,"score_norm":86.21,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.782771535580524 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.28,"score_norm":86.21,"config":"artificial analysis eval, gpt-5.4 (xhigh), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4","publisher":"OpenRouter (listed 2026-03-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-4.20","display_name":"Grok 4.20","family":{"id":"grok","vendor":"xAI"},"generation":"4.20","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.20","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":2000000,"release_date":"2026-03-31","release_status":"released","knowledge_cutoff":"2025-09-01","status":"superseded","supersedes":[],"superseded_by":"grok-4.7","aliases":[],"intelligence":{"index":75.41,"inputs":[{"benchmark":"aime","score_raw":96.46,"score_norm":95.57,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.516, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":65.14,"score_norm":66.82,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.92, eval=arc-agi-2 semi-private (v2_semi_private), model_group=grok-4.20-beta-0309b-reasoning, org=xai, run: reasoning, system_type=cot"},{"benchmark":"arena_elo","score_raw":1474.6,"score_norm":86.9,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"26770 votes, arena entry clanker-text, leaderboard text-overall-style_control, rank 25, run: grok-4.20-beta1, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":88.64,"score_norm":89.63,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.594, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":34.52,"score_norm":47.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-20","config":"grok 4.20 0309 v2 (reasoning), no tools"},{"benchmark":"mmlu_pro","score_raw":86.25,"score_norm":77.08,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.34, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":72.2,"score_norm":69.11,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=2.006, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":96.49,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.964912280701754 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":44.2,"score_norm":46.12,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=0.991, task=overall, updated=2026-08-19"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.2,"context_length":2000000,"source":"openrouter","source_model_id":"x-ai/grok-4.20","as_of":"2026-10-06","blended_usd_per_m":1.5625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":25.7,"score_norm":25.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.44,"score_norm":37.44,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":60.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.606666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":12.95,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":96.46,"score_norm":95.57,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.516, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":96.458,"score_norm":95.57,"config":"60 questions), grok-4.20-0309-reasoning, vals aime (2024+2025","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-04-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":14.23,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.14233038348082597 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":89.5,"score_norm":null,"config":"cost_per_task_usd=0.35, eval=arc-agi-1 semi-private (v1_semi_private), model_group=grok-4.20-beta-0309b-reasoning, org=xai, run: reasoning, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":65.14,"score_norm":66.82,"config":"cost_per_task_usd=0.92, eval=arc-agi-2 semi-private (v2_semi_private), model_group=grok-4.20-beta-0309b-reasoning, org=xai, run: reasoning, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":65.14,"score_norm":66.82,"config":"arc prize leaderboard, grok-4.20-beta-0309b-reasoning, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/media/data/evaluations.json","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.09,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=xai-grok-4-20-beta-0309-reasoning, org=xai, run: beta reasoning, system_type=cot, total_cost_usd=3775","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1474.6,"score_norm":86.9,"config":"26770 votes, arena entry clanker-text, leaderboard text-overall-style_control, rank 25, run: grok-4.20-beta1, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1475,"score_norm":87.08,"config":"arena.ai text leaderboard, grok-4.20-beta1 (+/-5)","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1413.9,"score_norm":null,"config":"18666 votes, arena entry grok-4.20-beta-0309-reasoning, document/overall, leaderboard document-overall-raw, rank 34, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1189.4,"score_norm":null,"config":"53877 votes, arena entry arastradero, leaderboard search-overall-raw, rank 15, raw (no style control), search/overall, vote cutoff 2026-07-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1256.3,"score_norm":null,"config":"25114 votes, arena entry grok-4.20-beta-0309-reasoning, leaderboard vision-overall-style_control, rank 36, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1373.6,"score_norm":null,"config":"14590 votes, arena entry grok-4.20-beta-0309-reasoning-code, code/overall, leaderboard webdev-overall-raw, rank 77, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":6,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.06 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1208,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1060,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1186,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1227,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1226,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1052,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1205,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1089,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1086,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1164,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1206,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1137,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1236,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.64,"score_norm":89.63,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.594, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.636,"score_norm":89.63,"config":"grok-4.20-0309-reasoning, no tools, temp 0.7, top_p 0.95","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.48,"score_norm":89.41,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.884848484848485 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":32.39,"score_norm":44.69,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.323911028730306 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":34.52,"score_norm":47.82,"config":"grok 4.20 0309 v2 (reasoning), no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":82.93,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.829251700680272 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.25,"score_norm":77.08,"config":"edition=mmlu pro v1, runner=platform, stderr=0.34, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.254,"score_norm":77.09,"config":"grok-4.20-0309-reasoning, temp 0.7, top_p 0.95","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.18,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.731791907514451 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":44.68,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.446759259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.2,"score_norm":69.11,"config":"edition=swe-bench verified v1, runner=external, stderr=2.006, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.2,"score_norm":69.11,"config":"grok-4.20-0309-reasoning, vals harness","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":96.49,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.964912280701754 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.98,"score_norm":97.31,"config":"artificial analysis tau-squared bench (telecom), grok 4.20 0309 v2 (reasoning), tau2","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.2,"score_norm":46.12,"config":"edition=terminal-bench 2.1, runner=external, stderr=0.991, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":44.195,"score_norm":46.11,"config":"2.1, grok-4.20-0309-reasoning","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":40.91,"score_norm":42.25,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.409090909090909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-20-0309","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20","publisher":"OpenRouter (listed 2026-03-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-4.3","display_name":"Grok 4.3","family":{"id":"grok","vendor":"xAI"},"generation":"4.3","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-04-30","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"grok-4.7","aliases":[],"intelligence":{"index":70.1,"inputs":[{"benchmark":"arena_elo","score_raw":1441.4,"score_norm":72.1,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":91.41,"score_norm":93.54,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.406, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":37.21,"score_norm":51.78,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-3","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.372103799814643 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":62.25,"score_norm":49.44,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":85.84,"score_norm":76.4,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.33, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":71.4,"score_norm":68.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.023, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-3","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":41.95,"score_norm":43.47,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.622, task=overall, updated=2026-08-19"}],"coverage":0.8413,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.2,"context_length":1000000,"source":"openrouter","source_model_id":"x-ai/grok-4.3","as_of":"2026-10-06","blended_usd_per_m":1.5625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":15.5,"score_norm":15.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":24.16,"score_norm":24.16,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":42.2,"score_norm":42.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.9,"score_norm":24.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":37.95,"score_norm":37.95,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=medium, reasoning=true, run: medium, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":17.98,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":8.75,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.0875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":17.04,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.170353982300885 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.4,"score_norm":72.1,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1441.9,"score_norm":72.33,"config":"61016 votes, arena entry grok-4.3-text, leaderboard text-overall-style_control, rank 77, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1373.5,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1164.8,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1163.2,"score_norm":null,"config":"68373 votes, arena entry grok-4.3-search, leaderboard search-overall-raw, rank 21, raw (no style control), search/overall, vote cutoff 2026-07-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1242.9,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1241.1,"score_norm":null,"config":"20041 votes, arena entry grok-4.3-vision, leaderboard vision-overall-style_control, rank 47, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1356.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1356.7,"score_norm":null,"config":"12942 votes, arena entry grok-4.3-webdev, code/overall, leaderboard webdev-overall-raw, rank 86, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":8.12,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.08118872318781814 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.08 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1140,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1007,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1066,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1072,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":975,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1193,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1188,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1004,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1188,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":914,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1008,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1055,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1071,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1072,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1080,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1199,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1120,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1202,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":26.26,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.26260817666368247 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":29.93,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, run: non-reasoning, source value 0.29929999999999995 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.41,"score_norm":93.54,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.406, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.1,"score_norm":91.69,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.901010101010101 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":68.39,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.683890577507599 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":37.21,"score_norm":51.78,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.372103799814643 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":83.33,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=medium, ifbench, reasoning=true, run: medium, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":32.72,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=high, it-bench sre, reasoning=true, run: high, source value 0.327212806026365 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":62.25,"score_norm":49.44,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":10,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.1 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":85.84,"score_norm":76.4,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.33, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":78.09,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.78092485549133 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":47.34,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.47337962962963 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":71.4,"score_norm":68.22,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.023, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":12.37,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.123711340206186 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":41.95,"score_norm":43.47,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.622, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.88,"score_norm":38.68,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":39.7,"score_norm":40.82,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.397003745318352 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-3","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.3","publisher":"OpenRouter (listed 2026-04-30)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-4.5","display_name":"Grok 4.5","family":{"id":"grok","vendor":"xAI"},"generation":"4.5","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.5","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":500000,"release_date":"2026-07-16","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"grok-4.7","aliases":[],"intelligence":{"index":78.9,"inputs":[{"benchmark":"arc_agi_2","score_raw":52.64,"score_norm":52.93,"trust":3,"source_url":"https://arcprize.org/results/xai-grok-4-5","config":"cost_per_task_usd=0.78, eval=arc-agi-2 semi-private (v2_semi_private), model_group=xai-grok-4-5, org=xai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1465.8,"score_norm":82.98,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.288, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":42.68,"score_norm":59.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.42678405931418 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":75.77,"score_norm":79.49,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":89.22,"score_norm":82.03,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.31, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":86.6,"score_norm":85.11,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.525, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=4.416, task=overall, updated=2026-08-19"}],"coverage":0.873,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.3,"context_length":500000,"source":"openrouter","source_model_id":"x-ai/grok-4.5","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":41.2,"score_norm":41.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":48.85,"score_norm":48.85,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":72.4,"score_norm":72.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":38.8,"score_norm":38.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":55.76,"score_norm":55.76,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":74,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.74 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":25.32,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":35,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.35 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":87.17,"score_norm":null,"config":"cost_per_task_usd=0.31, eval=arc-agi-1 semi-private (v1_semi_private), model_group=xai-grok-4-5, org=xai, run: medium, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":52.64,"score_norm":52.93,"config":"cost_per_task_usd=0.78, eval=arc-agi-2 semi-private (v2_semi_private), model_group=xai-grok-4-5, org=xai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":52.64,"score_norm":52.93,"config":"run xai-grok-4-5-high, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":0.32,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=xai-grok-4-5, org=xai, run: medium, system_type=cot, total_cost_usd=8458.3","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-5","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1465.8,"score_norm":82.98,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1470.3,"score_norm":84.98,"config":"22030 votes, arena entry grok-4.5-text, leaderboard text-overall-style_control, rank 35, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1452,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1453.7,"score_norm":null,"config":"2435 votes, arena entry grok-4.5-document, document/overall, leaderboard document-overall-raw, rank 21, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1563,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1212.6,"score_norm":null,"config":"search","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-25","observed_at":"2026-08-25","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1279,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1282.2,"score_norm":null,"config":"5722 votes, arena entry grok-4.5-vision, leaderboard vision-overall-style_control, rank 17, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1551.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1556.4,"score_norm":null,"config":"6325 votes, arena entry grok-4.5-webdev, code/overall, leaderboard webdev-overall-raw, rank 17, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":51.44,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.5143740721186943 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":15.43,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.154285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":53,"score_norm":null,"config":"1.1, mini-swe-agent harness run by datacurve","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-5","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":62,"score_norm":null,"config":"1.0, eval by datacurve, pass@1, run with provider harnesses by artificial analysis","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-5","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1235,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1250,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1264,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1282,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1247,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1275,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1228,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1204,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1216,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1230,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1219,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1286,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1209,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1289,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":40.82,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.40823634735899733 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":51.22,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=high, gdpval-aa, reasoning=true, run: high, source value 0.51218 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.288, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":92.93,"score_norm":95.68,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.13,"score_norm":95.96,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.931313131313131 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.13,"score_norm":95.96,"config":"artificial analysis eval, grok 4.5 (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":92.42,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.924156896801274 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":42.68,"score_norm":59.82,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.42678405931418 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.68,"score_norm":59.82,"config":"artificial analysis eval, grok 4.5 (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":75.77,"score_norm":79.49,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":75.77,"score_norm":79.49,"config":"global average, livebench-2026-06-25 release, run grok-4.5","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":15,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.15 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.22,"score_norm":82.03,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.31, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.22,"score_norm":82.03,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":80.4,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.804046242774566 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":54.05,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.540509259259259 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":64.7,"score_norm":99.5,"config":"resolve rate","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-5","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":86.6,"score_norm":85.11,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=1.525, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":86.6,"score_norm":85.11,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_marathon","score_raw":29,"score_norm":null,"config":"pass@1 resolution rate","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-5","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":42.06,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.420618556701031 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=4.416, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":67.79,"score_norm":73.87,"config":"high reasoning effort, overall, terminal-bench 2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":81.65,"score_norm":90.18,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.816479400749064 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":81.65,"score_norm":90.18,"config":"artificial analysis eval, grok 4.5 (high), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-5","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":83.3,"score_norm":92.12,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-5","measured_date":"2026-07-16","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://x.ai/api","publisher":"xAI"},{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.5","publisher":"OpenRouter (listed 2026-07-08)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"grok-4.6","display_name":"Grok 4.6","family":{"id":"grok","vendor":"xAI"},"generation":"4.6","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.6","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":500000,"release_date":"2026-08-12","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"grok-4.7","aliases":["grok-4.6-reasoning"],"intelligence":{"index":83.8,"inputs":[{"benchmark":"arc_agi_2","score_raw":67.08,"score_norm":68.98,"trust":3,"source_url":"https://arcprize.org/results/xai-grok-4-6","config":"cost_per_task_usd=0.7574904499999999, eval=arc-agi-2 semi-private (v2_semi_private), model_group=xai-grok-4-6, org=xai, run: xhigh, setting=evaluation conducted with 'xhigh' reasoning effort., system_type=cot"},{"benchmark":"arena_elo","score_raw":1453.8,"score_norm":77.63,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":94.7,"score_norm":98.17,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.125, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":44.07,"score_norm":61.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/grok-4-6-xhigh","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.440685820203893 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":78.04,"score_norm":84.53,"trust":3,"source_url":"https://livebench.ai/","config":"global average, livebench-2026-06-25 release, run grok-4.6"},{"benchmark":"mmlu_pro","score_raw":89.4,"score_norm":82.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.305, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":95.6,"score_norm":95.11,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=0.918, task=overall, updated=2026-08-19"},{"benchmark":"terminal_bench","score_raw":78.28,"score_norm":86.21,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.085, task=overall, updated=2026-08-19"}],"coverage":0.873,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.5,"context_length":500000,"source":"openrouter","source_model_id":"x-ai/grok-4.6","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":53,"score_norm":53,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":58.68,"score_norm":58.68,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=high, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_briefcase","score_raw":1577,"score_norm":null,"config":"elo, grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"aa_coding_index","score_raw":76.8,"score_norm":76.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":44.3,"score_norm":44.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":60.92,"score_norm":60.92,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":78.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=low, reasoning=true, run: low, source value 0.786666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6-low","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":30.48,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=high, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"analyst_agent","score_raw":41.25,"score_norm":null,"config":"aa field analystagent, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.4125 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":57.5,"score_norm":null,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"apex_swe","score_raw":56.4,"score_norm":null,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":87.5,"score_norm":null,"config":"cost_per_task_usd=0.30111846000000003, eval=arc-agi-1 semi-private (v1_semi_private), model_group=xai-grok-4-6, org=xai, run: medium, setting=evaluation conducted with 'medium' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-6","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":67.08,"score_norm":68.98,"config":"cost_per_task_usd=0.7574904499999999, eval=arc-agi-2 semi-private (v2_semi_private), model_group=xai-grok-4-6, org=xai, run: xhigh, setting=evaluation conducted with 'xhigh' reasoning effort., system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-6","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":67.08,"score_norm":68.98,"config":"run xai-grok-4-6-xhigh, v2 semi-private eval","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arc_agi_3","score_raw":2.11,"score_norm":null,"config":"eval=arc-agi-3 semi-private (v3_semi_private), model_group=xai-grok-4-6, org=xai, run: xhigh, setting=evaluation conducted with 'xhigh' reasoning effort., system_type=cot, total_cost_usd=5612","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/results/xai-grok-4-6","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1453.8,"score_norm":77.63,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1461.2,"score_norm":80.93,"config":"3473 votes, arena entry grok-4.6-high-text, leaderboard text-overall-style_control, rank 46, run: high, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1463.6,"score_norm":82,"config":"arena entry grok-4.6-high, rank 44, style control, text overall","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1452.3,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1595.8,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1263.5,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1619.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1628.8,"score_norm":null,"config":"1523 votes, arena entry grok-4.6-high-webdev, code/overall, leaderboard webdev-overall-raw, rank 5, raw (no style control), run: high, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":19.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.197142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6-xhigh","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"cursorbench_v3_2","score_raw":69.9,"score_norm":null,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"deepswe","score_raw":65.9,"score_norm":null,"config":"1.1, grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1292,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1210,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1303,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1288,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1298,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1292,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1263,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1276,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1249,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1233,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1242,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1239,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1295,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1241,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1297,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"frontiercode_v1_1_extended","score_raw":61.3,"score_norm":null,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":63.29,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.63288 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6-xhigh","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_v2","score_raw":1753,"score_norm":null,"config":"elo, grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.7,"score_norm":98.17,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=1.125, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.697,"score_norm":98.16,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.95,"score_norm":98.52,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.94949494949495 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.95,"score_norm":98.52,"config":"artificial analysis eval, grok 4.6 (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_vals","score_raw":15.8,"score_norm":null,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":44.07,"score_norm":61.87,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.440685820203893 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6-xhigh","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42.91,"score_norm":60.16,"config":"artificial analysis eval, grok 4.6 (high)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":78.04,"score_norm":84.53,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":78.04,"score_norm":84.53,"config":"global average, livebench-2026-06-25 release, run grok-4.6","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mlcr_overall","score_raw":12.22,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.122222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.4,"score_norm":82.33,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.305, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":89.4,"score_norm":82.33,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":54.63,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=medium, reasoning=true, run: medium, scicode, source value 0.546296296296296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6-medium","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95.6,"score_norm":95.11,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=0.918, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":95.6,"score_norm":95.11,"config":"high reasoning effort, overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":50.72,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.507216494845361 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.28,"score_norm":86.21,"config":"edition=terminal-bench 2.1, reasoning_effort=high, runner=external, stderr=2.085, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":78.277,"score_norm":86.21,"config":"high reasoning effort, overall, v2.1","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.39,"score_norm":98.11,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.883895131086142 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/grok-4-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":88.39,"score_norm":98.11,"config":"artificial analysis eval, grok 4.6 (high), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":26,"score_norm":24.71,"config":"grok 4.6 high","trust":1,"source_type":"vendor_claim","source_url":"https://x.ai/news/grok-4-6","measured_date":"2026-08-12","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.x.ai/developers/models","publisher":"xAI"},{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.6","publisher":"OpenRouter (listed 2026-08-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2.5-0127","display_name":"Kimi K2.5","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2.5","size_class":"frontier-k","params_total_b":1026.88,"params_active_b":32,"params_exact":1026879376368,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2054,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"moonshotai/Kimi-K2.5","openrouter_id":"moonshotai/kimi-k2.5","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-01-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"kimi-k3","aliases":["kimi-k2.5","kimi-k2.5-1t"],"intelligence":{"index":65.32,"inputs":[{"benchmark":"arc_agi_2","score_raw":11.81,"score_norm":7.57,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.28, eval=arc-agi-2 semi-private (v2_semi_private), org=moonshot ai, system_type=cot"},{"benchmark":"arena_elo","score_raw":1450,"score_norm":75.94,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"70943 votes, arena entry kimi-k2.5-text, leaderboard text-overall-style_control, rank 64, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":87.88,"score_norm":88.56,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-5","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.878787878787879 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":30.72,"score_norm":42.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.307228915662651 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":87.1,"score_norm":78.5,"trust":1,"source_url":"https://huggingface.co/moonshotai/Kimi-K2.5","config":null},{"benchmark":"swe_bench_verified","score_raw":70.8,"score_norm":67.56,"trust":2,"source_url":"https://www.swebench.com/","config":"high reasoning"},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":43.2,"score_norm":44.94,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.45,"completion_usd_per_m":2.25,"cache_read_usd_per_m":0.07,"context_length":262144,"source":"openrouter","source_model_id":"moonshotai/kimi-k2.5","as_of":"2026-10-06","blended_usd_per_m":0.9,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2053.76,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2157.024,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":616.128,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.51,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":595.178,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":625.513,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2.5/tree/main"},{"quant":"int8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":21.7,"score_norm":21.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":21.69,"score_norm":21.69,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":46.8,"score_norm":46.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23.5,"score_norm":23.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":36.02,"score_norm":36.02,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":73,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.73 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"apex_agents","score_raw":11.5,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.11504424778761062 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":65.33,"score_norm":null,"config":"cost_per_task_usd=0.15, eval=arc-agi-1 semi-private (v1_semi_private), org=moonshot ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":11.81,"score_norm":7.57,"config":"cost_per_task_usd=0.28, eval=arc-agi-2 semi-private (v2_semi_private), org=moonshot ai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1450,"score_norm":75.94,"config":"70943 votes, arena entry kimi-k2.5-text, leaderboard text-overall-style_control, rank 64, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1429.5,"score_norm":null,"config":"19891 votes, arena entry kimi-k2.5-document, document/overall, leaderboard document-overall-raw, rank 29, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1249.7,"score_norm":null,"config":"26666 votes, arena entry kimi-k2.5-vision, leaderboard vision-overall-style_control, rank 42, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1435.9,"score_norm":null,"config":"18367 votes, arena entry kimi-k2.5-webdev, code/overall, leaderboard webdev-overall-raw, rank 54, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.0314285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1220,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1133,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1242,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1228,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1112,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1219,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1219,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1144,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1243,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1129,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1255,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":25.31,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, run: reasoning, source value 0.253065 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.88,"score_norm":88.56,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.878787878787879 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.6,"score_norm":88.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.72,"score_norm":42.24,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.307228915662651 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":50.2,"score_norm":70.88,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.5","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"ifbench","score_raw":70.2,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.702040816326531 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.1,"score_norm":78.5,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.38,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.753757225433526 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":48.96,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.489583333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":50.7,"score_norm":76.17,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":70.8,"score_norm":67.56,"config":"high reasoning","trust":2,"source_type":"aggregator","source_url":"https://www.swebench.com/","measured_date":"2026-03-02","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":14.23,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.142268041237113 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.2,"score_norm":44.94,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2026-02-04","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":45.69,"score_norm":47.87,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.456928838951311 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.5","publisher":"OpenRouter (listed 2026-01-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2.6","display_name":"Kimi K2.6","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2.6","size_class":"frontier-k","params_total_b":1026.88,"params_active_b":32,"params_exact":1026879376368,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2054,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"moonshotai/Kimi-K2.6","openrouter_id":"moonshotai/kimi-k2.6","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-04","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"kimi-k3","aliases":["kimi-k2.6-1t","kimi-k2.6-vision"],"intelligence":{"index":76.92,"inputs":[{"benchmark":"aime","score_raw":96.4,"score_norm":95.5,"trust":1,"source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","config":"thinking mode; aime 2026"},{"benchmark":"arena_elo","score_raw":1461,"score_norm":80.84,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":89.14,"score_norm":90.34,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.997, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":37.49,"score_norm":52.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-6","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: kimi k2.6, source value 0.374884151992586 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":70.54,"score_norm":67.87,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":87.57,"score_norm":79.28,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.33, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":76.2,"score_norm":73.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.906, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2-6","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: kimi k2.6, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=2.085, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.95,"completion_usd_per_m":4,"cache_read_usd_per_m":0.16,"context_length":262144,"source":"openrouter","source_model_id":"moonshotai/kimi-k2.6","as_of":"2026-10-06","blended_usd_per_m":1.7125,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2053.76,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2157.024,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":616.128,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.51,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":595.178,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":625.513,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6/tree/main"},{"quant":"int8","weights_gb":1088.493,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1143.493,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":20.5,"score_norm":20.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":31.16,"score_norm":31.16,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":61.8,"score_norm":61.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":27,"score_norm":27,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":45.14,"score_norm":45.14,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: kimi k2.6","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":76.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: kimi k2.6, source value 0.766666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":5.3,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: kimi k2.6","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":96.4,"score_norm":95.5,"config":"thinking mode; aime 2026","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"apex_agents","score_raw":28.47,"score_norm":null,"config":"aa field apexagents, apex-agents, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.284660766961652 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1461,"score_norm":80.84,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1460.7,"score_norm":80.71,"config":"37543 votes, arena entry kimi-k2.6-text, leaderboard text-overall-style_control, rank 47, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1450.7,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1450.9,"score_norm":null,"config":"11181 votes, arena entry kimi-k2.6-document, document/overall, leaderboard document-overall-raw, rank 22, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1521.4,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1264.9,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1262.9,"score_norm":null,"config":"15381 votes, arena entry kimi-k2.6-vision, leaderboard vision-overall-style_control, rank 32, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1508.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-30","observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1508.7,"score_norm":null,"config":"9589 votes, arena entry kimi-k2.6-code, code/overall, leaderboard webdev-overall-raw, rank 32, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":19.55,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.19552711052930297 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"browsecomp","score_raw":83.2,"score_norm":null,"config":"with search/browsing tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: kimi k2.6, source value 0.08 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1283,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1141,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_agentichtmlslides","score_raw":1248,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticslides","score_raw":1187,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1174,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1176,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1278,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1258,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1154,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1263,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1120,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1184,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1153,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_pptxslides","score_raw":1181,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1180,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1181,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1273,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1268,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":38.53,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.3852581319009251 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":34.58,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.34583 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.14,"score_norm":90.34,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.997, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.11,"score_norm":93.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: kimi k2.6, source value 0.911111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.5,"score_norm":92.25,"config":"thinking mode, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":84.12,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.841221595020987 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":37.49,"score_norm":52.19,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: kimi k2.6, source value 0.374884151992586 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":54,"score_norm":76.47,"config":"code-interpreter, hle-full with search, web-browsing tools, with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":34.7,"score_norm":48.09,"config":"hle-full, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hmmt_2026_feb","score_raw":92.7,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":75.99,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: kimi k2.6, source value 0.759863945578231 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"imo_answerbench","score_raw":86,"score_norm":null,"config":"thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":31.19,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, source value 0.311864406779661 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":70.54,"score_norm":67.87,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":70.5,"score_norm":67.78,"config":"global average (mean of 7 category averages), kimi-k2.6-thinking, livebench 2026-06-25 release","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/benchmarks/livebench","measured_date":"2026-06-25","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":89.6,"score_norm":null,"config":"v6","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mathvision","score_raw":87.4,"score_norm":null,"config":"avg@3, no python tool","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.57,"score_norm":79.28,"config":"edition=mmlu pro v1, runner=platform, stderr=0.33, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.2,"score_norm":78.67,"config":null,"trust":0,"source_type":"community","source_url":"https://canitrun.dev/models/kimi-k2.6/","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":79.36,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.79364161849711 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":79.4,"score_norm":null,"config":"avg@3, max-tokens 98304, no python tool","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"osworld","score_raw":73.1,"score_norm":null,"config":"thinking mode, verified","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":53.47,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: kimi k2.6, scicode, source value 0.534722222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":52.2,"score_norm":null,"config":"thinking mode; avg of 10 runs","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":76.7,"score_norm":null,"config":"thinking mode; avg of 10 runs","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":58.6,"score_norm":89.33,"config":"thinking mode; avg of 10 runs","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.2,"score_norm":73.56,"config":"edition=swe-bench verified v1, runner=external, stderr=1.906, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":80.2,"score_norm":78,"config":"thinking mode; in-house swe-agent-adapted framework; avg of 10 runs","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: kimi k2.6, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":96,"score_norm":100,"config":"tau-squared-bench telecom, tau2, vendor-reported","trust":0,"source_type":"community","source_url":"https://kili-technology.com/blog/data-story-kimi-k2-6","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":23.3,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.232989690721649 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.56,"score_norm":57.13,"config":"edition=terminal-bench 2.1, runner=external, stderr=2.085, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.94,"score_norm":45.81,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: kimi k2.6, source value 0.439393939393939 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":65.92,"score_norm":71.67,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.659176029962547 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2-6","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":66.7,"score_norm":72.59,"config":"terminus-2 agent framework, preserve thinking mode","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2.6","measured_date":"2026-04-20","observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://platform.kimi.ai/docs/models","publisher":"Moonshot AI"},{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2.6","publisher":"OpenRouter (listed 2026-04-20)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m2","display_name":"MiniMax M2","family":{"id":"minimax","vendor":"MiniMax"},"generation":"2","size_class":"frontier-m","params_total_b":228.69,"params_active_b":10,"params_exact":228689764864,"params_source":"hf_safetensors","vram_est":{"bf16_gb":457,"int8_gb":242,"int4_gb":137,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"MiniMaxAI/MiniMax-M2","openrouter_id":"minimax/minimax-m2","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2025-10-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"minimax-m2.7","aliases":["minimax-m2-230b"],"intelligence":{"index":57.62,"inputs":[{"benchmark":"aime","score_raw":78.33,"score_norm":72.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.783333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1343.4,"score_norm":28.42,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.68,"score_norm":74.2,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.776767676767677 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":13.67,"score_norm":17.16,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.136700648748842 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":82,"score_norm":70,"trust":1,"source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2","config":null},{"benchmark":"swe_bench_verified","score_raw":69.4,"score_norm":66,"trust":1,"source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2","config":null},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":30,"score_norm":29.41,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.8016,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.2,"cache_read_usd_per_m":null,"context_length":204800,"source":"openrouter","source_model_id":"minimax/minimax-m2","as_of":"2026-10-06","blended_usd_per_m":0.5249999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":457.38,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":482.329,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":230.134,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":243.721,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2/tree/main"},{"quant":"int4","weights_gb":137.214,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":146.155,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":18.6,"score_norm":18.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":28.92,"score_norm":28.92,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":65,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.65 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":78.33,"score_norm":72.91,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.783333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1343.4,"score_norm":28.42,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1345.6,"score_norm":29.4,"config":"6903 votes, arena entry minimax-m2, leaderboard text-overall-style_control, rank 210, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1296.6,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1297,"score_norm":null,"config":"6538 votes, arena entry minimax-m2, code/overall, leaderboard webdev-overall-raw, rank 98, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.00857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1113,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1141,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1131,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1099,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1143,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1149,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.68,"score_norm":74.2,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.776767676767677 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":13.67,"score_norm":17.16,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.136700648748842 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.5,"score_norm":15.44,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":72.31,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.72312925170068 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":82.65,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.826455026455027 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82,"score_norm":70,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":36.11,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.361111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.4,"score_norm":66,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":30,"score_norm":29.41,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2025-11-01","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":25.76,"score_norm":24.42,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.257575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2","publisher":"OpenRouter (listed 2025-10-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m2.1","display_name":"MiniMax M2.1","family":{"id":"minimax","vendor":"MiniMax"},"generation":"2.1","size_class":"frontier-m","params_total_b":228.69,"params_active_b":null,"params_exact":228689764864,"params_source":"hf_safetensors","vram_est":{"bf16_gb":457,"int8_gb":242,"int4_gb":137,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"MiniMaxAI/MiniMax-M2.1","openrouter_id":"minimax/minimax-m2.1","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2025-12-23","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"minimax-m2.7","aliases":[],"intelligence":{"index":65.7,"inputs":[{"benchmark":"aime","score_raw":77.92,"score_norm":72.4,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.289, task=overall, updated=2026-04-16"},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.128, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":23.17,"score_norm":31.13,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-1","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.231696014828545 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":87.05,"score_norm":78.42,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.333, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":74.8,"score_norm":72,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.944, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":85.38,"score_norm":87.17,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.853801169590643 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":29.2,"score_norm":28.47,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.7223,"subset_bias":{"offset":4.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.03,"context_length":204800,"source":"openrouter","source_model_id":"minimax/minimax-m2.1","as_of":"2026-10-06","blended_usd_per_m":0.5249999999999999,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":457.38,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":482.329,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":230.134,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":243.721,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.1/tree/main"},{"quant":"int4","weights_gb":137.214,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":146.155,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":242.411,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.612,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":20.9,"score_norm":20.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":32.09,"score_norm":32.09,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":66.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.663333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":77.92,"score_norm":72.4,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.289, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":82.67,"score_norm":78.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.826666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1180,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1196,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1215,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1147,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1131,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1230,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1208,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.54,"score_norm":75.41,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.128, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.03,"score_norm":81.73,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.83030303030303 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.17,"score_norm":31.13,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.231696014828545 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.2,"score_norm":29.71,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":69.86,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.698639455782313 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":80.95,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.80952380952381 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.05,"score_norm":78.42,"config":"edition=mmlu pro v1, runner=platform, stderr=0.333, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":40.74,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.407407407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":36.81,"score_norm":53.02,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74.8,"score_norm":72,"config":"edition=swe-bench verified v1, runner=external, stderr=1.944, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74,"score_norm":71.11,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.1","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":85.38,"score_norm":87.17,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.853801169590643 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":29.2,"score_norm":28.47,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2025-12-23","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":28.79,"score_norm":27.99,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.287878787878788 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.1","publisher":"OpenRouter (listed 2025-12-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m2.5","display_name":"MiniMax-M2.5","family":{"id":"minimax","vendor":"MiniMax"},"generation":"2.5","size_class":"frontier-m","params_total_b":228.7,"params_active_b":10,"params_exact":228703644928,"params_source":"hf_safetensors","vram_est":{"bf16_gb":457,"int8_gb":242,"int4_gb":137,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"MiniMaxAI/MiniMax-M2.5","openrouter_id":"minimax/minimax-m2.5","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":204800,"release_date":"2026-01-01","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"minimax-m2.7","aliases":[],"intelligence":{"index":59.32,"inputs":[{"benchmark":"aime","score_raw":88.75,"score_norm":85.94,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.877, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.17, eval=arc-agi-2 semi-private (v2_semi_private), org=minimax, system_type=cot"},{"benchmark":"arena_elo","score_raw":1390.8,"score_norm":49.55,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":82.07,"score_norm":80.38,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.258, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":20.53,"score_norm":27.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-5","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.205282669138091 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":80.09,"score_norm":66.82,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.39, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":74.2,"score_norm":71.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.959, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.32,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-5","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.953216374269006 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/minimax-m2-5","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.8809,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.27,"completion_usd_per_m":1.08,"cache_read_usd_per_m":0.027,"context_length":204800,"source":"openrouter","source_model_id":"minimax/minimax-m2.5","as_of":"2026-10-06","blended_usd_per_m":0.47250000000000003,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":457.4,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":482.35,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":242.422,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.623,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":230.134,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":243.721,"method":"vendor","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.5/tree/main"},{"quant":"int4","weights_gb":137.22,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":146.161,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":242.422,"kv_cache_gb_per_8k":2.0804,"min_serve_gb":256.623,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":22.8,"score_norm":22.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.47,"score_norm":34.47,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.72 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":88.75,"score_norm":85.94,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.877, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":63.67,"score_norm":null,"config":"cost_per_task_usd=0.07, eval=arc-agi-1 semi-private (v1_semi_private), org=minimax, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":4.86,"score_norm":0,"config":"cost_per_task_usd=0.17, eval=arc-agi-2 semi-private (v2_semi_private), org=minimax, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1390.8,"score_norm":49.55,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1390.4,"score_norm":49.37,"config":"41032 votes, arena entry minimax-m2.5, leaderboard text-overall-style_control, rank 155, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1386.2,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1383.9,"score_norm":null,"config":"7218 votes, arena entry minimax-m2.5, code/overall, leaderboard webdev-overall-raw, rank 75, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":1.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0114285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1185,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1213,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1180,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1189,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1148,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1179,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1229,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":82.07,"score_norm":80.38,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.258, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.85,"score_norm":84.3,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.848484848484848 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.2,"score_norm":84.79,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":20.53,"score_norm":27.25,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.205282669138091 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":19.4,"score_norm":25.59,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.63,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.716326530612245 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":80.09,"score_norm":66.82,"config":"edition=mmlu pro v1, runner=platform, stderr=0.39, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":42.59,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.425925925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":55.4,"score_norm":84,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.5","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":74.2,"score_norm":71.33,"config":"edition=swe-bench verified v1, runner=external, stderr=1.959, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":75.8,"score_norm":73.11,"config":"high reasoning","trust":2,"source_type":"aggregator","source_url":"https://www.swebench.com/","measured_date":"2026-03-10","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.32,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.953216374269006 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/minimax-m2-5","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/MiniMaxAI/MiniMax-M2.5","publisher":"Hugging Face"},{"url":"https://huggingface.co/blog/mlabonne/minimax-m25","publisher":"Hugging Face Blog"},{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2.5","publisher":"OpenRouter (listed 2026-02-12)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3-medium","display_name":"Mistral Medium 3","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3","size_class":"frontier-medium","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-medium-3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2025-05-07","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":[],"superseded_by":"mistral-3.1-medium","aliases":["mistral-medium-3"],"intelligence":{"index":20.16,"inputs":[{"benchmark":"aime","score_raw":30.33,"score_norm":12.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3","config":"2025, artificial analysis independent run"},{"benchmark":"gpqa_diamond","score_raw":57.78,"score_norm":46.17,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3","config":"artificial analysis independent run"},{"benchmark":"hle","score_raw":4.05,"score_norm":3.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3","config":"artificial analysis independent run, no tools"},{"benchmark":"tau_bench","score_raw":24.27,"score_norm":5.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3","config":"artificial analysis independent run, tau2, with tools"},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-medium-3","config":"artificial analysis independent run"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":2,"cache_read_usd_per_m":0.04,"context_length":131072,"source":"openrouter","source_model_id":"mistralai/mistral-medium-3","as_of":"2026-10-06","blended_usd_per_m":0.8,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9,"score_norm":9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":31.33,"score_norm":null,"config":"artificial analysis aa-lcr v1.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":30.33,"score_norm":12.91,"config":"2025, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1099,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1075,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1042,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1030,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1031,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1085,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.78,"score_norm":46.17,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.05,"score_norm":3.01,"config":"artificial analysis independent run, no tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":39.25,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":40,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":53.01,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":24.27,"score_norm":5.69,"config":"artificial analysis independent run, tau2, with tools","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-medium-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-medium-3","publisher":"OpenRouter (listed 2025-05-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-spark-1.1","display_name":"Muse Spark 1.1","family":{"id":"muse","vendor":"Meta"},"generation":"1.1","size_class":"frontier-spark","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"meta/muse-spark-1.1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file","audio"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-07-16","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"muse-spark-1.3","aliases":[],"intelligence":{"index":81.66,"inputs":[{"benchmark":"arena_elo","score_raw":1491.3,"score_norm":94.34,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":91.16,"score_norm":93.18,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"overall, xhigh reasoning effort"},{"benchmark":"hle","score_raw":46.2,"score_norm":65,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","config":"artificial analysis eval, muse spark 1.1 (xhigh)"},{"benchmark":"livebench","score_raw":75.3,"score_norm":78.44,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh effort"},{"benchmark":"mmlu_pro","score_raw":88.73,"score_norm":81.22,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"overall, xhigh reasoning effort"},{"benchmark":"swe_bench_verified","score_raw":82,"score_norm":80,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"overall, xhigh reasoning effort"},{"benchmark":"terminal_bench","score_raw":69.29,"score_norm":75.64,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"overall, terminal-bench 2.1, xhigh reasoning effort"}],"coverage":0.7936,"subset_bias":{"offset":0.31,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":4.25,"cache_read_usd_per_m":0.15,"context_length":1048576,"source":"openrouter","source_model_id":"meta/muse-spark-1.1","as_of":"2026-10-06","blended_usd_per_m":2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":25.8,"score_norm":25.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":39.74,"score_norm":39.74,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":71.3,"score_norm":71.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":33.7,"score_norm":33.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":53.2,"score_norm":53.2,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":81.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.813333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":28.08,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=xhigh, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1491.3,"score_norm":94.34,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1490.8,"score_norm":94.12,"config":"20242 votes, arena entry super-nova-ext-3tam-text, leaderboard text-overall-style_control, rank 8, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1464.7,"score_norm":null,"config":"document","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_document","score_raw":1471.7,"score_norm":null,"config":"2054 votes, arena entry super-nova-ext-3tam-document, document/overall, leaderboard document-overall-raw, rank 13, raw (no style control), vote cutoff 2026-07-26","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1536.7,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1281.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1281.4,"score_norm":null,"config":"6142 votes, arena entry super-nova-ext-3tam-vision, leaderboard vision-overall-style_control, rank 18, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1541.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1539,"score_norm":null,"config":"6204 votes, arena entry super-nova-ext-3tam-webdev, code/overall, leaderboard webdev-overall-raw, rank 20, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":42.81,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.42813308486650653 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":15.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.151428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1275,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1183,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1188,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1291,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1283,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1280,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1198,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1285,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1104,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1194,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1172,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1156,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1238,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1296,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1199,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1276,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"enterprise_ops_gym","score_raw":47.24,"score_norm":null,"config":"aa field enterpriseopsgym, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.4723962996120561 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":43.71,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.43711 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":91.16,"score_norm":93.18,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.8,"score_norm":91.27,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.897979797979798 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.8,"score_norm":91.27,"config":"artificial analysis eval, muse spark 1.1 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"harvey_lab_criteria_pass","score_raw":93.08,"score_norm":null,"config":"aa field harveylabcriteriapass, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.930814879143147 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":46.2,"score_norm":65,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.462001853568119 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":46.2,"score_norm":65,"config":"artificial analysis eval, muse spark 1.1 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":75.3,"score_norm":78.44,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":75.3,"score_norm":78.44,"config":"global average, livebench-2026-06-25 release, run muse-spark-1.1-xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.73,"score_norm":81.22,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":58.22,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.582175925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":82,"score_norm":80,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":31.75,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.317525773195876 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":69.29,"score_norm":75.64,"config":"overall, terminal-bench 2.1, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.9,"score_norm":85.76,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.779026217228464 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":77.9,"score_norm":85.76,"config":"artificial analysis eval, muse spark 1.1 (xhigh), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-1","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.1","publisher":"OpenRouter (listed 2026-07-16)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-spark-1.2","display_name":"Muse Spark 1.2","family":{"id":"muse","vendor":"Meta"},"generation":"1.2","size_class":"frontier-spark","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"meta/muse-spark-1.2","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file","audio"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-08-05","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"muse-spark-1.3","aliases":[],"intelligence":{"index":83.21,"inputs":[{"benchmark":"arena_elo","score_raw":1493.5,"score_norm":95.32,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":90.4,"score_norm":92.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","config":"artificial analysis eval, muse spark 1.2 (xhigh)"},{"benchmark":"hle","score_raw":45.46,"score_norm":63.91,"trust":2,"source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","config":"artificial analysis eval, muse spark 1.2 (xhigh)"},{"benchmark":"livebench","score_raw":77.95,"score_norm":84.33,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release, run: xhigh effort"},{"benchmark":"mmlu_pro","score_raw":88.28,"score_norm":80.47,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"overall, xhigh reasoning effort"},{"benchmark":"swe_bench_verified","score_raw":86.6,"score_norm":85.11,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"overall, xhigh reasoning effort"},{"benchmark":"terminal_bench","score_raw":69.66,"score_norm":76.07,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"overall, terminal-bench 2.1, xhigh reasoning effort"}],"coverage":0.7936,"subset_bias":{"offset":0.4,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":4.25,"cache_read_usd_per_m":0.15,"context_length":1048576,"source":"openrouter","source_model_id":"meta/muse-spark-1.2","as_of":"2026-10-06","blended_usd_per_m":2,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":43.2,"score_norm":43.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":49.31,"score_norm":49.31,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=xhigh, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":72.2,"score_norm":72.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":39.6,"score_norm":39.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-09-20","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":56.76,"score_norm":56.76,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":83.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.833333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":27.2,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=xhigh, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1493.5,"score_norm":95.32,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1497.7,"score_norm":97.2,"config":"3257 votes, arena entry muse-spark-1.2-xhigh-text, leaderboard text-overall-style_control, rank 4, run: xhigh, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1292.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1292,"score_norm":null,"config":"1853 votes, arena entry muse-spark-1.2-xhigh-vision, leaderboard vision-overall-style_control, rank 8, run: xhigh, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1531.8,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1533.7,"score_norm":null,"config":"2062 votes, arena entry muse-spark-1.2-xhigh-webdev, code/overall, leaderboard webdev-overall-raw, rank 24, raw (no style control), run: xhigh, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":17.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.177142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1319,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_agenticgamedev","score_raw":1179,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1236,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1306,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1320,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1352,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1208,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1315,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_htmlslides","score_raw":1134,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1174,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_python_pptxslides","score_raw":1198,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1290,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1307,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1215,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1320,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":56.4,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gdpval-aa, reasoning=true, run: xhigh, source value 0.564 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.4,"score_norm":92.11,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.904040404040404 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":90.4,"score_norm":92.11,"config":"artificial analysis eval, muse spark 1.2 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":45.46,"score_norm":63.91,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.454587581093605 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":45.46,"score_norm":63.91,"config":"artificial analysis eval, muse spark 1.2 (xhigh)","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":77.95,"score_norm":84.33,"config":"livebench-2026-06-25 release, run: xhigh effort","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":77.95,"score_norm":84.33,"config":"global average, livebench-2026-06-25 release, run muse-spark-1.2-xhigh","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":88.28,"score_norm":80.47,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-15","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":56.37,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.563657407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":86.6,"score_norm":85.11,"config":"overall, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":34.85,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.348453608247423 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":69.66,"score_norm":76.07,"config":"overall, terminal-bench 2.1, xhigh reasoning effort","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-13","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.15,"score_norm":88.41,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.801498127340824 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":80.15,"score_norm":88.41,"config":"artificial analysis eval, muse spark 1.2 (xhigh), terminal-bench 2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/muse-spark-1-2","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2","publisher":"OpenRouter (listed 2026-08-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-235b-a22b","display_name":"Qwen3-235B-A22B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.09,"params_active_b":22,"params_exact":235093634560,"params_source":"hf_safetensors","vram_est":{"bf16_gb":470,"int8_gb":249,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3-235B-A22B","openrouter_id":"qwen/qwen3-235b-a22b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":["qwen-3-235b-a22b-04-28"],"superseded_by":"qwen-3-235b-a22b-07-25","aliases":["qwen-3-235b-a22b-04-28","qwen3-235b-a22b","qwen3-235b-a22b-04-28"],"intelligence":{"index":52.71,"inputs":[{"benchmark":"aime","score_raw":83.96,"score_norm":79.95,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.768, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1375.2,"score_norm":42.6,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":70.2,"score_norm":63.66,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.373, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":10.98,"score_norm":13.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.1098 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":78.4,"score_norm":85.33,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","config":"livebench 20241125, thinking-2507"},{"benchmark":"mmlu_pro","score_raw":81.25,"score_norm":68.75,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.387, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":27.19,"score_norm":9.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.271929824561404 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7619,"subset_bias":{"offset":2.81,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.455,"completion_usd_per_m":1.82,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-235b-a22b","as_of":"2026-10-06","blended_usd_per_m":0.79625,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":470.18,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.266,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":470.192,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":495.279,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B/tree/main"},{"quant":"fp8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.054,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":149.684,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.195,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.232,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":8424.86,"quant":"fp16","batch_size":null,"context_tokens":8192,"method":"community","trust":0,"source_url":"https://docs.gpustack.ai/latest/performance-lab/qwen3-235b-a22b/h100/","derivation":null}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.5,"score_norm":9.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":13.45,"score_norm":13.45,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":83.96,"score_norm":79.95,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.768, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":82,"score_norm":77.5,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.82 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":92.3,"score_norm":90.38,"config":"aime 2025, qwen3-235b-a22b-thinking-2507, 81920 max output tokens","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1375.2,"score_norm":42.6,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1374.8,"score_norm":42.42,"config":"26331 votes, arena entry qwen3-235b-a22b, leaderboard text-overall-style_control, rank 173, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_hard_v2","score_raw":79.7,"score_norm":null,"config":"thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":874,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1008,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1004,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":940,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":964,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1037,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":70.2,"score_norm":63.66,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.373, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":70,"score_norm":63.38,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.7 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":81.1,"score_norm":79.01,"config":"qwen3-235b-a22b-thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.98,"score_norm":13.21,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.1098 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":18.2,"score_norm":23.82,"config":"text-only subset, thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":38.71,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.387074829931973 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":87.8,"score_norm":null,"config":"thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":78.4,"score_norm":85.33,"config":"livebench 20241125, thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":62.22,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.622222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":74.1,"score_norm":null,"config":"25.02-25.05, thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.25,"score_norm":68.75,"config":"edition=mmlu pro v1, runner=platform, stderr=0.387, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.4,"score_norm":74,"config":"qwen3-235b-a22b-thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":39.93,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.399305555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"supergpqa","score_raw":64.9,"score_norm":null,"config":"thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":21.41,"score_norm":27.35,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-02-28","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":27.19,"score_norm":9.59,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.271929824561404 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":71.9,"score_norm":69.2,"config":"tau2/retail, thinking-2507","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.06,"score_norm":1.25,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0606060606060606 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-235b-a22b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://huggingface.co/Qwen/Qwen3-235B-A22B","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-max","display_name":"Qwen3 Max","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3-max","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2025-09-23","release_status":"released","knowledge_cutoff":"2025-06-30","status":"superseded","supersedes":[],"superseded_by":"qwen-3.7-max","aliases":["qwen3-max"],"intelligence":{"index":58.54,"inputs":[{"benchmark":"aime","score_raw":81.04,"score_norm":76.3,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, run: max, runner=platform, stderr=1.1, task=overall, updated=2026-04-16"},{"benchmark":"gpqa_diamond","score_raw":79.55,"score_norm":76.83,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=2.026, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":11.91,"score_norm":14.57,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: max, source value 0.119091751621872 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":84.36,"score_norm":73.93,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.356, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":74.27,"score_norm":72.36,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, source value 0.742690058479532 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":20.45,"score_norm":18.18,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-max","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, source value 0.204545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5635,"subset_bias":{"offset":3.71,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.78,"completion_usd_per_m":3.9,"cache_read_usd_per_m":0.156,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3-max","as_of":"2026-10-06","blended_usd_per_m":1.56,"endpoint_expires_at":"2026-10-09"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":15.6,"score_norm":15.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":24.45,"score_norm":24.45,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: max","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":48.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: max, source value 0.483333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":81.04,"score_norm":76.3,"config":"archived=true, edition=aime v1, run: max, runner=platform, stderr=1.1, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":80.67,"score_norm":75.84,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, source value 0.806666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: preview, source value 0.00857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1089,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1139,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1115,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1109,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1107,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1010,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1085,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1125,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.55,"score_norm":76.83,"config":"edition=gpqa diamond v1, run: max, runner=platform, stderr=2.026, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.36,"score_norm":72.34,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: max, source value 0.763636363636364 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":11.91,"score_norm":14.57,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: max, source value 0.119091751621872 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":48.03,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, run: preview, source value 0.480272108843537 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max-preview","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":76.72,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, run: max, source value 0.767195767195767 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":84.36,"score_norm":73.93,"config":"edition=mmlu pro v1, run: max, runner=platform, stderr=0.356, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":38.31,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, scicode, source value 0.383101851851852 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":74.27,"score_norm":72.36,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, source value 0.742690058479532 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":20.45,"score_norm":18.18,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: max, source value 0.204545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-max","publisher":"OpenRouter (listed 2025-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.6-plus-04-02","display_name":"Qwen3.6 Plus","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.6","size_class":"frontier-plus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.6-plus","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-04-02","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.7-plus","aliases":["qwen3.6-plus","qwen3.6-plus-04-02"],"intelligence":{"index":72.9,"inputs":[{"benchmark":"aime","score_raw":94.58,"score_norm":93.23,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=0.579, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1443.4,"score_norm":72.99,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":87.37,"score_norm":87.85,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=1.668, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":27.85,"score_norm":38.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.278498609823911 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":68.91,"score_norm":64.24,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"mmlu_pro","score_raw":87.67,"score_norm":79.45,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.323, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":73.4,"score_norm":70.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.978, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":53.18,"score_norm":56.68,"trust":3,"source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","config":"edition=terminal-bench 2.1, runner=external, stderr=1.633, task=overall, updated=2026-08-19"}],"coverage":0.9207,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.325,"completion_usd_per_m":1.95,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.6-plus","as_of":"2026-10-06","blended_usd_per_m":0.73125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":29,"score_norm":29,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":29,"score_norm":29,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":54.5,"score_norm":54.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":27,"score_norm":27,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":40.49,"score_norm":40.49,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":0.88,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":94.58,"score_norm":93.23,"config":"archived=true, edition=aime v1, runner=platform, stderr=0.579, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1443.4,"score_norm":72.99,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1443.4,"score_norm":72.99,"config":"45376 votes, arena entry qwen3.6-plus-text, leaderboard text-overall-style_control, rank 74, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_image_to_webdev","score_raw":1468.6,"score_norm":null,"config":"image_to_webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-17","observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1460.7,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1459.7,"score_norm":null,"config":"16336 votes, arena entry qwen3.6-plus-v1-webdev, code/overall, leaderboard webdev-overall-raw, rank 45, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"automation_bench_partial_score","score_raw":18.65,"score_norm":null,"config":"aa field automationbenchpartialscore, aa field automationbenchpartialscore (partial score), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.18645616010518853 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":2.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1218,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_asciiart","score_raw":1129,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1241,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-04","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1237,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1224,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1160,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-06","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1238,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-10-05","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1248,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","measured_date":null,"observed_at":"2026-09-25","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":32,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=true, source value 0.32004499999999997 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":87.37,"score_norm":87.85,"config":"edition=gpqa diamond v1, runner=platform, stderr=1.668, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.18,"score_norm":88.99,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.881818181818182 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.85,"score_norm":38.01,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.278498609823911 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.17,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.751700680272109 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":68.91,"score_norm":64.24,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":87.67,"score_norm":79.45,"config":"edition=mmlu pro v1, runner=platform, stderr=0.323, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":77.98,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, source value 0.779768786127168 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.74,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.407407407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":73.4,"score_norm":70.44,"config":"edition=swe-bench verified v1, runner=external, stderr=1.978, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":97.66,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.976608187134503 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":20.82,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.208247422680412 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":53.18,"score_norm":56.68,"config":"edition=terminal-bench 2.1, runner=external, stderr=1.633, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/terminal-bench-2-1","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.94,"score_norm":45.81,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.439393939393939 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":61.42,"score_norm":66.38,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.614232209737828 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-plus","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-plus","publisher":"OpenRouter (listed 2026-04-02)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"deepseek-3-0324-instruct","display_name":"DeepSeek V3 0324","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3","size_class":"frontier","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"deepseek-ai/DeepSeek-V3-0324","openrouter_id":"deepseek/deepseek-chat-v3-0324","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":"2025-03-24","release_status":"released","knowledge_cutoff":"2024-07-31","status":"superseded","supersedes":[],"superseded_by":"deepseek-3.1-instruct","aliases":["deepseek-chat-v3-0324","deepseek-v3-0324"],"intelligence":{"index":40.53,"inputs":[{"benchmark":"aime","score_raw":52.2,"score_norm":40.25,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.712, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1395.6,"score_norm":51.69,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":61.62,"score_norm":51.58,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.447, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":4.74,"score_norm":4.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0474 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":79.47,"score_norm":65.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.396, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":47.08,"score_norm":36.11,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.470760233918129 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"trust":2,"source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":1.96,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.29,"completion_usd_per_m":1.14,"cache_read_usd_per_m":0.11,"context_length":163840,"source":"openrouter","source_model_id":"deepseek/deepseek-chat-v3-0324","as_of":"2026-10-06","blended_usd_per_m":0.5025,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":1369.06,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1438.089,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":688.587,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":723.592,"method":"vendor","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/tree/main"},{"quant":"int4","weights_gb":410.718,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":431.83,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":725.602,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":762.458,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.8,"score_norm":0.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3-0324","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":1.58,"score_norm":1.58,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":21.2,"score_norm":21.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3-0324","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.7,"score_norm":9.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3-0324","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.23,"score_norm":15.23,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":41.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.413333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":52.2,"score_norm":40.25,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.712, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":41,"score_norm":26.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.41 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1395.6,"score_norm":51.69,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1395.5,"score_norm":51.64,"config":"45595 votes, arena entry deepseek-v3-0324, leaderboard text-overall-style_control, rank 148, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.62,"score_norm":51.58,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.447, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":65.45,"score_norm":56.97,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.654545454545455 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.74,"score_norm":4.03,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.02,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.410204081632653 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":40.53,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.405291005291005 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.47,"score_norm":65.78,"config":"edition=mmlu pro v1, runner=platform, stderr=0.396, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.2,"score_norm":68.67,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/deepseek-ai/DeepSeek-V3-0324","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":35.76,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.357638888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":47.08,"score_norm":36.11,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.470760233918129 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":4.74,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0474226804123711 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.15,"score_norm":11.94,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.151515151515152 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":13.86,"score_norm":10.42,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.138576779026217 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/deepseek-v3-0324","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat-v3-0324","publisher":"OpenRouter (listed 2025-03-24)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-opus-4","display_name":"claude-4-opus","family":{"id":"claude","vendor":"Anthropic"},"generation":"4","size_class":"frontier-opus","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-opus-4","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-05-22","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-opus-5.5","aliases":["claude-4-opus"],"intelligence":{"index":46.08,"inputs":[{"benchmark":"aime","score_raw":41.25,"score_norm":26.56,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.152, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":8.61,"score_norm":4.01,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=1.9284, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-opus-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot"},{"benchmark":"arena_elo","score_raw":1412.6,"score_norm":59.27,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"44326 votes, arena entry claude-opus-4-20250514, leaderboard text-overall-style_control, rank 128, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":71.72,"score_norm":65.8,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.261, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":12.33,"score_norm":15.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.123262279888786 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":86.17,"score_norm":76.95,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.337, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":73.39,"score_norm":71.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.733918128654971 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":31.69,"score_norm":31.69,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":40,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0.4 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":41.25,"score_norm":26.56,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.152, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":73.33,"score_norm":66.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.733333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":35.67,"score_norm":null,"config":"cost_per_task_usd=1.2496, eval=arc-agi-1 semi-private (v1_semi_private), model_group=claude-opus-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":8.61,"score_norm":4.01,"config":"cost_per_task_usd=1.9284, eval=arc-agi-2 semi-private (v2_semi_private), model_group=claude-opus-4-20250514-thinking, org=anthropic, run: thinking 16k, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1412.6,"score_norm":59.27,"config":"44326 votes, arena entry claude-opus-4-20250514, leaderboard text-overall-style_control, rank 128, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1188.7,"score_norm":null,"config":"2201 votes, arena entry claude-opus-4-20250514, leaderboard vision-overall-style_control, rank 81, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_3d","score_raw":1167,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1174,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-21","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1160,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-17","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1193,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_svg","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-20","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1168,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-12","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1177,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","measured_date":null,"observed_at":"2026-09-11","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.72,"score_norm":65.8,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.261, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79.6,"score_norm":76.9,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.795959595959596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":12.33,"score_norm":15.19,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.123262279888786 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":53.74,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.537414965986395 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":63.6,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.635978835978836 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":86.17,"score_norm":76.95,"config":"edition=mmlu pro v1, runner=platform, stderr=0.337, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":40.86,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, scicode, source value 0.408564814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":73.39,"score_norm":71.19,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.733918128654971 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":31.06,"score_norm":30.66,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.310606060606061 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-4-opus-thinking","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-opus-4","publisher":"OpenRouter (listed 2025-05-22)"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"gemma-3-12b-instruct","display_name":"Gemma 3 12B","family":{"id":"gemma","vendor":"Google DeepMind"},"generation":"3","size_class":"12b","params_total_b":12.19,"params_active_b":12.19,"params_exact":12187325040,"params_source":"hf_safetensors","vram_est":{"bf16_gb":24.4,"int8_gb":12.9,"int4_gb":7.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"google/gemma-3-12b-it","openrouter_id":"google/gemma-3-12b-it","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"gemma","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-03-12","release_status":"released","knowledge_cutoff":"2024-08-31","status":"superseded","supersedes":[],"superseded_by":"gemma-4-12b-unified-instruct","aliases":["gemma-3-12b","gemma-3-12b-it"],"intelligence":{"index":9.13,"inputs":[{"benchmark":"aime","score_raw":18.33,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-12b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.183333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1341.8,"score_norm":27.71,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":34.95,"score_norm":14.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-12b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.349494949494949 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.21,"score_norm":3.25,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-12b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0421 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":10.82,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-12b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.108187134502924 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemma-3-12b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.05,"completion_usd_per_m":0.15,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"google/gemma-3-12b-it","as_of":"2026-10-06","blended_usd_per_m":0.07500000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":24.38,"kv_cache_gb_per_8k":null,"min_serve_gb":26.087,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":24.375,"kv_cache_gb_per_8k":null,"min_serve_gb":26.081,"method":"vendor","source_url":"https://huggingface.co/google/gemma-3-12b-it/tree/main"},{"quant":"fp8","weights_gb":12.921,"kv_cache_gb_per_8k":null,"min_serve_gb":14.409,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.314,"kv_cache_gb_per_8k":null,"min_serve_gb":8.802,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":12.921,"kv_cache_gb_per_8k":null,"min_serve_gb":14.409,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":29.93,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":608,"params_active_b":12.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":49.88,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":13.44,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":12.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":22.4,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":30120.29,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"deepseek-v4-flash-reasoning","value":1262,"params_active_b":290.94,"quant":"fp8","batch_size":32,"context_tokens":8192,"trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash"},"scaling":"active_param_ratio","quant_scale":1,"quant":"fp8","params_active_b":12.19,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":164.89,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":12.19,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":274.82,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":0.1,"score_norm":0.1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-12b-it","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":0.27,"score_norm":0.27,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":5.8,"score_norm":5.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-12b-it","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":3.8,"score_norm":3.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemma-3-12b-it","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":5.51,"score_norm":5.51,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":8,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.08 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":18.33,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.183333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1341.8,"score_norm":27.71,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1341.8,"score_norm":27.71,"config":"3829 votes, arena entry gemma-3-12b-it, leaderboard text-overall-style_control, rank 214, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":34.95,"score_norm":14.01,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.349494949494949 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.21,"score_norm":3.25,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0421 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":36.73,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.36734693877551 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":13.65,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.136507936507936 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":37.51,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.37514450867052 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":17.36,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.173611111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":10.82,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.108187134502924 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":0.82,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00824742268041237 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemma-3-12b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/blog/gemma3","publisher":"Hugging Face"},{"url":"https://huggingface.co/google/gemma-3-27b-it","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-4-o-mini","display_name":"GPT-4o-mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"4","size_class":"frontier-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-4o-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-07-18","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"gpt-5.4-mini","aliases":["gpt-4o-mini","gpt-4o-mini-2024-07-18"],"intelligence":{"index":10.22,"inputs":[{"benchmark":"aime","score_raw":14.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.146666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.01, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=base llm"},{"benchmark":"arena_elo","score_raw":1317.6,"score_norm":16.92,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":42.63,"score_norm":24.83,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.426262626262626 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.2,"score_norm":3.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.042 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":5.62,"score_norm":0.73,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0561797752808989 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.5554,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.6,"cache_read_usd_per_m":0.075,"context_length":128000,"source":"openrouter","source_model_id":"openai/gpt-4o-mini","as_of":"2026-10-06","blended_usd_per_m":0.26249999999999996,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1,"score_norm":1,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":0.96,"score_norm":0.96,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":11.4,"score_norm":11.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-mini","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.67,"score_norm":6.67,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":14.67,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.146666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":0,"score_norm":0,"config":"cost_per_task_usd=0.01, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=base llm","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1317.6,"score_norm":16.92,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1097.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":0,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":42.63,"score_norm":24.83,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.426262626262626 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.2,"score_norm":3.24,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.042 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":30.95,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.30952380952381 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":23.39,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.233862433862434 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":0,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":41.5,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.415028901734104 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":22.92,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.229166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":2.89,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0288659793814433 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":5.62,"score_norm":0.73,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0561797752808989 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-4o-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-4o-mini","publisher":"OpenRouter (listed 2024-07-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o1","display_name":"OpenAI o1","family":{"id":"gpt","vendor":"OpenAI"},"generation":"1","size_class":"frontier-o","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2024-12-17","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"gpt-o3","aliases":["o1","o1-2024-12-17"],"intelligence":{"index":51.06,"inputs":[{"benchmark":"aime","score_raw":71.46,"score_norm":64.32,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.458, task=overall, updated=2026-04-16"},{"benchmark":"arena_elo","score_raw":1402.3,"score_norm":54.68,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":73.23,"score_norm":67.93,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.231, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":7.01,"score_norm":7.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/o1","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0701 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":83.49,"score_norm":72.48,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.364, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":62.57,"score_norm":56.76,"trust":2,"source_url":"https://artificialanalysis.ai/models/o1","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.625730994152047 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":12.88,"score_norm":9.27,"trust":2,"source_url":"https://artificialanalysis.ai/models/o1","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.128787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6428,"subset_bias":{"offset":1.09,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":15,"completion_usd_per_m":60,"cache_read_usd_per_m":7.5,"context_length":200000,"source":"openrouter","source_model_id":"openai/o1","as_of":"2026-10-06","blended_usd_per_m":26.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":39.7,"score_norm":39.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o1","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15.2,"score_norm":15.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":23.86,"score_norm":23.86,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":63.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.633333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":71.46,"score_norm":64.32,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.458, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1402.3,"score_norm":54.68,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1402.2,"score_norm":54.63,"config":"27807 votes, arena entry o1-2024-12-17, leaderboard text-overall-style_control, rank 140, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1192.8,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1193.2,"score_norm":null,"config":"3694 votes, arena entry o1-2024-12-17, leaderboard vision-overall-style_control, rank 78, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.29,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.00285714285714286 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":73.23,"score_norm":67.93,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.231, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.75,"score_norm":70.07,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.747474747474747 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.01,"score_norm":7.37,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0701 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":70.34,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.703401360544218 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":67.94,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.679365079365079 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.49,"score_norm":72.48,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.364, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":35.76,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.357638888888889 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":62.57,"score_norm":56.76,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.625730994152047 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":12.88,"score_norm":9.27,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.128787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o1","publisher":"OpenRouter (listed 2024-12-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o3-mini","display_name":"OpenAI o3 Mini","family":{"id":"gpt","vendor":"OpenAI"},"generation":"3","size_class":"frontier-o-mini","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o3-mini","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-01-31","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"gpt-o4-mini","aliases":["o3-mini","o3-mini-2025-01-31"],"intelligence":{"index":40.98,"inputs":[{"benchmark":"aime","score_raw":86.46,"score_norm":83.07,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.054, task=overall, updated=2026-04-16"},{"benchmark":"arc_agi_2","score_raw":2.99,"score_norm":0,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=0.5472, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3-mini, org=openai, run: high, system_type=cot"},{"benchmark":"arena_elo","score_raw":1348.1,"score_norm":30.52,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":75.5,"score_norm":71.13,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.176, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":7.91,"score_norm":8.69,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0791 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":78.69,"score_norm":64.48,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.394, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":28.65,"score_norm":11.53,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.286549707602339 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/o3-mini","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7222,"subset_bias":{"offset":-1.5,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.1,"completion_usd_per_m":4.4,"cache_read_usd_per_m":0.55,"context_length":200000,"source":"openrouter","source_model_id":"openai/o3-mini","as_of":"2026-10-06","blended_usd_per_m":1.9250000000000003,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.5,"score_norm":12.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o3-mini","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.22,"score_norm":19.22,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":86.46,"score_norm":83.07,"config":"archived=true, edition=aime v1, reasoning_effort=high, runner=platform, stderr=1.054, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":34.5,"score_norm":null,"config":"cost_per_task_usd=0.3989, eval=arc-agi-1 semi-private (v1_semi_private), model_group=o3-mini, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":2.99,"score_norm":0,"config":"cost_per_task_usd=0.5472, eval=arc-agi-2 semi-private (v2_semi_private), model_group=o3-mini, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1348.1,"score_norm":30.52,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1347.8,"score_norm":30.38,"config":"57440 votes, arena entry o3-mini, leaderboard text-overall-style_control, rank 202, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.5,"score_norm":71.13,"config":"edition=gpqa diamond v1, reasoning_effort=high, runner=platform, stderr=2.176, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":74.85,"score_norm":70.21,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.748484848484848 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.91,"score_norm":8.69,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.0791 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":71.75,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, source value 0.717460317460317 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":78.69,"score_norm":64.48,"config":"edition=mmlu pro v1, reasoning_effort=high, runner=platform, stderr=0.394, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":39.93,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.399305555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":28.65,"score_norm":11.53,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.286549707602339 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o3-mini","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o3-mini","publisher":"OpenRouter (listed 2025-01-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"granite-4.1-8b","display_name":"granite-4.1-8b","family":{"id":"granite","vendor":"IBM"},"generation":"4.1","size_class":"8b","params_total_b":8.79,"params_active_b":8.79,"params_exact":8791592960,"params_source":"hf_safetensors","vram_est":{"bf16_gb":17.6,"int8_gb":9.3,"int4_gb":5.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"ibm-granite/granite-4.1-8b","openrouter_id":"ibm-granite/granite-4.1-8b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2026-04-30","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"granite-4.2-8b","aliases":[],"intelligence":{"index":15.84,"inputs":[{"benchmark":"arena_elo","score_raw":1304.1,"score_norm":10.91,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":43.33,"score_norm":25.82,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.433333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.8,"score_norm":2.65,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0379981464318814 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":55.99,"score_norm":26.65,"trust":1,"source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","config":"'8b dense' column, 5-shot, cot, instruct"},{"benchmark":"tau_bench","score_raw":27.78,"score_norm":10.37,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5635,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":17.58,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":19.922,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":17.583,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":19.925,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b/tree/main"},{"quant":"fp8","weights_gb":9.317,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.659,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":5.274,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":7.616,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":9.317,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.659,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":41.5,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":608,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":69.17,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":18.63,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":31.06,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":41770.91,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"deepseek-v4-flash-reasoning","value":1262,"params_active_b":290.94,"quant":"fp8","batch_size":32,"context_tokens":8192,"trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash"},"scaling":"active_param_ratio","quant_scale":1,"quant":"fp8","params_active_b":8.79,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":228.67,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":8.79,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":381.11,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_coding_index","score_raw":9.5,"score_norm":9.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.1-8b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":9.5,"score_norm":9.5,"config":"aa's own index scale, artificial analysis 'coding index' as restated by designforonline.com, not a 0-100 percentage","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.4,"score_norm":6.4,"config":"aa's own index scale, designforonline.com's restated/rounded value of the artificial analysis intelligence index, differs slightly from the value (6) read directly off artificialanalysis.ai at fetch time, likely snapshot-date drift — recorded as a separate claim per no-averaging/no-borrowing rule, not 0-100 percent, not merged with the direct aa reading","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6,"score_norm":6,"config":"aa-lcr), aa-omniscience, artificial analysis intelligence index v4.1.1, composite of 9 evals (gdpval-aa v2, critpt, gpqa diamond, humanity's last exam, individual component scores are not exposed in the rendered page (js-rendered chart), not a 0-100 percentage, page shows this value as '6' and explicitly marks it '(estimated)', scale is aa's own index scale, scicode, tau3-banking, terminal-bench v2.1","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.42,"score_norm":6.42,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":12,"score_norm":null,"config":"0-100 scale, corresponds to artificial analysis's long-context-reasoning 'aa-lcr' index component, labeled 'lcr' on page, site aggregates without per-figure attribution","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"aa_lcr","score_raw":11.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.113333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"agi_eval","score_raw":72.43,"score_norm":null,"config":"'8b dense' column, 0-shot, cot, instruct, row is labeled 'agi eval' verbatim on the model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"alpacaeval_2_0","score_raw":50.08,"score_norm":null,"config":"'8b dense' column, 0-100 scale, instruct, judge model/baseline not stated on card, win-rate %","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1304.1,"score_norm":10.91,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1305.5,"score_norm":11.53,"config":"4034 votes, arena entry granite-4.1-8b, leaderboard text-overall-style_control, rank 260, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1190.3,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1191.3,"score_norm":null,"config":"1704 votes, arena entry granite-4.1-8b, code/overall, leaderboard webdev-overall-raw, rank 114, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arenahard","score_raw":69,"score_norm":null,"config":"'8b dense' column, 0-100 scale, instruct, judge model not stated on card, win-rate %","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"attaq","score_raw":81.19,"score_norm":null,"config":"'8b dense' column, instruct, safety benchmark reporting % safe/non-harmful responses to adversarial attack prompts — higher = safer, this is not an attack-success-rate metric","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":80.51,"score_norm":null,"config":"'8b dense' column, 3-shot, cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bfcl_v3","score_raw":68.27,"score_norm":null,"config":"'8b dense' column, berkeley function-calling leaderboard v3, instruct, subset/shot count not stated on card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bfcl_v3_tool_calling","score_raw":42.2,"score_norm":null,"config":"000 prompts (250 each: simple/multiple/parallel/parallel-multiple), 1, 36gb unified memory, ast-match scoring with lenient type coercion, do not conflate), exact bit-width unspecified), harness: ollama local inference on a macbook m3, independent community reproduction of bfcl v3, no publish date given on the repo, not ibm's own result (ibm's card reports 68.27 for bfcl v3 — different harness/subset, quantization not stated (ollama default pull, temperature=0","trust":0,"source_type":"community","source_url":"https://github.com/Abivarma/Granite4-1","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"bigcodebench","score_raw":35,"score_norm":null,"config":"'8b dense' column, edition/subset (full vs hard) not specified on card, instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"cruxeval_o","score_raw":47.63,"score_norm":null,"config":"'8b dense' column, instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"deepmind_math","score_raw":80.07,"score_norm":null,"config":"'8b dense' column, 0-shot, cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"eval_avg","score_raw":80.21,"score_norm":null,"config":"'8b dense' column, ibm's own average across the 'eval+' family (humaneval+/mbpp+-style tests) — not an independently defined external benchmark, instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":43.3,"score_norm":25.77,"config":"0-100 percentage scale, artificial analysis, likely restates artificial analysis's gpqa diamond index component, page's stated 'assessment date' used as measured_date, site states data is 'sourced from the openrouter api, the hugging face open llm leaderboard and our own internal testing' without attributing this specific figure to one of those","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":43.33,"score_norm":25.82,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.433333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":41.96,"score_norm":23.89,"config":"'8b dense' column, 0-shot, cot, does not state whether this is the diamond subset, instruct, model card labels the row simply 'gpqa'","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"gsm8k","score_raw":92.49,"score_norm":null,"config":"'8b dense' column, 8-shot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"gsm_symbolic","score_raw":83.7,"score_norm":null,"config":"'8b dense' column, 8-shot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.8,"score_norm":2.65,"config":"0-100 scale, humanity's last exam, likely an aa-derived component, low absolute score is expected/normal on hle across models, site aggregates from openrouter/artificial analysis/hf open llm leaderboard/'internal testing' without per-figure attribution","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.8,"score_norm":2.65,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0379981464318814 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":79.88,"score_norm":null,"config":"'8b dense' column, instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":38.6,"score_norm":null,"config":"0-100 scale, likely an aa-derived component, site aggregates from multiple upstream sources without per-figure attribution","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":38.64,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.386394557823129 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval_avg","score_raw":87.06,"score_norm":null,"config":"'8b dense' column, 'avg' is ibm's own average across ifeval sub-metrics, instruct, shot count/cot not stated on card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifeval_instruction_level_accuracy","score_raw":85.3,"score_norm":null,"config":"'instruction-level accuracy' metric specifically (distinct from prompt-level), 024-token cap, 1, 120s timeout, 541 prompts with 1-3 instructions each across 8 instruction types, google ifeval, harness: ollama on macbook m3 36gb, no publish date given on the repo, quantization not stated, temperature=0","trust":0,"source_type":"community","source_url":"https://github.com/Abivarma/Granite4-1","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"ifeval_prompt_level_accuracy","score_raw":79.7,"score_norm":null,"config":"'prompt-level accuracy' metric specifically, 024-token cap, 1, 120s timeout, 541 prompts with 1-3 instructions each across 8 instruction types, google ifeval, harness: ollama on macbook m3 36gb, no publish date given on the repo, quantization not stated, temperature=0","trust":0,"source_type":"community","source_url":"https://github.com/Abivarma/Granite4-1","measured_date":null,"observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"include","score_raw":58.89,"score_norm":null,"config":"'8b dense' column, 14 languages, 5-shot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":73.81,"score_norm":null,"config":"'8b dense' column, instruct, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":82.32,"score_norm":null,"config":"'8b dense' column, 5 languages, 8-shot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"minerva_math","score_raw":80.1,"score_norm":null,"config":"'8b dense' column, 0-shot, cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":73.84,"score_norm":null,"config":"'8b dense' column of ibm's model-card eval table, 5-shot, instruct model","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":55.99,"score_norm":26.65,"config":"'8b dense' column, 5-shot, cot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"mmmlu","score_raw":64.84,"score_norm":null,"config":"'8b dense' column, 11 languages, 5-shot, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"mtbench_avg","score_raw":8.61,"score_norm":null,"config":"'8b dense' column, instruct, judge model not stated on card, not a 0-100 percentage — do not treat 8.61 as a percent, scale note: this is a 0-10 rating","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"multiple","score_raw":60.26,"score_norm":null,"config":"'8b dense' column, instruct, languages tested not enumerated on card, multi-language code-translation benchmark, pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"salad_bench","score_raw":95.8,"score_norm":null,"config":"'8b dense' column, instruct, safety benchmark reporting % safe responses — higher = safer, this is not an attack-success-rate metric","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":21.8,"score_norm":null,"config":"0-100 scale, likely an aa-derived component, site aggregates from multiple upstream sources without per-figure attribution","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":21.76,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.217592592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":4.82,"score_norm":null,"config":"'8b dense' column, 3-shot, factual-recall accuracy metric, higher=better (low absolute scores are typical/expected on simpleqa across model sizes), instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":27.8,"score_norm":10.4,"config":"0-100 scale, artificial analysis's own methodology text (fetched separately) names a related component 'τ3-banking' — naming/version between the two aa-linked mentions is not fully reconciled, labeled 'τ2-bench' on this page, site aggregates without per-figure attribution, tau2","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/ibm-granite-4-1-8b/","measured_date":"2026-08-09","observed_at":"2026-09-02","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":27.78,"score_norm":10.37,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.277777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.37,"score_norm":0,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0337078651685393 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/granite-4-1-8b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tulu3_safety_eval_avg","score_raw":75.57,"score_norm":null,"config":"'8b dense' column, higher = safer, ibm's own average across the tulu3 safety-eval suite sub-metrics, instruct","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/ibm-granite/granite-4.1-8b","measured_date":"2026-04-29","observed_at":"2026-09-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#ibm-granite/granite-4.1-8b","publisher":"OpenRouter (listed 2026-04-30)"}],"last_verified":"2026-09-02","last_enriched_at":null},{"id":"llama-3.1-70b-instruct","display_name":"Llama 3.1 70B Instruct","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"meta-llama/Meta-Llama-3.1-70B-Instruct","openrouter_id":"meta-llama/llama-3.1-70b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"llama3.1","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-07-23","release_status":"released","knowledge_cutoff":"2023-12-31","status":"superseded","supersedes":[],"superseded_by":"llama-3.3-70b-instruct","aliases":["meta-llama-3.1-70b-instruct"],"intelligence":{"index":8.48,"inputs":[{"benchmark":"aime","score_raw":4,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.04 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1293.3,"score_norm":6.09,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":40.91,"score_norm":22.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.409090909090909 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.47,"score_norm":3.63,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0447 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":15.2,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.152046783625731 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":0.4,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"meta-llama/llama-3.1-70b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.4,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":null,"min_serve_gb":150.984,"method":"vendor","source_url":"https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct/tree/main"},{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":null,"min_serve_gb":150.977,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":null,"min_serve_gb":81.344,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":null,"min_serve_gb":47.269,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":null,"min_serve_gb":81.344,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.6,"score_norm":6.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.1-70b-instruct","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":6.51,"score_norm":6.51,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":8,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.08 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":4,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.04 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1293.3,"score_norm":6.09,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1293.3,"score_norm":6.09,"config":"55240 votes, arena entry llama-3.1-70b-instruct, leaderboard text-overall-style_control, rank 269, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40.91,"score_norm":22.41,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.409090909090909 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.47,"score_norm":3.63,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0447 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":34.42,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.34421768707483 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":23.17,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.231746031746032 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":26.74,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.267361111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":15.2,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.152046783625731 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.03,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0303030303030303 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/llama-3-1-instruct-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta-llama/llama-3.1-70b-instruct","publisher":"OpenRouter (listed 2024-07-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-2407-large","display_name":"Mistral Large 2407","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"2407","size_class":"frontier-large","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-large-2407","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2024-11-19","release_status":"released","knowledge_cutoff":"2024-03-31","status":"superseded","supersedes":[],"superseded_by":"mistral-2512-large","aliases":["mistral-1-large-2407","mistral-large-2407"],"intelligence":{"index":15.06,"inputs":[{"benchmark":"aime","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-2407","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: jul '24, source value 0 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1314.4,"score_norm":15.5,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":47.17,"score_norm":31.23,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-2407","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: jul '24, source value 0.471717171717172 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":2.95,"score_norm":1.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-2407","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: jul '24, source value 0.0295 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":33.04,"score_norm":17.39,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large-2407","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: jul '24, source value 0.330409356725146 is a 0-1 fraction, x100 at harvest"}],"coverage":0.4762,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.2,"context_length":131072,"source":"openrouter","source_model_id":"mistralai/mistral-large-2407","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.6,"score_norm":7.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2407","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.04,"score_norm":7.04,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: jul '24","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":1.67,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: jul '24, source value 0.0166666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":0,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: jul '24, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1314.4,"score_norm":15.5,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1314.3,"score_norm":15.45,"config":"45459 votes, arena entry mistral-large-2407, leaderboard text-overall-style_control, rank 250, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":47.17,"score_norm":31.23,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: jul '24, source value 0.471717171717172 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":2.95,"score_norm":1.4,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: jul '24, source value 0.0295 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":31.63,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, run: jul '24, source value 0.316326530612245 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":26.67,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, run: jul '24, source value 0.266666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":27.08,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: jul '24, scicode, source value 0.270833333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":33.04,"score_norm":17.39,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: jul '24, source value 0.330409356725146 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large-2407","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large-2407","publisher":"OpenRouter (listed 2024-11-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-3.1-small-24b-2503-instruct","display_name":"Mistral-Small-3.1-24B-Instruct-2503","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"3.1","size_class":"24b","params_total_b":24.01,"params_active_b":24.01,"params_exact":24011361280,"params_source":"hf_safetensors","vram_est":{"bf16_gb":48,"int8_gb":25.5,"int4_gb":14.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"mistralai/Mistral-Small-3.1-24B-Instruct-2503","openrouter_id":"mistralai/mistral-small-3.1-24b-instruct","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text","image"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":128000,"release_date":"2025-03-17","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"mistral-3.2-small-24b-2506-instruct","aliases":["mistral-3.1-small-24b","mistral-small-3.1-24b","mistral-small-3.1-24b-instruct","mistral-small-3.1-24b-instruct-2503"],"intelligence":{"index":19.32,"inputs":[{"benchmark":"aime","score_raw":3.67,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"2025, artificial analysis independent evaluation"},{"benchmark":"arena_elo","score_raw":1303.1,"score_norm":10.46,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":45.35,"score_norm":28.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":4.31,"score_norm":3.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"artificial analysis independent evaluation"},{"benchmark":"mmlu_pro","score_raw":65.93,"score_norm":43.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"artificial analysis independent evaluation"},{"benchmark":"tau_bench","score_raw":25.15,"score_norm":6.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"artificial analysis independent evaluation, tau2/telecom"},{"benchmark":"terminal_bench","score_raw":26.22,"score_norm":24.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","config":"2.1, artificial analysis independent evaluation"}],"coverage":0.6428,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.351,"completion_usd_per_m":0.555,"cache_read_usd_per_m":null,"context_length":128000,"source":"openrouter","source_model_id":"mistralai/mistral-small-3.1-24b-instruct","as_of":"2026-10-06","blended_usd_per_m":0.402,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":48.02,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":51.763,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":48.023,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":51.766,"method":"vendor","source_url":"https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/tree/main"},{"quant":"fp8","weights_gb":25.451,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":28.066,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":14.406,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":16.748,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":25.451,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":28.066,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_lcr","score_raw":22.33,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":3.67,"score_norm":0,"config":"2025, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1303.1,"score_norm":10.46,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1303.6,"score_norm":10.68,"config":"33293 votes, arena entry mistral-small-3.1-24b-instruct-2503, leaderboard text-overall-style_control, rank 264, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1127.1,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1127.8,"score_norm":null,"config":"28980 votes, arena entry mistral-small-3.1-24b-instruct-2503, leaderboard vision-overall-style_control, rank 108, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":45.35,"score_norm":28.66,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.31,"score_norm":3.4,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":29.93,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":21.16,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":70.67,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":65.93,"score_norm":43.22,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":25.15,"score_norm":6.87,"config":"artificial analysis independent evaluation, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":26.22,"score_norm":24.96,"config":"2.1, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-small-3-1","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://mistral.ai/news/mistral-small-3-1/","publisher":"Mistral AI"},{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-small-3.1-24b-instruct","publisher":"OpenRouter (listed 2025-03-17)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-235b-a22b-reasoning-vision","display_name":"Qwen3-VL-235B-A22B-Thinking","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.67,"params_active_b":22,"params_exact":235670022896,"params_source":"hf_safetensors","vram_est":{"bf16_gb":471,"int8_gb":250,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3-VL-235B-A22B-Thinking","openrouter_id":"qwen/qwen3-vl-235b-a22b-thinking","is_moe":true,"variant_axis":"vision","kind":"model","modalities":["text","image"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-05-14","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":[],"superseded_by":"qwen-3-235b-a22b-instruct-vision","aliases":["qwen3-vl-235b-a22b-thinking"],"intelligence":{"index":74.26,"inputs":[{"benchmark":"aime","score_raw":89.7,"score_norm":87.13,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v2","config":"aime 2025. qwen3-vl technical report table 3, claude-opus-4 thinking 75.5. thinking mode, first data column (qwen3-vl-235b-a22b-thinking), no tools. [raw: aime-25], o3 medium 88.9(high), siblings: thinking-2507 92.3"},{"benchmark":"arena_elo","score_raw":1395,"score_norm":51.42,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":77.1,"score_norm":73.38,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v2","config":"claude-opus-4 thinking 79.6. note: row is labeled only 'g [raw: gpqa], first data column (qwen3-vl-235b-a22b-thinking), o3 medium 83.3(high), qwen3-vl technical report table 3, siblings in row: thinking-2507 81.1"},{"benchmark":"livebench","score_raw":79.6,"score_norm":88,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v2","config":"claude-opus-4 thinking 7 [raw: livebench 2024-11-25], first data column (qwen3-vl-235b-a22b-thinking), global average. qwen3-vl technical report table 3, livebench 2024-11-25 release, o3 medium 78.3, siblings: thinking-2507 78.4"},{"benchmark":"mmlu_pro","score_raw":83.8,"score_norm":73,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v2","config":"'comparison among qwen3-vl-235b-a22b (thinking) and other reasoning baselines', qwen3-vl technical report (arxiv:2511.21631) table 3, value read from the first data column = qwen3-vl-235b-a22b-thinkin [raw: mmlu-pro]"},{"benchmark":"tau_bench","score_raw":67,"score_norm":62.67,"trust":2,"source_url":"https://arxiv.org/html/2511.21631v2","config":"retail domain only. the report publishes three separate domain rows for this model and no average: tau2-retail 67.0, tau-2 (τ2-bench), tau2-airline 62.0, tau2-telecom 44.7. retail reported here becaus [raw: tau2-retail]"}],"coverage":0.6032,"subset_bias":{"offset":9.59,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.4,"completion_usd_per_m":4,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-vl-235b-a22b-thinking","as_of":"2026-10-06","blended_usd_per_m":1.3,"endpoint_expires_at":"2026-10-09"},"memory":[{"quant":"fp16","weights_gb":471.34,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":496.484,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":471.34,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":496.484,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Thinking/tree/main"},{"quant":"fp8","weights_gb":249.81,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.878,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":141.402,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":150.049,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":249.81,"kv_cache_gb_per_8k":1.5771,"min_serve_gb":263.878,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":13.4,"score_norm":13.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-235b-a22b-thinking","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":89.7,"score_norm":87.13,"config":"aime 2025. qwen3-vl technical report table 3, claude-opus-4 thinking 75.5. thinking mode, first data column (qwen3-vl-235b-a22b-thinking), no tools. [raw: aime-25], o3 medium 88.9(high), siblings: thinking-2507 92.3","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1395,"score_norm":51.42,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1395,"score_norm":51.42,"config":"7986 votes, arena entry qwen3-vl-235b-a22b-thinking, leaderboard text-overall-style_control, rank 151, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1188,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-29","observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1189.4,"score_norm":null,"config":"2356 votes, arena entry qwen3-vl-235b-a22b-thinking, leaderboard vision-overall-style_control, rank 79, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":77.1,"score_norm":73.38,"config":"claude-opus-4 thinking 79.6. note: row is labeled only 'g [raw: gpqa], first data column (qwen3-vl-235b-a22b-thinking), o3 medium 83.3(high), qwen3-vl technical report table 3, siblings in row: thinking-2507 81.1","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":79.6,"score_norm":88,"config":"claude-opus-4 thinking 7 [raw: livebench 2024-11-25], first data column (qwen3-vl-235b-a22b-thinking), global average. qwen3-vl technical report table 3, livebench 2024-11-25 release, o3 medium 78.3, siblings: thinking-2507 78.4","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":83.8,"score_norm":73,"config":"'comparison among qwen3-vl-235b-a22b (thinking) and other reasoning baselines', qwen3-vl technical report (arxiv:2511.21631) table 3, value read from the first data column = qwen3-vl-235b-a22b-thinkin [raw: mmlu-pro]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":67,"score_norm":62.67,"config":"retail domain only. the report publishes three separate domain rows for this model and no average: tau2-retail 67.0, tau-2 (τ2-bench), tau2-airline 62.0, tau2-telecom 44.7. retail reported here becaus [raw: tau2-retail]","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2511.21631v2","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Thinking","publisher":"Hugging Face"},{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-vl-235b-a22b-thinking","publisher":"OpenRouter (listed 2025-09-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3-30b-a3b-instruct","display_name":"Qwen3-30B-A3B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":30.53,"params_active_b":3,"params_exact":30532122624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":61.1,"int8_gb":32.4,"int4_gb":18.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3-30B-A3B","openrouter_id":"qwen/qwen3-30b-a3b","is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-04-29","release_status":"released","knowledge_cutoff":"2025-03-31","status":"superseded","supersedes":["qwen-3-30b-a3b-04-28"],"superseded_by":"qwen-3-30b-a3b-2507-instruct","aliases":["qwen-3-30b-a3b","qwen-3-30b-a3b-04-28","qwen3-30b-a3b","qwen3-30b-a3b-04-28"],"intelligence":{"index":30.93,"inputs":[{"benchmark":"aime","score_raw":72.33,"score_norm":65.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1326.5,"score_norm":20.89,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":61.62,"score_norm":51.58,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.616161616161616 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.16,"score_norm":6.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0616 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":26.02,"score_norm":8.03,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.260233918128655 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.5238,"subset_bias":{"offset":-3.01,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.12,"completion_usd_per_m":0.5,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"qwen/qwen3-30b-a3b","as_of":"2026-10-06","blended_usd_per_m":0.215,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":61.06,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.918,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":61.067,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":64.926,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3-30B-A3B/tree/main"},{"quant":"fp8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":18.318,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":20.123,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":32.362,"kv_cache_gb_per_8k":0.8053,"min_serve_gb":34.785,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":243.2,"quant":"int4","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":608,"params_active_b":3,"bytes_per_param":0.5,"quant":"int4"},"bw_ceiling_tok_s":405.33,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":54.6,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":91,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":123564.61,"quant":"fp8","batch_size":null,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"qwen-3-235b-a22b","value":8424.86,"params_active_b":22,"quant":"fp16","batch_size":null,"context_tokens":8192,"trust":0,"source_url":"https://docs.gpustack.ai/latest/performance-lab/qwen3-235b-a22b/h100/"},"scaling":"active_param_and_bytes_per_param_ratio","quant_scale":2,"quant":"fp8","params_active_b":3,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":670,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":3,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":1116.67,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.6,"score_norm":7.6,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":9.17,"score_norm":9.17,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":72.33,"score_norm":65.41,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1326.5,"score_norm":20.89,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1326.9,"score_norm":21.07,"config":"26549 votes, arena entry qwen3-30b-a3b, leaderboard text-overall-style_control, rank 231, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":948,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-10-01","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":974,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":911,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-09-30","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":952,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":961,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":61.62,"score_norm":51.58,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.616161616161616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.16,"score_norm":6.12,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.5,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.414965986394558 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":50.58,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.505820105820106 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":28.47,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.284722222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":26.02,"score_norm":8.03,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.260233918128655 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":6.82,"score_norm":2.14,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0681818181818182 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-30b-a3b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://qwenlm.github.io/blog/qwen3/","publisher":"Qwen"},{"url":"https://huggingface.co/Qwen/Qwen3-30B-A3B","publisher":"Hugging Face"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-27b","display_name":"Qwen3.5-27B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"27b","params_total_b":27.78,"params_active_b":27.78,"params_exact":27781427952,"params_source":"hf_safetensors","vram_est":{"bf16_gb":55.6,"int8_gb":29.4,"int4_gb":16.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3.5-27B","openrouter_id":"qwen/qwen3.5-27b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-02-25","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.8-27b","aliases":["qwen3.5-27b"],"intelligence":{"index":64.62,"inputs":[{"benchmark":"arena_elo","score_raw":1408.7,"score_norm":57.53,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":85.76,"score_norm":85.58,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.857575757575758 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":23.91,"score_norm":32.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.239110287303058 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":72.4,"score_norm":69.33,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","config":null},{"benchmark":"tau_bench","score_raw":93.86,"score_norm":98.48,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.93859649122807 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6032,"subset_bias":{"offset":0.32,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.195,"completion_usd_per_m":1.56,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.5-27b","as_of":"2026-10-06","blended_usd_per_m":0.53625,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":55.56,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.875,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":55.563,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":58.878,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B/tree/main"},{"quant":"fp8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":16.668,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":18.205,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":29.447,"kv_cache_gb_per_8k":0.5369,"min_serve_gb":31.456,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":22.9,"score_norm":22.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-27b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":34.6,"score_norm":34.6,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.723333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1408.7,"score_norm":57.53,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1407.8,"score_norm":57.13,"config":"27282 votes, arena entry qwen3.5-27b, leaderboard text-overall-style_control, rank 135, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1216.4,"score_norm":null,"config":"vision","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1219.1,"score_norm":null,"config":"23081 votes, arena entry qwen3.5-27b, leaderboard vision-overall-style_control, rank 60, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1357.3,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1357.5,"score_norm":null,"config":"7437 votes, arena entry qwen3.5-27b-code, code/overall, leaderboard webdev-overall-raw, rank 85, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.85,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00853061224489796 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.76,"score_norm":85.58,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.857575757575758 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.5,"score_norm":85.21,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":23.91,"score_norm":32.22,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.239110287303058 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":48.5,"score_norm":68.38,"config":"with tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","measured_date":null,"observed_at":"2026-09-27","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":24.3,"score_norm":32.79,"config":"chain of thought","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":75.58,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.75578231292517 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":35.5,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.354990583804143 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":75.03,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.75028901734104 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":39.47,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.394675925925926 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.4,"score_norm":69.33,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":93.86,"score_norm":98.48,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.93859649122807 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":32.58,"score_norm":32.45,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.325757575757576 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-27b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":41.6,"score_norm":43.06,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-27B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-27b","publisher":"OpenRouter (listed 2026-02-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-35b-a3b","display_name":"Qwen3.5-35B-A3B","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"35b-a3b","params_total_b":35.95,"params_active_b":3,"params_exact":35951822704,"params_source":"hf_safetensors","vram_est":{"bf16_gb":71.9,"int8_gb":38.1,"int4_gb":21.6,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen3.5-35B-A3B","openrouter_id":"qwen/qwen3.5-35b-a3b","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2026-02-25","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.6-35b-a3b","aliases":["qwen3.5-35b-a3b"],"intelligence":{"index":61.85,"inputs":[{"benchmark":"arena_elo","score_raw":1393.8,"score_norm":50.89,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":84.55,"score_norm":83.87,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.845454545454545 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":21.04,"score_norm":28,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.210379981464319 is a 0-1 fraction, x100 at harvest"},{"benchmark":"swe_bench_verified","score_raw":69.2,"score_norm":65.78,"trust":1,"source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B","config":null},{"benchmark":"tau_bench","score_raw":89.18,"score_norm":92.24,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.891812865497076 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":40.82,"score_norm":42.14,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b-non-reasoning","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.408239700374532 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.6032,"subset_bias":{"offset":-0.43,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":1,"cache_read_usd_per_m":0.05,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.5-35b-a3b","as_of":"2026-10-06","blended_usd_per_m":0.3625,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":71.9,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":75.663,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":71.904,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":75.667,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B/tree/main"},{"quant":"fp8","weights_gb":38.107,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":40.18,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":21.57,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":22.816,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":38.107,"kv_cache_gb_per_8k":0.1678,"min_serve_gb":40.18,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":11.8,"score_norm":11.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-35b-a3b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":11.82,"score_norm":11.82,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: non-reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":37,"score_norm":37,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-35b-a3b","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.3,"score_norm":19.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-35b-a3b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":29.91,"score_norm":29.91,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":68.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.683333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1393.8,"score_norm":50.89,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1395.3,"score_norm":51.56,"config":"29097 votes, arena entry qwen3.5-35b-a3b, leaderboard text-overall-style_control, rank 149, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1253.4,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1249.8,"score_norm":null,"config":"1539 votes, arena entry qwen3.5-35b-a3b-code, code/overall, leaderboard webdev-overall-raw, rank 105, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0.86,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, run: reasoning, source value 0.00857142857142857 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":14.81,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, run: non-reasoning, source value 0.14811000000000002 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.55,"score_norm":83.87,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.845454545454545 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":84.2,"score_norm":83.38,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":21.04,"score_norm":28,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.210379981464319 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":22.4,"score_norm":30,"config":"chain of thought","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":72.52,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.725170068027211 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"it_bench_sre","score_raw":21.52,"score_norm":null,"config":"aa field itbenchsre, artificial analysis intelligence index v4.1.1 suite, effort=none, it-bench sre, reasoning=true, run: reasoning, source value 0.215160075329567 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":72.66,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.726589595375723 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":37.73,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.377314814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":69.2,"score_norm":65.78,"config":null,"trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":89.18,"score_norm":92.24,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.891812865497076 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":4.95,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.0494845360824742 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":40.82,"score_norm":42.14,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.408239700374532 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b-non-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":26.52,"score_norm":25.32,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.265151515151515 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-5-35b-a3b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":40.5,"score_norm":41.76,"config":"2.0","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B","measured_date":null,"observed_at":"2026-09-27","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-35b-a3b","publisher":"OpenRouter (listed 2026-02-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.5-flash","display_name":"Qwen3.5-Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.5","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.5-flash-02-23","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-02-25","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.8-flash","aliases":["qwen-3.5-flash-02-23","qwen3.5-flash","qwen3.5-flash-02-23"],"intelligence":{"index":56.47,"inputs":[{"benchmark":"arena_elo","score_raw":1395.9,"score_norm":51.82,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":78.5,"score_norm":75.35,"trust":2,"source_url":"https://benchgecko.ai/model/qwen3-5-flash-02-23","config":"benchmark suite mirrors epoch ai benchmarking hub (otis mock aime, chess puzzles), frontiermath v2, model page titled \"qwen3.5-flash\" (api id qwen3-5-flash-02-23), no tools, simpleqa verified, though [raw: gpqa diamond]"},{"benchmark":"swe_bench_verified","score_raw":54.9,"score_norm":49.89,"trust":2,"source_url":"https://www.vals.ai/models/alibaba_qwen3.5-flash","config":"\"qwen 3.5 flash on vals index\" update dated mar 2, 2026, vals ai independent run, vals harness (resolved rate) — verbatim: \"it scores 54.9% on swe-bench verified but struggles on terminal-bench 2.0 (2 [raw: swe-bench verified]"},{"benchmark":"terminal_bench","score_raw":24.7,"score_norm":23.18,"trust":2,"source_url":"https://www.vals.ai/models/alibaba_qwen3.5-flash","config":"2026 vals index update, mar 2, same sentence as the swe-bench verified figure [raw: terminal-bench 2.0], terminal-bench 2.0, vals ai harness"}],"coverage":0.4445,"subset_bias":{"offset":3.34,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.065,"completion_usd_per_m":0.26,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.5-flash-02-23","as_of":"2026-10-06","blended_usd_per_m":0.11375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_elo","score_raw":1395.9,"score_norm":51.82,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1243.1,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":78.5,"score_norm":75.35,"config":"benchmark suite mirrors epoch ai benchmarking hub (otis mock aime, chess puzzles), frontiermath v2, model page titled \"qwen3.5-flash\" (api id qwen3-5-flash-02-23), no tools, simpleqa verified, though [raw: gpqa diamond]","trust":2,"source_type":"aggregator","source_url":"https://benchgecko.ai/model/qwen3-5-flash-02-23","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":54.9,"score_norm":49.89,"config":"\"qwen 3.5 flash on vals index\" update dated mar 2, 2026, vals ai independent run, vals harness (resolved rate) — verbatim: \"it scores 54.9% on swe-bench verified but struggles on terminal-bench 2.0 (2 [raw: swe-bench verified]","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/alibaba_qwen3.5-flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.7,"score_norm":23.18,"config":"2026 vals index update, mar 2, same sentence as the swe-bench verified figure [raw: terminal-bench 2.0], terminal-bench 2.0, vals ai harness","trust":2,"source_type":"aggregator","source_url":"https://www.vals.ai/models/alibaba_qwen3.5-flash","measured_date":null,"observed_at":"2026-08-19","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.5-flash-02-23","publisher":"OpenRouter (listed 2026-02-25)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.6-max","display_name":"Qwen3.6 Max Preview","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.6","size_class":"frontier-max","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.6-max-preview","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-04-27","release_status":"preview","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.7-max","aliases":["qwen3.6-max-preview"],"intelligence":{"index":71.93,"inputs":[{"benchmark":"arena_elo","score_raw":1460.3,"score_norm":80.53,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":88.79,"score_norm":89.85,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-max","config":"artificial analysis eval, qwen3.6 max preview"},{"benchmark":"hle","score_raw":30.82,"score_norm":42.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-max","config":"artificial analysis eval, qwen3.6 max preview"},{"benchmark":"swe_bench_verified","score_raw":72.8,"score_norm":69.78,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, runner=external, stderr=1.992, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-max","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":43.94,"score_norm":45.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen3-6-max","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.439393939393939 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.6032,"subset_bias":{"offset":-0.1,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.027,"completion_usd_per_m":6.162,"cache_read_usd_per_m":null,"context_length":262144,"source":"openrouter","source_model_id":"qwen/qwen3.6-max-preview","as_of":"2026-10-06","blended_usd_per_m":2.3107499999999996,"endpoint_expires_at":"2026-10-09"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":28.4,"score_norm":28.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-max-preview","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":41.07,"score_norm":41.07,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":72,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, source value 0.72 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aa_omniscience_index","score_raw":9.17,"score_norm":null,"config":"aa field omniscience, aa-omniscience index, artificial analysis intelligence index v4.1.1 suite, can be negative), effort=none, index unit -100..100 emitted as-is (accuracy minus hallucination penalty, reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1460.3,"score_norm":80.53,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-03","observed_at":"2026-10-03","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1460.1,"score_norm":80.44,"config":"5196 votes, arena entry kizen-beta, leaderboard text-overall-style_control, rank 48, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1481.5,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1478.9,"score_norm":null,"config":"2602 votes, arena entry qwen3.6-max-preview-code, code/overall, leaderboard webdev-overall-raw, rank 39, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":3.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=true, source value 0.0370612244897959 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.79,"score_norm":89.85,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, source value 0.887878787878788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.79,"score_norm":89.85,"config":"artificial analysis eval, qwen3.6 max preview","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.82,"score_norm":42.38,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, source value 0.308155699721965 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":30.82,"score_norm":42.38,"config":"artificial analysis eval, qwen3.6 max preview","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":76.6,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, source value 0.765986394557823 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":46.88,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, scicode, source value 0.46875 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.8,"score_norm":69.78,"config":"edition=swe-bench verified v1, runner=external, stderr=1.992, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.8,"score_norm":69.78,"config":"overall","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-14","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":95.91,"score_norm":100,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.95906432748538 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":43.94,"score_norm":45.81,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, source value 0.439393939393939 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen3-6-max","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-max-preview","publisher":"OpenRouter (listed 2026-04-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.1-codex","display_name":"GPT-5.1-Codex","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.1","size_class":"frontier-codex","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.1-codex","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-11-13","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.3-codex","aliases":[],"intelligence":{"index":69.05,"inputs":[{"benchmark":"aime","score_raw":95.67,"score_norm":94.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.956666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":85.96,"score_norm":85.86,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.85959595959596 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":25.72,"score_norm":34.88,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.257182576459685 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":83.04,"score_norm":84.05,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.830409356725146 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":3.26,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":10,"cache_read_usd_per_m":0.13,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.1-codex","as_of":"2026-10-06","blended_usd_per_m":3.4375,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":23.7,"score_norm":23.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":35.6,"score_norm":35.6,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":69,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.69 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":95.67,"score_norm":94.59,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.956666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1337.1,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1336.3,"score_norm":null,"config":"4873 votes, arena entry gpt-5.1-codex, code/overall, leaderboard webdev-overall-raw, rank 92, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":5.71,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0571428571428571 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1158,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1222,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":1086,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":1151,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1191,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1244,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1011,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1168,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":85.96,"score_norm":85.86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.85959595959596 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":25.72,"score_norm":34.88,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.257182576459685 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":70,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.7 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":84.87,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.848677248677249 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":72.49,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.72485549132948 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.16,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.40162037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":83.04,"score_norm":84.05,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.830409356725146 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":34.85,"score_norm":35.12,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.348484848484849 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-1-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.1-codex","publisher":"OpenRouter (listed 2025-11-13)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.2-codex","display_name":"GPT-5.2-Codex","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.2","size_class":"frontier-codex","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.2-codex","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-12-18","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.3-codex","aliases":[],"intelligence":{"index":72.17,"inputs":[{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.898989898989899 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":35.73,"score_norm":49.6,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.357275254865616 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":73.98,"score_norm":75.51,"trust":3,"source_url":"https://livebench.ai/","config":"livebench-2026-06-25 release"},{"benchmark":"swe_bench_verified","score_raw":72.4,"score_norm":69.33,"trust":3,"source_url":"https://www.vals.ai/benchmarks/swebench","config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.001, task=overall, updated=2026-08-19"},{"benchmark":"tau_bench","score_raw":92.11,"score_norm":96.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.921052631578947 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":37.12,"score_norm":37.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.371212121212121 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.643,"subset_bias":{"offset":-1.32,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":1.75,"completion_usd_per_m":14,"cache_read_usd_per_m":0.175,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.2-codex","as_of":"2026-10-06","blended_usd_per_m":4.8125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":28.5,"score_norm":28.5,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":41.22,"score_norm":41.22,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=xhigh, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: xhigh","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":79.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=xhigh, reasoning=true, run: xhigh, source value 0.793333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1338.9,"score_norm":null,"config":"webdev","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-02","observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"arena_elo_webdev","score_raw":1338.3,"score_norm":null,"config":"6424 votes, arena entry gpt-5.2-codex, code/overall, leaderboard webdev-overall-raw, rank 91, raw (no style control), vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/code","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":8.67,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.0866938775510204 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_androidnative","score_raw":1176,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_fullstack","score_raw":991,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_godotgamedev","score_raw":1142,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"design_arena_mobileapps","score_raw":1087,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1043,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.9,"score_norm":91.41,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, gpqa diamond, reasoning=true, run: xhigh, source value 0.898989898989899 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":35.73,"score_norm":49.6,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, humanity's last exam, reasoning=true, run: xhigh, source value 0.357275254865616 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":77.62,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, ifbench, reasoning=true, run: xhigh, source value 0.776190476190476 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":73.98,"score_norm":75.51,"config":"livebench-2026-06-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmmu_pro","score_raw":76.3,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, mmmu-pro, reasoning=true, run: xhigh, source value 0.763005780346821 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":54.63,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, scicode, source value 0.546296296296296 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":72.4,"score_norm":69.33,"config":"edition=swe-bench verified v1, reasoning_effort=high, runner=external, stderr=2.001, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/swebench","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":92.11,"score_norm":96.15,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.921052631578947 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.12,"score_norm":37.79,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=xhigh, reasoning=true, run: xhigh, source value 0.371212121212121 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-2-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-codex","publisher":"OpenRouter (listed 2026-01-14)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"kimi-k2","display_name":"Kimi K2","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2","size_class":"frontier-k","params_total_b":1026.41,"params_active_b":32,"params_exact":1026408235864,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2053,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"moonshotai/Kimi-K2-Instruct","openrouter_id":"moonshotai/kimi-k2","is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-07","release_status":"released","knowledge_cutoff":"2024-12-31","status":"superseded","supersedes":["kimi-k2-1t-instruct"],"superseded_by":"kimi-k2-0905","aliases":["kimi-k2-1t","kimi-k2-1t-instruct","kimi-k2-instruct"],"intelligence":{"index":55.5,"inputs":[{"benchmark":"aime","score_raw":62.71,"score_norm":53.39,"trust":3,"source_url":"https://www.vals.ai/benchmarks/aime","config":"archived=true, edition=aime v1, runner=platform, stderr=1.217, task=overall, updated=2026-04-16"},{"benchmark":"gpqa_diamond","score_raw":71.46,"score_norm":65.44,"trust":3,"source_url":"https://www.vals.ai/benchmarks/gpqa","config":"edition=gpqa diamond v1, runner=platform, stderr=2.266, task=overall, updated=2026-08-19"},{"benchmark":"hle","score_raw":7.41,"score_norm":7.96,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0741427247451344 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":76.4,"score_norm":80.89,"trust":1,"source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","config":"pass@1"},{"benchmark":"mmlu_pro","score_raw":79.39,"score_norm":65.65,"trust":3,"source_url":"https://www.vals.ai/benchmarks/mmlu_pro","config":"edition=mmlu pro v1, runner=platform, stderr=0.394, task=overall, updated=2026-08-19"},{"benchmark":"swe_bench_verified","score_raw":65.8,"score_norm":62,"trust":1,"source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","config":"agentic coding, single attempt, bash/editor tools, 8k output limit"},{"benchmark":"tau_bench","score_raw":61.11,"score_norm":54.81,"trust":2,"source_url":"https://artificialanalysis.ai/models/kimi-k2","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.611111111111111 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":27.8,"score_norm":26.82,"trust":2,"source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","config":"2.0, agent: terminus 2"}],"coverage":0.8413,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.57,"completion_usd_per_m":2.3,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"moonshotai/kimi-k2","as_of":"2026-10-06","blended_usd_per_m":1.0025,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":2052.82,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":2156.037,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1087.995,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1142.97,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":1029.191,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1081.226,"method":"vendor","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct/tree/main"},{"quant":"int4","weights_gb":615.846,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":647.214,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":1087.995,"kv_cache_gb_per_8k":0.5757,"min_serve_gb":1142.97,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.7,"score_norm":12.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.65,"score_norm":19.65,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":53,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.53 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":62.71,"score_norm":53.39,"config":"archived=true, edition=aime v1, runner=platform, stderr=1.217, task=overall, updated=2026-04-16","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/aime","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":57,"score_norm":46.25,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.57 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":49.5,"score_norm":36.88,"config":"avg@64, non-thinking","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_codecategories","score_raw":1039,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_dataviz","score_raw":1025,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-10-03","admissible":false,"excluded":null},{"benchmark":"design_arena_gamedev","score_raw":987,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_uicomponent","score_raw":1037,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"design_arena_website","score_raw":1057,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":71.46,"score_norm":65.44,"config":"edition=gpqa diamond v1, runner=platform, stderr=2.266, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/gpqa","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.57,"score_norm":72.63,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.765656565656566 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":75.1,"score_norm":70.56,"config":"avg@8","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":7.41,"score_norm":7.96,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0741427247451344 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.7,"score_norm":3.97,"config":"text only, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":41.5,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.414965986394558 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":76.4,"score_norm":80.89,"config":"pass@1","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":55.56,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.555555555555556 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":53.7,"score_norm":null,"config":"pass@1, aug24-may25 window","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":97.4,"score_norm":null,"config":"accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":79.39,"score_norm":65.65,"config":"edition=mmlu pro v1, runner=platform, stderr=0.394, task=overall, updated=2026-08-19","trust":3,"source_type":"leaderboard","source_url":"https://www.vals.ai/benchmarks/mmlu_pro","measured_date":"2026-08-19","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":81.1,"score_norm":68.5,"config":"em","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":34.49,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.344907407407407 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"swe_bench_multilingual","score_raw":47.3,"score_norm":null,"config":"agentic coding, single attempt","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":27.67,"score_norm":37.78,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://scale.com/leaderboard/swe_bench_pro_public","measured_date":"2026-04-23","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"swe_bench_pro","score_raw":27.67,"score_norm":37.78,"config":"post-release eval section of model card","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":65.8,"score_norm":62,"config":"agentic coding, single attempt, bash/editor tools, 8k output limit","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":61.11,"score_norm":54.81,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.611111111111111 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench","score_raw":70.6,"score_norm":67.47,"config":"avg@4, tau2/retail","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":27.8,"score_norm":26.82,"config":"2.0, agent: terminus 2","trust":2,"source_type":"aggregator","source_url":"https://www.tbench.ai/leaderboard/terminal-bench/2.0","measured_date":"2025-11-01","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":15.91,"score_norm":12.84,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.159090909090909 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/kimi-k2","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":30,"score_norm":29.41,"config":"in-house framework","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#moonshotai/kimi-k2","publisher":"OpenRouter (listed 2025-07-11)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gemini-2.5-flash-image","display_name":"Nano Banana (Gemini 2.5 Flash Image)","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-image-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-flash-image","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":32768,"release_date":"2025-10-07","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"gemini-3.1-flash-image","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.03,"context_length":32768,"source":"openrouter","source_model_id":"google/gemini-2.5-flash-image","as_of":"2026-10-06","blended_usd_per_m":0.85,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"design_arena_graphicdesign","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-image","measured_date":null,"observed_at":"2026-09-29","admissible":false,"excluded":null},{"benchmark":"design_arena_image","score_raw":1189,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-image","measured_date":null,"observed_at":"2026-09-10","admissible":false,"excluded":null},{"benchmark":"design_arena_logo","score_raw":1170,"score_norm":null,"config":"arena=models","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-image","measured_date":null,"observed_at":"2026-10-02","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-image","publisher":"OpenRouter (listed 2025-10-07)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nemotron-3-super","display_name":"Nemotron 3 Super","family":{"id":"nemotron","vendor":"NVIDIA"},"generation":"3","size_class":"frontier","params_total_b":120,"params_active_b":12,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":240,"int8_gb":127,"int4_gb":72,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B","openrouter_id":null,"is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"NVIDIA Open Model License","is_open_weights":true,"self_hostable":true,"context_window_tok":262144,"release_date":"2025-12-15","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"nemotron-3.5-lightning","aliases":["nvidia-nemotron-3-super-120b-a12b","nvidia-nemotron-3-super-120b-a12b-bf16"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":240,"kv_cache_gb_per_8k":null,"min_serve_gb":252.48,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":127.2,"kv_cache_gb_per_8k":null,"min_serve_gb":134.04,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":72,"kv_cache_gb_per_8k":null,"min_serve_gb":76.08,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":127.2,"kv_cache_gb_per_8k":null,"min_serve_gb":134.04,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_elo","score_raw":1360.5,"score_norm":36.04,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-10-01","observed_at":"2026-10-01","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B","publisher":"Hugging Face"}],"last_verified":"2026-09-28","last_enriched_at":null},{"id":"granite-3.2-8b-instruct","display_name":"granite-3.2-8b-instruct","family":{"id":"granite","vendor":"IBM"},"generation":"3.2","size_class":"8b","params_total_b":8.17,"params_active_b":8,"params_exact":8170848256,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.3,"int8_gb":8.7,"int4_gb":4.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"ibm-granite/granite-3.2-8b-instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-02-26","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"granite-3.3-8b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":16.34,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":18.682,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":16.342,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":18.684,"method":"vendor","source_url":"https://huggingface.co/ibm-granite/granite-3.2-8b-instruct/tree/main"},{"quant":"fp8","weights_gb":8.66,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.002,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":4.902,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":7.244,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":8.66,"kv_cache_gb_per_8k":1.3422,"min_serve_gb":11.002,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://www.ibm.com/new/announcements/ibm-granite-3-2-open-source-reasoning-and-vision","publisher":"IBM"}],"last_verified":"2026-10-01","last_enriched_at":null},{"id":"qwen-2.5-14b-instruct","display_name":"Qwen2.5-14B-Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"14b","params_total_b":14.77,"params_active_b":14.77,"params_exact":14770033664,"params_source":"hf_safetensors","vram_est":{"bf16_gb":29.5,"int8_gb":15.7,"int4_gb":8.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen2.5-14B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2024-09","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3-14b-instruct","aliases":["qwen2.5-14b-instruct"],"intelligence":{"index":19,"inputs":[{"benchmark":"gpqa_diamond","score_raw":45.5,"score_norm":28.87,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"5-shot, qwen2.5 technical report table 7, subset not specified by the source (not stated as diamond)"},{"benchmark":"livebench","score_raw":44.4,"score_norm":9.78,"trust":2,"source_url":"https://arxiv.org/html/2412.15115v2","config":"livebench 0831, qwen2.5 technical report table 7, qwen2.5-14b column"},{"benchmark":"mmlu_pro","score_raw":49.04,"score_norm":15.07,"trust":3,"source_url":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard","config":"5-shot raw accuracy (leaderboard-normalised value 43.38), bfloat16, huggingface open llm leaderboard v2"}],"coverage":0.3969,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":29.54,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":32.628,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":29.54,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":32.628,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-14B-Instruct/tree/main"},{"quant":"fp8","weights_gb":15.656,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":18.267,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":8.862,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":11.473,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":15.656,"kv_cache_gb_per_8k":1.6106,"min_serve_gb":18.267,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"arena_hard","score_raw":68.3,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":45.5,"score_norm":28.87,"config":"5-shot, qwen2.5 technical report table 7, subset not specified by the source (not stated as diamond)","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":83.5,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":81,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":44.4,"score_norm":9.78,"config":"livebench 0831, qwen2.5 technical report table 7, qwen2.5-14b column","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":42.6,"score_norm":null,"config":"2305-2409, qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"math","score_raw":80,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mbpp","score_raw":82,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":49.04,"score_norm":15.07,"config":"5-shot raw accuracy (leaderboard-normalised value 43.38), bfloat16, huggingface open llm leaderboard v2","trust":3,"source_type":"leaderboard","source_url":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_pro","score_raw":63.7,"score_norm":39.5,"config":"qwen2.5 technical report table 7 (14b-30b+ instruction-tuned models), qwen2.5-14b column","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"mmlu_redux","score_raw":80,"score_norm":null,"config":"qwen2.5 technical report table 7","trust":2,"source_type":"paper","source_url":"https://arxiv.org/html/2412.15115v2","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/collections/Qwen/qwen25","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-2.5-32b-instruct","display_name":"Qwen2.5-32B-Instruct","family":{"id":"qwen","vendor":"Alibaba"},"generation":"2.5","size_class":"32b","params_total_b":32.76,"params_active_b":32.76,"params_exact":32763876352,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.5,"int8_gb":34.7,"int4_gb":19.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"Qwen/Qwen2.5-32B-Instruct","openrouter_id":null,"is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2024-09","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3-32b-instruct","aliases":["qwen2.5-32b-instruct"],"intelligence":{"index":28.37,"inputs":[{"benchmark":"gpqa_diamond","score_raw":46.57,"score_norm":30.38,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":3.99,"score_norm":2.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0399 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":69.66,"score_norm":49.43,"trust":2,"source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","config":"artificial analysis independent evaluation"}],"coverage":0.3889,"subset_bias":null},"pricing":null,"memory":[{"quant":"fp16","weights_gb":65.52,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.943,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":65.528,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":70.952,"method":"vendor","source_url":"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct/tree/main"},{"quant":"fp8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":19.656,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":22.804,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":34.726,"kv_cache_gb_per_8k":2.1475,"min_serve_gb":38.61,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":7.23,"score_norm":7.23,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.57,"score_norm":30.38,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":46.57,"score_norm":30.38,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.465656565656566 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.99,"score_norm":2.93,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.99,"score_norm":2.93,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0399 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":24.76,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":24.76,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.247619047619048 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":80.53,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":69.66,"score_norm":49.43,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":22.92,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.229166666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/qwen2.5-32b-instruct","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/collections/Qwen/qwen25","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"grok-4.20-multi-agent","display_name":"Grok 4.20 Multi-Agent","family":{"id":"grok","vendor":"xAI"},"generation":"4.20","size_class":"frontier","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"x-ai/grok-4.20-multi-agent","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":2000000,"release_date":"2026-03-31","release_status":"released","knowledge_cutoff":"2025-09-01","status":"superseded","supersedes":[],"superseded_by":"grok-4.7","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":1.25,"completion_usd_per_m":2.5,"cache_read_usd_per_m":0.2,"context_length":2000000,"source":"openrouter","source_model_id":"x-ai/grok-4.20-multi-agent","as_of":"2026-10-06","blended_usd_per_m":1.5625,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arc_agi_2","score_raw":53.3,"score_norm":53.67,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/grok-4-20-beta","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1470.2,"score_norm":84.94,"config":"60864 votes, arena entry grok-4.20-multi-agent-beta-0309, leaderboard text-overall-style_control, rank 36, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1496,"score_norm":96.44,"config":"lmarena text leaderboard snapshot","trust":0,"source_type":"community","source_url":"https://www.swfte.com/lmarena","measured_date":"2026-08-17","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"arena_elo_search","score_raw":1205.5,"score_norm":null,"config":"87118 votes, arena entry grok-4.20-multi-agent-beta-0309, leaderboard search-overall-raw, rank 10, raw (no style control), search/overall, vote cutoff 2026-07-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1251.8,"score_norm":null,"config":"22919 votes, arena entry grok-4.20-multi-agent-beta-0309, leaderboard vision-overall-style_control, rank 40, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":88.5,"score_norm":89.44,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/grok-4-20-beta","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":31.6,"score_norm":43.53,"config":"no tools","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/grok-4-20-beta","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"swe_bench_verified","score_raw":76.7,"score_norm":74.11,"config":null,"trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/grok-4-20-beta","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":96.5,"score_norm":100,"config":"high/xhigh = 16 agents, low/medium = 4 agents, multi-agent beta, tau2","trust":2,"source_type":"aggregator","source_url":"https://designforonline.com/ai-models/xai-grok-4-20-multi-agent-beta/","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":47.1,"score_norm":49.53,"config":"terminal-bench 2.0","trust":0,"source_type":"aggregator","source_url":"https://benchlm.ai/models/grok-4-20-beta","measured_date":null,"observed_at":"2026-08-18","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#x-ai/grok-4.20-multi-agent","publisher":"OpenRouter (listed 2026-03-31)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"phi-4-14b-instruct","display_name":"Phi 4","family":{"id":"phi","vendor":"Microsoft"},"generation":"4","size_class":"14b","params_total_b":14.66,"params_active_b":14.66,"params_exact":14659507200,"params_source":"hf_safetensors","vram_est":{"bf16_gb":29.3,"int8_gb":15.5,"int4_gb":8.8,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"microsoft/phi-4","openrouter_id":"microsoft/phi-4","is_moe":false,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_mid","license":"mit","is_open_weights":true,"self_hostable":true,"context_window_tok":16384,"release_date":"2024-12-12","release_status":"released","knowledge_cutoff":"2024-06-30","status":"retired","supersedes":[],"superseded_by":null,"aliases":["phi-4","phi-4-14b"],"intelligence":{"index":20.06,"inputs":[{"benchmark":"aime","score_raw":18,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/phi-4","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.18 is a 0-1 fraction, x100 at harvest"},{"benchmark":"arena_elo","score_raw":1256.3,"score_norm":0,"trust":3,"source_url":"https://lmarena.ai/leaderboard","config":"text"},{"benchmark":"gpqa_diamond","score_raw":56.1,"score_norm":43.8,"trust":3,"source_url":"https://arxiv.org/html/2412.08905v1","config":"simple-evals, temperature=0.5"},{"benchmark":"hle","score_raw":3.76,"score_norm":2.59,"trust":2,"source_url":"https://artificialanalysis.ai/models/phi-4","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0376 is a 0-1 fraction, x100 at harvest"},{"benchmark":"livebench","score_raw":47.6,"score_norm":16.89,"trust":3,"source_url":"https://arxiv.org/html/2412.08905v1","config":"as reported in phi-4 tech report"},{"benchmark":"mmlu_pro","score_raw":70.4,"score_norm":50.67,"trust":3,"source_url":"https://arxiv.org/html/2412.08905v1","config":null},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/phi-4","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/phi-4","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.7619,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.07,"completion_usd_per_m":0.14,"cache_read_usd_per_m":null,"context_length":16384,"source":"openrouter","source_model_id":"microsoft/phi-4","as_of":"2026-10-06","blended_usd_per_m":0.08750000000000001,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":30.786,"kv_cache_gb_per_8k":null,"min_serve_gb":32.765,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":15.393,"kv_cache_gb_per_8k":null,"min_serve_gb":16.833,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":7.697,"kv_cache_gb_per_8k":null,"min_serve_gb":9.137,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":15.393,"kv_cache_gb_per_8k":null,"min_serve_gb":16.833,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"intel_b70","metric":"single_stream_tok_s","value":24.88,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":608,"params_active_b":14.66,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":41.47,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":11.17,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":14.66,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":18.62,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":25045.45,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"deepseek-v4-flash-reasoning","value":1262,"params_active_b":290.94,"quant":"fp8","batch_size":32,"context_tokens":8192,"trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash"},"scaling":"active_param_ratio","quant_scale":1,"quant":"fp8","params_active_b":14.66,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":137.11,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":14.66,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":228.51,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.9,"score_norm":5.9,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#microsoft/phi-4","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.55,"score_norm":4.55,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":0,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":18,"score_norm":0,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.18 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1256.3,"score_norm":0,"config":"text","trust":3,"source_type":"leaderboard","source_url":"https://lmarena.ai/leaderboard","measured_date":"2026-09-27","observed_at":"2026-09-27","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1256.2,"score_norm":0,"config":"24126 votes, arena entry phi-4, leaderboard text-overall-style_control, rank 298, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1221,"score_norm":0,"config":"213 votes, 25, 95% ci +4/-3, daily mirror of the official lmarena leaderboard api, microsoft, mit, model row \"phi-4\", overall (no style control), rank 108 (ub) / 116 (stylectrl), snapshot d [raw: lmarena (chatbot arena) arena score], text arena","trust":0,"source_type":"community","source_url":"https://huggingface.co/datasets/mathewhe/chatbot-arena-elo/raw/main/elo.csv","measured_date":null,"observed_at":"2026-08-19","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"drop","score_raw":75.5,"score_norm":null,"config":"simple-evals","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":56.1,"score_norm":43.8,"config":"simple-evals, temperature=0.5","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":57.47,"score_norm":45.73,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.574747474747475 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.76,"score_norm":2.59,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0376 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"humaneval","score_raw":82.8,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"humaneval","score_raw":82.6,"score_norm":null,"config":"simple-evals","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"ifbench","score_raw":23.54,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.235374149659864 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":63,"score_norm":null,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"livebench","score_raw":47.6,"score_norm":16.89,"config":"as reported in phi-4 tech report","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":23.07,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.230687830687831 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math","score_raw":80.4,"score_norm":null,"config":"simple-evals, temperature=0.5","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mgsm","score_raw":80.6,"score_norm":null,"config":"simple-evals","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":84.8,"score_norm":null,"config":"simple-evals, temperature=0.5","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":70.4,"score_norm":50.67,"config":null,"trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"phibench","score_raw":56.2,"score_norm":null,"config":"internal microsoft benchmark","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":26.04,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.260416666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"simpleqa","score_raw":3,"score_norm":null,"config":"simple-evals","trust":3,"source_type":"paper","source_url":"https://arxiv.org/html/2412.08905v1","measured_date":"2024-12-12","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":0,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":3.79,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/phi-4","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/microsoft/phi-4","publisher":"Hugging Face"},{"url":"https://arxiv.org/html/2412.08905v1","publisher":"arXiv"},{"url":"https://openrouter.ai/api/v1/models#microsoft/phi-4","publisher":"OpenRouter (listed 2025-01-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"llama-3.1-nemotron-ultra-253b","display_name":"Llama-3.1-Nemotron-Ultra-253B","family":{"id":"llama","vendor":"Meta"},"generation":"3.1","size_class":"253b","params_total_b":253,"params_active_b":253,"params_exact":null,"params_source":null,"vram_est":{"bf16_gb":506,"int8_gb":268,"int4_gb":152,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"NVIDIA Open Model License","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2025-03-18","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"llama-3.1-nemotron-ultra-253b-v1","aliases":["nemotron-llama-3.1-ultra-253b"],"intelligence":{"index":45.33,"inputs":[{"benchmark":"aime","score_raw":74.7,"score_norm":68.38,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, reasoning variant"},{"benchmark":"gpqa_diamond","score_raw":72.8,"score_norm":67.32,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, reasoning variant"},{"benchmark":"hle","score_raw":8.1,"score_norm":8.97,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, reasoning variant"},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, reasoning variant"},{"benchmark":"tau_bench","score_raw":11.4,"score_norm":0,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, domain not stated on page, reasoning variant, tau2"},{"benchmark":"terminal_bench","score_raw":2.3,"score_norm":0,"trust":2,"source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","config":"artificial analysis independent evaluation, reasoning variant"}],"coverage":0.5635,"subset_bias":{"offset":0.3,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[{"quant":"fp16","weights_gb":506,"kv_cache_gb_per_8k":5.4358,"min_serve_gb":536.736,"method":"derived_estimate","source_url":null},{"quant":"fp8","weights_gb":268.18,"kv_cache_gb_per_8k":5.4358,"min_serve_gb":287.025,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":151.8,"kv_cache_gb_per_8k":5.4358,"min_serve_gb":164.826,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":151,"kv_cache_gb_per_8k":null,"min_serve_gb":151,"method":"vendor","source_url":"https://huggingface.co/DevQuasar/nvidia.Llama-3_1-Nemotron-Ultra-253B-v1-GGUF"},{"quant":"int8","weights_gb":268.18,"kv_cache_gb_per_8k":5.4358,"min_serve_gb":287.025,"method":"derived_estimate","source_url":null}],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":9.1,"score_norm":9.1,"config":"aa intelligence index, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":15,"score_norm":15,"config":"artificial analysis intelligence index, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/llama-3-1-nemotron-ultra-253b-v1-reasoning/","measured_date":"2026-03-26","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":74.7,"score_norm":68.38,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":63.7,"score_norm":54.63,"config":"2025, artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":16.67,"score_norm":0,"config":"25, 32k sequence length, avg of up to 16 runs, greedy decoding, pass@1, reasoning off","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":72.5,"score_norm":65.63,"config":"25, 32k sequence length, avg of up to 16 runs, pass@1, reasoning on, temperature=0.6, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"bfcl_v2_live","score_raw":73.62,"score_norm":null,"config":"32k sequence length, avg of up to 16 runs, reasoning off","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"bfcl_v2_live","score_raw":74.1,"score_norm":null,"config":"32k sequence length, avg of up to 16 runs, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"coding_index","score_raw":13.1,"score_norm":null,"config":"artificial analysis coding index, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/llama-3-1-nemotron-ultra-253b-v1-reasoning/","measured_date":"2026-03-26","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":72.8,"score_norm":67.32,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":56.6,"score_norm":44.51,"config":"32k sequence length, avg of up to 16 runs, greedy decoding, pass@1, reasoning off","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":76.01,"score_norm":71.85,"config":"32k sequence length, avg of up to 16 runs, pass@1, reasoning on, temperature=0.6, top_p=0.95","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":8.1,"score_norm":8.97,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":38.2,"score_norm":null,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"ifeval","score_raw":89.45,"score_norm":null,"config":"compared vs deepseek r1 in vendor evaluation writeup, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://venturebeat.com/technology/nvidias-new-llama-3-1-nemotron-ultra-outperforms-deepseek-r1-at-half-the-size","measured_date":"2025-04-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"judgebench_overall_score","score_raw":79.14,"score_norm":null,"config":"reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://socket.dev/huggingface/package/unsloth/llama-3_1-nemotron-ultra-253b-v1-gguf","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":64.1,"score_norm":null,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":29.03,"score_norm":null,"config":"20240801.20250201, 32k sequence length, avg of up to 16 runs, pass@1, reasoning off","trust":1,"source_type":"vendor_claim","source_url":"https://build.nvidia.com/nvidia/llama-3_1-nemotron-ultra-253b-v1/modelcard","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":66.31,"score_norm":null,"config":"20240801.20250201, 32k sequence length, avg of up to 16 runs, pass@1, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://build.nvidia.com/nvidia/llama-3_1-nemotron-ultra-253b-v1/modelcard","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"long_context_reasoning","score_raw":7.3,"score_norm":null,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"math500","score_raw":80.4,"score_norm":null,"config":"32k sequence length, avg of up to 16 runs, pass@1, reasoning off","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"math500","score_raw":97,"score_norm":null,"config":"32k sequence length, avg of up to 16 runs, pass@1, reasoning on","trust":1,"source_type":"vendor_claim","source_url":"https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1/blob/main/README.md","measured_date":"2025-04-07","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":95.2,"score_norm":null,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"math_index","score_raw":63.7,"score_norm":null,"config":"artificial analysis math index, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"math_index","score_raw":63.7,"score_norm":null,"config":"artificial analysis math index, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://dataconomy.com/ai-models/llama-3-1-nemotron-ultra-253b-v1-reasoning/","measured_date":"2026-03-26","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":82.5,"score_norm":70.83,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":34.7,"score_norm":null,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":11.4,"score_norm":0,"config":"artificial analysis independent evaluation, domain not stated on page, reasoning variant, tau2","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":2.3,"score_norm":0,"config":"artificial analysis independent evaluation, reasoning variant","trust":2,"source_type":"aggregator","source_url":"https://computeprices.com/models/llama-3-1-nemotron-ultra-253b-v1","measured_date":"2026-08","observed_at":"2026-09-13","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://developer.nvidia.com/blog/build-enterprise-ai-agents-with-advanced-open-nvidia-llama-nemotron-reasoning-models","publisher":"NVIDIA"},{"url":"https://inferencebench.io/models/nvidia/nemotron-ultra-253b/","publisher":"InferenceBench"}],"last_verified":"2026-09-28","last_enriched_at":"2026-09-13T06:18:27.520Z"},{"id":"gpt-5-pro","display_name":"GPT-5 Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-10-06","release_status":"released","knowledge_cutoff":"2024-09-30","status":"superseded","supersedes":[],"superseded_by":"gpt-5.5-pro","aliases":["gpt-5-pro-2025-10-06"],"intelligence":{"index":65.23,"inputs":[{"benchmark":"aime","score_raw":96.7,"score_norm":95.88,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5/","config":"aime 2025, no tools"},{"benchmark":"arc_agi_2","score_raw":18.33,"score_norm":14.81,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=7.1432, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":88.4,"score_norm":89.3,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5/","config":"no tools"},{"benchmark":"hle","score_raw":30.7,"score_norm":42.21,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5/","config":"full set, no tools"},{"benchmark":"livebench","score_raw":70.5,"score_norm":67.78,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2026-01-08 release"}],"coverage":0.5475,"subset_bias":{"offset":-1.49,"peers":9,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":15,"completion_usd_per_m":120,"cache_read_usd_per_m":null,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5-pro","as_of":"2026-10-06","blended_usd_per_m":41.25,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":100,"score_norm":100,"config":"aime 2025, with python tool","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"aime","score_raw":96.7,"score_norm":95.88,"config":"aime 2025, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":70.17,"score_norm":null,"config":"cost_per_task_usd=4.7785, eval=arc-agi-1 semi-private (v1_semi_private), org=openai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":18.33,"score_norm":14.81,"config":"cost_per_task_usd=7.1432, eval=arc-agi-2 semi-private (v2_semi_private), org=openai, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":89.4,"score_norm":90.7,"config":"with python tool","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"gpqa_diamond","score_raw":88.4,"score_norm":89.3,"config":"no tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":42,"score_norm":58.82,"config":"full set, with python and search tools (blocklist)","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":false,"excluded":"tools"},{"benchmark":"hle","score_raw":30.7,"score_norm":42.21,"config":"full set, no tools","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5/","measured_date":"2025-08-07","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":70.5,"score_norm":67.78,"config":"livebench-2026-01-08 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2026-01-08","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-pro","publisher":"OpenRouter (listed 2025-10-06)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.2-pro","display_name":"GPT-5.2 Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.2","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.2-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["image","text","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-12-10","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.5-pro","aliases":[],"intelligence":{"index":77.41,"inputs":[{"benchmark":"aime","score_raw":100,"score_norm":100,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5-2/","config":"2025, no tools, xhigh reasoning effort"},{"benchmark":"arc_agi_2","score_raw":54.16,"score_norm":54.62,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=15.721, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-2-pro-2025-12-11-thinking, org=openai, run: high, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":93.2,"score_norm":96.06,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5-2/","config":"no tools, xhigh reasoning effort"},{"benchmark":"hle","score_raw":36.6,"score_norm":50.88,"trust":1,"source_url":"https://openai.com/index/introducing-gpt-5-2/","config":"no tools, xhigh reasoning effort"}],"coverage":0.4285,"subset_bias":{"offset":-2.16,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":21,"completion_usd_per_m":168,"cache_read_usd_per_m":null,"context_length":400000,"source":"openrouter","source_model_id":"openai/gpt-5.2-pro","as_of":"2026-10-06","blended_usd_per_m":57.75,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aime","score_raw":100,"score_norm":100,"config":"2025, no tools, xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"arc_agi_1","score_raw":90.5,"score_norm":null,"config":"cost_per_task_usd=11.6542, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-2-pro-2025-12-11-thinking, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_1","score_raw":90.5,"score_norm":null,"config":"verified, xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":54.16,"score_norm":54.62,"config":"cost_per_task_usd=15.721, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-2-pro-2025-12-11-thinking, org=openai, run: high, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":54.2,"score_norm":54.67,"config":"high reasoning effort, verified","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"browsecomp","score_raw":77.9,"score_norm":null,"config":"xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":93.2,"score_norm":96.06,"config":"no tools, xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":36.6,"score_norm":50.88,"config":"no tools, xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hmmt","score_raw":100,"score_norm":null,"config":"feb 2025, no tools, xhigh reasoning effort","trust":1,"source_type":"vendor_claim","source_url":"https://openai.com/index/introducing-gpt-5-2/","measured_date":"2025-12-11","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.2-pro","publisher":"OpenRouter (listed 2025-12-10)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-5.4-pro","display_name":"GPT-5.4 Pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5.4","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5.4-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1050000,"release_date":"2026-03-05","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.5-pro","aliases":[],"intelligence":{"index":84.14,"inputs":[{"benchmark":"arc_agi_2","score_raw":83.33,"score_norm":87.03,"trust":3,"source_url":"https://arcprize.org/leaderboard","config":"cost_per_task_usd=16.41, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-pro, org=openai, run: xhigh, system_type=cot"},{"benchmark":"gpqa_diamond","score_raw":94.6,"score_norm":98.03,"trust":2,"source_url":"https://epoch.ai/models/gpt-5-4-pro","config":"90% ci ±1.6%, best score across settings, epoch ai independent eval"},{"benchmark":"hle","score_raw":44.3,"score_norm":62.21,"trust":2,"source_url":"https://epoch.ai/models/gpt-5-4-pro","config":"best score across settings, epoch ai independent eval, no tools"}],"coverage":0.3492,"subset_bias":{"offset":-1.16,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":30,"completion_usd_per_m":180,"cache_read_usd_per_m":null,"context_length":1050000,"source":"openrouter","source_model_id":"openai/gpt-5.4-pro","as_of":"2026-10-06","blended_usd_per_m":67.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"arc_agi_1","score_raw":94.5,"score_norm":null,"config":"cost_per_task_usd=7.21, eval=arc-agi-1 semi-private (v1_semi_private), model_group=gpt-5-4-pro, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arc_agi_2","score_raw":83.33,"score_norm":87.03,"config":"cost_per_task_usd=16.41, eval=arc-agi-2 semi-private (v2_semi_private), model_group=gpt-5-4-pro, org=openai, run: xhigh, system_type=cot","trust":3,"source_type":"leaderboard","source_url":"https://arcprize.org/leaderboard","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arc_agi_2","score_raw":83.3,"score_norm":87,"config":"best score across settings, epoch ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/gpt-5-4-pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":30,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=xhigh, reasoning=true, run: xhigh, source value 0.3 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-4-pro","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":94.6,"score_norm":98.03,"config":"90% ci ±1.6%, best score across settings, epoch ai independent eval","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/gpt-5-4-pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":44.3,"score_norm":62.21,"config":"best score across settings, epoch ai independent eval, no tools","trust":2,"source_type":"aggregator","source_url":"https://epoch.ai/models/gpt-5-4-pro","measured_date":null,"observed_at":"2026-08-18","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5.4-pro","publisher":"OpenRouter (listed 2026-03-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"hermes-3-llama-3.1-70b","display_name":"Hermes 3 70B Instruct","family":{"id":"hermes","vendor":"nousresearch"},"generation":"3","size_class":"70b","params_total_b":70.55,"params_active_b":70.55,"params_exact":70553706496,"params_source":"hf_safetensors","vram_est":{"bf16_gb":141,"int8_gb":74.8,"int4_gb":42.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":"NousResearch/Hermes-3-Llama-3.1-70B","openrouter_id":"nousresearch/hermes-3-llama-3.1-70b","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_large","license":"llama3","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":"2024-08-18","release_status":"released","knowledge_cutoff":"2023-12-31","status":"superseded","supersedes":[],"superseded_by":"hermes-4-70b","aliases":[],"intelligence":{"index":13.22,"inputs":[{"benchmark":"gpqa_diamond","score_raw":40.1,"score_norm":21.27,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.401010101010101 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.99,"score_norm":2.93,"trust":2,"source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0399 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":47.24,"score_norm":12.07,"trust":2,"source_url":"https://arxiv.org/pdf/2408.11857","config":"5-shot"}],"coverage":0.3889,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.7,"completion_usd_per_m":0.7,"cache_read_usd_per_m":null,"context_length":131072,"source":"openrouter","source_model_id":"nousresearch/hermes-3-llama-3.1-70b","as_of":"2026-10-06","blended_usd_per_m":0.7,"endpoint_expires_at":null},"memory":[{"quant":"fp16","weights_gb":141.1,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.839,"method":"derived_estimate","source_url":null},{"quant":"fp16","weights_gb":141.107,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":150.847,"method":"vendor","source_url":"https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B/tree/main"},{"quant":"fp8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null},{"quant":"int4","weights_gb":42.33,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":47.131,"method":"derived_estimate","source_url":null},{"quant":"int8","weights_gb":74.783,"kv_cache_gb_per_8k":2.6844,"min_serve_gb":81.207,"method":"derived_estimate","source_url":null}],"hardware_perf":[{"hardware":"nvidia_dgx_spark","metric":"single_stream_tok_s","value":2.32,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":273,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":3.87,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"batched_tok_s","value":5204.34,"quant":"fp8","batch_size":32,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","anchor":{"model_id":"deepseek-v4-flash-reasoning","value":1262,"params_active_b":290.94,"quant":"fp8","batch_size":32,"context_tokens":8192,"trust":1,"source_url":"https://lambda.ai/inference-models/deepseek-ai/deepseek-v4-flash"},"scaling":"active_param_ratio","quant_scale":1,"quant":"fp8","params_active_b":70.55,"provisional_hardware":false}},{"hardware":"nvidia_h100_sxm_80g","metric":"single_stream_tok_s","value":28.49,"quant":"fp8","batch_size":1,"context_tokens":8192,"method":"derived_estimate","trust":0,"source_url":null,"derivation":{"model":"bandwidth_saturation_v1","inputs":{"memory_bw_gbps":3350,"params_active_b":70.55,"bytes_per_param":1,"quant":"fp8"},"bw_ceiling_tok_s":47.48,"calibration_factor":0.6,"calibration_anchors":0,"calibration_fallback":true,"provisional_hardware":false}}],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-3-llama-3.1-70b","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.78,"score_norm":4.78,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"agieval","score_raw":56.18,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"arc_challenge","score_raw":65.53,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"bbh","score_raw":67.82,"score_norm":null,"config":"3-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":40.1,"score_norm":21.27,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.401010101010101 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hellaswag","score_raw":88.19,"score_norm":null,"config":"10-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"hle","score_raw":3.99,"score_norm":2.93,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0399 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifeval","score_raw":81.21,"score_norm":null,"config":"strict","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":18.84,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.188359788359788 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math_lvl_5","score_raw":20.8,"score_norm":null,"config":"4-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu","score_raw":79.09,"score_norm":null,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":47.24,"score_norm":12.07,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"musr","score_raw":50.67,"score_norm":null,"config":"0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":23.15,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.231481481481481 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/hermes-3-llama-3-1-70b","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"truthfulqa","score_raw":63.29,"score_norm":null,"config":"mc2 0-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"winogrande","score_raw":83.19,"score_norm":null,"config":"5-shot","trust":2,"source_type":"paper","source_url":"https://arxiv.org/pdf/2408.11857","measured_date":"2024-08-15","observed_at":"2026-09-05","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#nousresearch/hermes-3-llama-3.1-70b","publisher":"OpenRouter (listed 2024-08-18)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"ling-2.6-flash","display_name":"ling-2.6-flash","family":{"id":"ling","vendor":"inclusionAI"},"generation":"2.6","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"inclusionai/ling-2.6-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":262144,"release_date":"2026-04-21","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"ling-3.0-flash","aliases":[],"intelligence":{"index":37.36,"inputs":[{"benchmark":"gpqa_diamond","score_raw":59.29,"score_norm":48.3,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.592929292929293 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.26,"score_norm":6.26,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0625579240037071 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":85.96,"score_norm":87.95,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.859649122807018 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":24.34,"score_norm":22.75,"trust":2,"source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.243445692883895 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest"}],"coverage":0.3651,"subset_bias":{"offset":-7.33,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":2.3,"score_norm":2.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-2.6-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_agentic_index","score_raw":2.25,"score_norm":2.25,"config":"aa field agenticindex, artificial analysis agentic index (intelligence index v4.1.1), effort=none, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":25.3,"score_norm":25.3,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-2.6-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.2,"score_norm":14.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#inclusionai/ling-2.6-flash","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":14.22,"score_norm":14.22,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=false, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":28,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.28 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gdpval_aa_normalized","score_raw":2.22,"score_norm":null,"config":"aa field gdpvalnormalized, aa-normalised 0-1 form, artificial analysis intelligence index v4.1.1 suite, effort=none, gdpval-aa, reasoning=false, source value 0.022225000000000023 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":59.29,"score_norm":48.3,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.592929292929293 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.26,"score_norm":6.26,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0625579240037071 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":57.41,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.574149659863946 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":27.08,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.270833333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":85.96,"score_norm":87.95,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.859649122807018 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"tau_bench_banking","score_raw":2.89,"score_norm":null,"config":"aa field taubanking, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.0288659793814433 is a 0-1 fraction, tau3-banking v1.0.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"terminal_bench","score_raw":21.21,"score_norm":19.07,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.212121212121212 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":24.34,"score_norm":22.75,"config":"aa field terminalbenchv21, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.243445692883895 is a 0-1 fraction, terminal-bench 2.1, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/ling-2-6-flash","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#inclusionai/ling-2.6-flash","publisher":"OpenRouter (listed 2026-04-21)"}],"last_verified":"2026-08-24","last_enriched_at":null},{"id":"deepseek-3-instruct","display_name":"deepseek-chat-v3","family":{"id":"deepseek","vendor":"DeepSeek"},"generation":"3","size_class":"frontier","params_total_b":684.53,"params_active_b":37,"params_exact":684531386000,"params_source":"hf_safetensors","vram_est":{"bf16_gb":1369,"int8_gb":726,"int4_gb":411,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":null,"is_open_weights":true,"self_hostable":true,"context_window_tok":163840,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"deepseek-v3","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#deepseek/deepseek-chat","publisher":"OpenRouter (listed 2024-12-26)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-8b-04-28","display_name":"qwen3-8b-04-28","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"8b","params_total_b":8.19,"params_active_b":8.19,"params_exact":8190735360,"params_source":"hf_safetensors","vram_est":{"bf16_gb":16.4,"int8_gb":8.7,"int4_gb":4.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-8b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-8b","publisher":"OpenRouter (listed 2025-04-28)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-32b-04-28","display_name":"qwen3-32b-04-28","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"32b","params_total_b":32.76,"params_active_b":32.76,"params_exact":32762123264,"params_source":"hf_safetensors","vram_est":{"bf16_gb":65.5,"int8_gb":34.7,"int4_gb":19.7,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-32b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-32b","publisher":"OpenRouter (listed 2025-04-28)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-30b-a3b-04-28","display_name":"qwen3-30b-a3b-04-28","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"30b-a3b","params_total_b":30.53,"params_active_b":3,"params_exact":30532122624,"params_source":"hf_safetensors","vram_est":{"bf16_gb":61.1,"int8_gb":32.4,"int4_gb":18.3,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-30b-a3b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-30b-a3b","publisher":"OpenRouter (listed 2025-04-28)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-235b-a22b-2507-instruct","display_name":"Qwen3-235B-A22B-Instruct-2507","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.09,"params_active_b":22,"params_exact":235093634560,"params_source":"hf_safetensors","vram_est":{"bf16_gb":470,"int8_gb":249,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-235b-a22b-07-25","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-235b-a22b-04-28","display_name":"qwen3-235b-a22b-04-28","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"235b-a22b","params_total_b":235.09,"params_active_b":22,"params_exact":235093634560,"params_source":"hf_safetensors","vram_est":{"bf16_gb":470,"int8_gb":249,"int4_gb":141,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_frontier","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-235b-a22b","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-235b-a22b","publisher":"OpenRouter (listed 2025-04-28)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"qwen-3-14b-04-28","display_name":"qwen3-14b-04-28","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3","size_class":"14b","params_total_b":14.77,"params_active_b":14.77,"params_exact":14768307200,"params_source":"hf_safetensors","vram_est":{"bf16_gb":29.5,"int8_gb":15.7,"int4_gb":8.9,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"open_mid","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":131072,"release_date":null,"release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"qwen-3-14b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3-14b","publisher":"OpenRouter (listed 2025-04-28)"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"gemini-2.5-flash-lite","display_name":"Gemini 2.5 Flash Lite","family":{"id":"gemini","vendor":"Google"},"generation":"2.5","size_class":"frontier-flash-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"google/gemini-2.5-flash-lite","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file","audio","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2025-07-22","release_status":"released","knowledge_cutoff":"2025-01-31","status":"superseded","supersedes":[],"superseded_by":"gemini-3.5-flash-lite","aliases":[],"intelligence":{"index":28.08,"inputs":[{"benchmark":"aime","score_raw":53.33,"score_norm":41.66,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.533333333333333 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":62.53,"score_norm":52.86,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.625252525252525 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":6.81,"score_norm":7.07,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0681186283595922 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":4.55,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0454545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":-4.18,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.4,"cache_read_usd_per_m":0.01,"context_length":1048576,"source":"openrouter","source_model_id":"google/gemini-2.5-flash-lite","as_of":"2026-10-06","blended_usd_per_m":0.17500000000000002,"endpoint_expires_at":"2026-10-20"},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":10.4,"score_norm":10.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-lite","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":11.35,"score_norm":11.35,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: reasoning","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":56.33,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=true, run: reasoning, source value 0.563333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":53.33,"score_norm":41.66,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.533333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: non-reasoning, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":62.53,"score_norm":52.86,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=true, run: reasoning, source value 0.625252525252525 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":6.81,"score_norm":7.07,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=true, run: reasoning, source value 0.0681186283595922 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":49.86,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=true, run: reasoning, source value 0.498639455782313 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":59.26,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=true, run: reasoning, source value 0.592592592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mlcr_overall","score_raw":2.22,"score_norm":null,"config":"aa field mlcroverall, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0222222222222222 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":58.21,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=true, run: reasoning, source value 0.582080924855491 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":19.33,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, scicode, source value 0.193287037037037 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":19.01,"score_norm":0,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: non-reasoning, source value 0.190058479532164 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":4.55,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=true, run: reasoning, source value 0.0454545454545455 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gemini-2-5-flash-lite-reasoning","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#google/gemini-2.5-flash-lite","publisher":"OpenRouter (listed 2025-07-22)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"mistral-1-large","display_name":"Mistral Large","family":{"id":"mistral","vendor":"Mistral AI"},"generation":"1","size_class":"frontier-large","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"mistralai/mistral-large","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","file"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":128000,"release_date":"2024-02-26","release_status":"released","knowledge_cutoff":"2024-11-30","status":"superseded","supersedes":[],"superseded_by":"mistral-2512-large","aliases":["mistral-large"],"intelligence":{"index":10.2,"inputs":[{"benchmark":"arena_elo","score_raw":1242,"score_norm":0,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"no style control, overall (text)"},{"benchmark":"gpqa_diamond","score_raw":35.05,"score_norm":14.15,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: feb '24, source value 0.350505050505051 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":3.5,"score_norm":2.21,"trust":2,"source_url":"https://artificialanalysis.ai/models/mistral-large","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: feb '24, source value 0.035 is a 0-1 fraction, x100 at harvest"},{"benchmark":"mmlu_pro","score_raw":51.5,"score_norm":19.17,"trust":2,"source_url":"http://web.archive.org/web/20250618051413/https://artificialanalysis.ai/models/mistral-large","config":"artificial analysis independent run, read from the 2025-06-18 wayback capture of aa's now-deprecated mistral large (feb '24) page"}],"coverage":0.4682,"subset_bias":null},"pricing":{"prompt_usd_per_m":2,"completion_usd_per_m":6,"cache_read_usd_per_m":0.2,"context_length":128000,"source":"openrouter","source_model_id":"mistralai/mistral-large","as_of":"2026-10-06","blended_usd_per_m":3,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":5.8,"score_norm":5.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":4.07,"score_norm":4.07,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false, run: feb '24","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1242,"score_norm":0,"config":"no style control, overall (text)","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":35.05,"score_norm":14.15,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, run: feb '24, source value 0.350505050505051 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":3.5,"score_norm":2.21,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: feb '24, source value 0.035 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"livecodebench","score_raw":17.78,"score_norm":null,"config":"artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":17.78,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, run: feb '24, source value 0.177777777777778 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"math_500","score_raw":52.7,"score_norm":null,"config":"2025-06-18 wayback capture of aa's deprecated mistral large (feb '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20250618051413/https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"mmlu_pro","score_raw":51.5,"score_norm":19.17,"config":"artificial analysis independent run, read from the 2025-06-18 wayback capture of aa's now-deprecated mistral large (feb '24) page","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20250618051413/https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"scicode","score_raw":20.8,"score_norm":null,"config":"2025-06-18 wayback capture of aa's deprecated mistral large (feb '24) page, artificial analysis independent run","trust":2,"source_type":"aggregator","source_url":"http://web.archive.org/web/20250618051413/https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":20.83,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, run: feb '24, scicode, source value 0.208333333333333 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/mistral-large","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#mistralai/mistral-large","publisher":"OpenRouter (listed 2024-02-26)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"nova-1-lite","display_name":"Nova Lite 1.0","family":{"id":"nova","vendor":"Amazon"},"generation":"1","size_class":"frontier-lite","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"amazon/nova-lite-v1","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":300000,"release_date":"2024-12-05","release_status":"released","knowledge_cutoff":"2024-10-31","status":"superseded","supersedes":[],"superseded_by":"nova-2-lite-v1","aliases":["nova-lite-v1"],"intelligence":{"index":1.21,"inputs":[{"benchmark":"arena_elo","score_raw":1260,"score_norm":0,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"19372 votes, arena entry amazon-nova-lite-v1.0, no style control, rank 302, text/overall"},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"trust":2,"source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","config":"artificial analysis independent eval"},{"benchmark":"livebench","score_raw":39.1,"score_norm":0,"trust":3,"source_url":"https://livebench.ai/#/","config":"livebench-2025-04-25 release"}],"coverage":0.3094,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.06,"completion_usd_per_m":0.24,"cache_read_usd_per_m":null,"context_length":300000,"source":"openrouter","source_model_id":"amazon/nova-lite-v1","as_of":"2026-10-06","blended_usd_per_m":0.105,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":6.7,"score_norm":6.7,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#amazon/nova-lite-v1","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"arena_elo","score_raw":1260,"score_norm":0,"config":"19372 votes, arena entry amazon-nova-lite-v1.0, no style control, rank 302, text/overall","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-09-02","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.3,"score_norm":3.38,"config":"artificial analysis independent eval","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/evaluations/humanitys-last-exam","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"livebench","score_raw":39.1,"score_norm":0,"config":"livebench-2025-04-25 release","trust":3,"source_type":"leaderboard","source_url":"https://livebench.ai/#/","measured_date":"2025-04-25","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#amazon/nova-lite-v1","publisher":"OpenRouter (listed 2024-12-05)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"gpt-o1-pro","display_name":"OpenAI o1-pro","family":{"id":"gpt","vendor":"OpenAI"},"generation":"1","size_class":"frontier-o-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/o1-pro","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","file"],"segment":"cloud_frontier","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2025-03-19","release_status":"released","knowledge_cutoff":"2023-10-31","status":"superseded","supersedes":[],"superseded_by":"gpt-o3-pro","aliases":["o1-pro"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":150,"completion_usd_per_m":600,"cache_read_usd_per_m":null,"context_length":200000,"source":"openrouter","source_model_id":"openai/o1-pro","as_of":"2026-10-06","blended_usd_per_m":262.5,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":12.4,"score_norm":12.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/o1-pro","measured_date":null,"observed_at":"2026-10-06","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":19.12,"score_norm":19.12,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/o1-pro","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aime","score_raw":86,"score_norm":82.5,"config":"aime 2024, o1 pro mode, openai 'introducing chatgpt pro' chart series 'o1 pro mode' (2024-12-05 internet archive snapshot of openai.com/index/introducing-chatgpt-pro/), pass@1 accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/20241206000000/https://openai.com/index/introducing-chatgpt-pro/","measured_date":"2024-12-05","observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":79,"score_norm":76.06,"config":"o1 pro mode, openai 'introducing chatgpt pro' chart series 'o1 pro mode' (read from the 2024-12-05 internet archive snapshot of openai.com/index/introducing-chatgpt-pro/ because openai.com blocks automated fetches), pass@1 accuracy","trust":1,"source_type":"vendor_claim","source_url":"https://web.archive.org/web/20241206000000/https://openai.com/index/introducing-chatgpt-pro/","measured_date":"2024-12-05","observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/o1-pro","publisher":"OpenRouter (listed 2025-03-19)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"minimax-m2-her","display_name":"MiniMax M2-her","family":{"id":"minimax","vendor":"MiniMax"},"generation":"2","size_class":"frontier-m","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"minimax/minimax-m2-her","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":65536,"release_date":"2026-01-23","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"minimax-m2.7","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.3,"completion_usd_per_m":1.2,"cache_read_usd_per_m":0.03,"context_length":65536,"source":"openrouter","source_model_id":"minimax/minimax-m2-her","as_of":"2026-10-06","blended_usd_per_m":0.5249999999999999,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#minimax/minimax-m2-her","publisher":"OpenRouter (listed 2026-01-23)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"solar-pro-3","display_name":"Solar Pro 3","family":{"id":"solar","vendor":"upstage"},"generation":"3","size_class":"frontier-pro","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"upstage/solar-pro-3","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":131072,"release_date":"2026-01-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"solar-pro-4","aliases":[],"intelligence":{"index":45.36,"inputs":[{"benchmark":"gpqa_diamond","score_raw":72.42,"score_norm":66.79,"trust":2,"source_url":"https://artificialanalysis.ai/models/solar-pro-3","config":"artificial analysis independent evaluation"},{"benchmark":"hle","score_raw":10.29,"score_norm":12.19,"trust":2,"source_url":"https://artificialanalysis.ai/models/solar-pro-3","config":"artificial analysis independent evaluation"},{"benchmark":"tau_bench","score_raw":86.26,"score_norm":88.35,"trust":2,"source_url":"https://artificialanalysis.ai/models/solar-pro-3","config":"artificial analysis independent evaluation, tau2/telecom"},{"benchmark":"terminal_bench","score_raw":11.99,"score_norm":8.22,"trust":2,"source_url":"https://artificialanalysis.ai/models/solar-pro-3","config":"2.1, artificial analysis independent evaluation"}],"coverage":0.3651,"subset_bias":{"offset":-8.16,"peers":10,"method":"subset-knn-v1"}},"pricing":{"prompt_usd_per_m":0.15,"completion_usd_per_m":0.6,"cache_read_usd_per_m":0.015,"context_length":131072,"source":"openrouter","source_model_id":"upstage/solar-pro-3","as_of":"2026-10-06","blended_usd_per_m":0.26249999999999996,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_agentic_index","score_raw":1.4,"score_norm":1.4,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_coding_index","score_raw":16.2,"score_norm":16.2,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"aa_intelligence_index","score_raw":7.8,"score_norm":7.8,"config":null,"trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#upstage/solar-pro-3","measured_date":null,"observed_at":"2026-09-08","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":32.33,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":72.42,"score_norm":66.79,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":10.29,"score_norm":12.19,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":71.2,"score_norm":null,"config":"artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":86.26,"score_norm":88.35,"config":"artificial analysis independent evaluation, tau2/telecom","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":11.99,"score_norm":8.22,"config":"2.1, artificial analysis independent evaluation","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/solar-pro-3","measured_date":null,"observed_at":"2026-09-05","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#upstage/solar-pro-3","publisher":"OpenRouter (listed 2026-01-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.6-flash","display_name":"Qwen3.6 Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.6","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.6-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-04-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.8-flash","aliases":["qwen3.6-flash"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1875,"completion_usd_per_m":1.125,"cache_read_usd_per_m":null,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.6-flash","as_of":"2026-10-06","blended_usd_per_m":0.421875,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.6-flash","publisher":"OpenRouter (listed 2026-04-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"qwen-3.7-flash","display_name":"Qwen3.7 Flash","family":{"id":"qwen","vendor":"Alibaba"},"generation":"3.7","size_class":"frontier-flash","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"qwen/qwen3.7-flash","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1000000,"release_date":"2026-07-27","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"qwen-3.8-flash","aliases":["qwen3.7-flash"],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.03,"completion_usd_per_m":0.13,"cache_read_usd_per_m":0.006,"context_length":1000000,"source":"openrouter","source_model_id":"qwen/qwen3.7-flash","as_of":"2026-10-06","blended_usd_per_m":0.055,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#qwen/qwen3.7-flash","publisher":"OpenRouter (listed 2026-07-27)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"muse-spark-1.2-contributor","display_name":"Muse Spark 1.2 Contributor","family":{"id":"muse","vendor":"Meta"},"generation":"1.2","size_class":"frontier-spark","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"meta/muse-spark-1.2-contributor","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image","video","file","audio"],"segment":"cloud_economy","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":1048576,"release_date":"2026-08-21","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"muse-spark-1.3","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":{"prompt_usd_per_m":0.1,"completion_usd_per_m":0.2,"cache_read_usd_per_m":0.002,"context_length":1048576,"source":"openrouter","source_model_id":"meta/muse-spark-1.2-contributor","as_of":"2026-10-06","blended_usd_per_m":0.125,"endpoint_expires_at":null},"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#meta/muse-spark-1.2-contributor","publisher":"OpenRouter (listed 2026-08-21)"}],"last_verified":"2026-10-06","last_enriched_at":null},{"id":"claude-haiku-3","display_name":"claude-3-haiku","family":{"id":"claude","vendor":"Anthropic"},"generation":"3","size_class":"frontier-haiku","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"anthropic/claude-3-haiku","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":200000,"release_date":"2024-03-13","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"claude-haiku-4.5","aliases":["claude-3-haiku"],"intelligence":{"index":7.14,"inputs":[{"benchmark":"arena_elo","score_raw":1261.4,"score_norm":0,"trust":3,"source_url":"https://arena.ai/leaderboard/text","config":"117701 votes, arena entry claude-3-haiku-20240307, leaderboard text-overall-style_control, rank 295, style control, text/overall, vote cutoff 2026-08-21"},{"benchmark":"gpqa_diamond","score_raw":37.37,"score_norm":17.42,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-3-haiku","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.373737373737374 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":4.08,"score_norm":3.06,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-3-haiku","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0407784986098239 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-3-haiku","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.210526315789474 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"trust":2,"source_url":"https://artificialanalysis.ai/models/claude-3-haiku","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":3.5,"score_norm":3.5,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=none, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=false","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":25,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=none, reasoning=false, source value 0.25 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"arena_elo","score_raw":1261.4,"score_norm":0,"config":"117701 votes, arena entry claude-3-haiku-20240307, leaderboard text-overall-style_control, rank 295, style control, text/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"arena_elo_vision","score_raw":1000.5,"score_norm":null,"config":"13380 votes, arena entry claude-3-haiku-20240307, leaderboard vision-overall-style_control, rank 136, style control, vision/overall, vote cutoff 2026-08-21","trust":3,"source_type":"leaderboard","source_url":"https://arena.ai/leaderboard/text","measured_date":"2026-08-21","observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"critpt","score_raw":0,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=none, reasoning=false, run: claude 3.5 haiku, source value 0 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-5-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":37.37,"score_norm":17.42,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=none, gpqa diamond, reasoning=false, source value 0.373737373737374 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.08,"score_norm":3.06,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, source value 0.0407784986098239 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":4.08,"score_norm":3.06,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=none, humanity's last exam, reasoning=false, run: claude 3 haiku, source value 0.0407784986098239 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":36.12,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=none, ifbench, reasoning=false, source value 0.361224489795918 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":15.45,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=none, livecodebench, reasoning=false, source value 0.154497354497354 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":30.81,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=none, mmmu-pro, reasoning=false, source value 0.308092485549133 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":18.63,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, scicode, source value 0.186342592592593 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":21.05,"score_norm":1.4,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.210526315789474 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":0.76,"score_norm":0,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=none, reasoning=false, source value 0.00757575757575758 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/claude-3-haiku","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#anthropic/claude-3-haiku","publisher":"OpenRouter (listed 2024-03-13)"}],"last_verified":"2026-09-25","last_enriched_at":null},{"id":"gpt-5-codex","display_name":"gpt-5-codex","family":{"id":"gpt","vendor":"OpenAI"},"generation":"5","size_class":"frontier-codex","params_total_b":null,"params_active_b":null,"params_exact":null,"params_source":null,"vram_est":null,"vram_frontier":null,"hugging_face_id":null,"openrouter_id":"openai/gpt-5-codex:batch","is_moe":false,"variant_axis":"general","kind":"model","modalities":["text","image"],"segment":"cloud_balanced","license":null,"is_open_weights":false,"self_hostable":false,"context_window_tok":400000,"release_date":"2025-09-15","release_status":"released","knowledge_cutoff":null,"status":"superseded","supersedes":[],"superseded_by":"gpt-5.3-codex","aliases":[],"intelligence":{"index":70.31,"inputs":[{"benchmark":"aime","score_raw":98.67,"score_norm":98.34,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-codex","config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.986666666666667 is a 0-1 fraction, x100 at harvest"},{"benchmark":"gpqa_diamond","score_raw":83.74,"score_norm":82.73,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-codex","config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.837373737373737 is a 0-1 fraction, x100 at harvest"},{"benchmark":"hle","score_raw":27.85,"score_norm":38.01,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-codex","config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.278498609823911 is a 0-1 fraction, x100 at harvest"},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-codex","config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest"},{"benchmark":"terminal_bench","score_raw":37.88,"score_norm":38.68,"trust":2,"source_url":"https://artificialanalysis.ai/models/gpt-5-codex","config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest"}],"coverage":0.4445,"subset_bias":{"offset":2.09,"peers":10,"method":"subset-knn-v1"}},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[{"benchmark":"aa_intelligence_index","score_raw":37.04,"score_norm":37.04,"config":"aa field intelligenceindex, artificial analysis intelligence index v4.1.1, effort=high, estimated=true, index unit 0-100 emitted as-is (no rescale), reasoning=true, run: high","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"aa_lcr","score_raw":71,"score_norm":null,"config":"aa field lcr, artificial analysis intelligence index v4.1.1 suite, artificial-analysis-long-context-reasoning, effort=high, reasoning=true, run: high, source value 0.71 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"aime","score_raw":98.67,"score_norm":98.34,"config":"aa field aime25, aime-2025, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.986666666666667 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"critpt","score_raw":5.14,"score_norm":null,"config":"aa field critpt, artificial analysis intelligence index v4.1.1 suite, critpt, effort=high, reasoning=true, run: high, source value 0.0514285714285714 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"design_arena_webapps","score_raw":1109,"score_norm":null,"config":"arena=agents","trust":2,"source_type":"aggregator","source_url":"https://openrouter.ai/api/v1/models#openai/gpt-5-codex:batch","measured_date":"2026-08-18","observed_at":"2026-08-18","admissible":false,"excluded":null},{"benchmark":"gpqa_diamond","score_raw":83.74,"score_norm":82.73,"config":"aa field gpqa, artificial analysis intelligence index v4.1.1 suite, effort=high, gpqa diamond, reasoning=true, run: high, source value 0.837373737373737 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"hle","score_raw":27.85,"score_norm":38.01,"config":"aa field hle, artificial analysis intelligence index v4.1.1 suite, effort=high, humanity's last exam, reasoning=true, run: high, source value 0.278498609823911 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"ifbench","score_raw":74.15,"score_norm":null,"config":"aa field ifbench, artificial analysis intelligence index v4.1.1 suite, effort=high, ifbench, reasoning=true, run: high, source value 0.741496598639456 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"livecodebench","score_raw":84.02,"score_norm":null,"config":"aa field livecodebench, artificial analysis intelligence index v4.1.1 suite, effort=high, livecodebench, reasoning=true, run: high, source value 0.84021164021164 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"mmmu_pro","score_raw":73.82,"score_norm":null,"config":"aa field mmmupro, artificial analysis intelligence index v4.1.1 suite, effort=high, mmmu-pro, reasoning=true, run: high, source value 0.738150289017341 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"scicode","score_raw":40.86,"score_norm":null,"config":"aa field scicode, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, scicode, source value 0.408564814814815 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":false,"excluded":null},{"benchmark":"tau_bench","score_raw":86.84,"score_norm":89.12,"config":"aa field tau2, aa field tau2 (not mapped to tau_bench), artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.868421052631579 is a 0-1 fraction, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null},{"benchmark":"terminal_bench","score_raw":37.88,"score_norm":38.68,"config":"aa field terminalbenchhard, artificial analysis intelligence index v4.1.1 suite, effort=high, reasoning=true, run: high, source value 0.378787878787879 is a 0-1 fraction, terminal-bench hard, x100 at harvest","trust":2,"source_type":"aggregator","source_url":"https://artificialanalysis.ai/models/gpt-5-codex","measured_date":null,"observed_at":"2026-08-23","admissible":true,"excluded":null}],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://openrouter.ai/api/v1/models#openai/gpt-5-codex:batch","publisher":"OpenRouter (listed 2025-09-23)"}],"last_verified":"2026-09-21","last_enriched_at":null},{"id":"kimi-k2-1t-instruct","display_name":"Kimi-K2-Instruct","family":{"id":"kimi","vendor":"Moonshot AI"},"generation":"2","size_class":"1t","params_total_b":1026.41,"params_active_b":32,"params_exact":1026408235864,"params_source":"hf_safetensors","vram_est":{"bf16_gb":2053,"int8_gb":1088,"int4_gb":616,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_frontier","license":"other","is_open_weights":true,"self_hostable":true,"context_window_tok":null,"release_date":"2025-07","release_status":"released","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"kimi-k2","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://huggingface.co/moonshotai/Kimi-K2-Instruct","publisher":"Hugging Face"}],"last_verified":"2026-08-18","last_enriched_at":null},{"id":"mixtral-8-x22b-v0.1-8x22b-instruct","display_name":"Mixtral-8x22B-Instruct-v0.1","family":{"id":"mixtral","vendor":"Mistral AI"},"generation":"8","size_class":"8x22b","params_total_b":140.63,"params_active_b":39,"params_exact":140630071296,"params_source":"hf_safetensors","vram_est":{"bf16_gb":281,"int8_gb":149,"int4_gb":84.4,"method":"linear-rom-v1","approximate":true,"basis":"params_total_b"},"vram_frontier":null,"hugging_face_id":null,"openrouter_id":null,"is_moe":true,"variant_axis":"instruct","kind":"model","modalities":["text"],"segment":"open_large","license":"apache-2.0","is_open_weights":true,"self_hostable":true,"context_window_tok":65536,"release_date":"2024-04","release_status":"deprecated","knowledge_cutoff":null,"status":"retired","supersedes":[],"superseded_by":"mixtral-8-x22b-instruct","aliases":[],"intelligence":{"index":null,"inputs":[],"coverage":null,"subset_bias":null},"pricing":null,"memory":[],"hardware_perf":[],"benchmarks":[],"summary":null,"strengths":null,"weaknesses":null,"best_use_cases":null,"sources":[{"url":"https://docs.mistral.ai/models/mixtral-8x22b-0-1-0-3","publisher":"Mistral AI"}],"last_verified":"2026-08-18","last_enriched_at":null}]}}