Gemini 4 Carbon's Reported Lead Raises Questions for Security Teams

Google's Gemini 4 Carbon posted an 88.2% score on SEC-bench Pro for vulnerability finding, according to a roundup from Latent Space ¹. The same roundup says access starts with trusted cyber defenders through the Fairwind Program, while wider developer access is promised “as soon as possible” ¹. For security teams, that's a model worth testing—not grounds for a production commitment. For the broader launch picture, see Gemini 4 Carbon Is an Announcement, Not Yet a Switching Decision.

Google DeepMind announced Carbon two weeks after Argon, positioning it for cybersecurity and long-horizon tasks ¹. The reported cyber scores are one input among several. Teams still need to understand what the evaluations measured, how Carbon compares with alternatives under matched conditions, and whether they can access it.

What Gemini 4 Carbon's cybersecurity benchmarks actually cover

The roundup reports three separate scores: 88.2% on SEC-bench Pro for vulnerability finding, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching ¹. These aren't interchangeable measures of “cybersecurity.” Finding a flaw, reproducing an exploit, and writing a patch are different jobs with different risk profiles—a model that spots a vulnerability can still produce a patch that breaks something else.

That distinction should drive evaluation design. Analysis: treat these as three separate use cases, each with its own acceptance criteria and review process, rather than averaging them into one “cyber score.”

The source summary does not provide the evaluations' protocols, datasets, sample sizes, scoring rules, or head-to-head results for named competitors under the same conditions ¹. Without those details, a percentage offers limited information about tool access, human review, pass/fail criteria, or reproducibility.

Before letting these numbers drive a selection decision, ask the provider or benchmark owners:

• What exactly counts as a successful finding, exploit, or patch?
• What tools and reference material could the model use?
• How were test cases chosen, and what guarded against contamination?
• Was there human review or correction of outputs?
• Can the test be rerun, and are results published for competitors under identical conditions?

These are diligence questions, not claims about how the original tests were run. Analysis: a strong score justifies requesting your own evaluation; it does not establish how the model will perform on your repositories, alerts, detection content, or incident workflow.

A frontier model evaluation: leader, or leader on selected measures?

Carbon's reported performance goes beyond the cyber suite. Artificial Analysis ranked it first on its Intelligence Index v5 with a score of 59, leading on four of the index's nine component evaluations: HLE-Diamond, TB-Science, AA-Omniscience, and AutomationBench ¹. That's a signal, but other comparisons in the roundup are less favorable.

On the Coding Agent Index, Carbon scored 59 using Antigravity CLI, against 62 for Claude Code with Opus 5.5 [max] ¹. On FrontierCode v1.1, Carbon scored 51% against Opus's 56% ¹. On FrontierSWE, Carbon's 61.2% trailed GPT-6 Astra (65.5%), Opus 5.5 (62.3%), and Sonnet 5.5 (61.9%) ¹. Separately, the roundup cites a state-of-the-art 9.6% result on SWE-sweep ¹.

Rankings shift by arena elsewhere: Carbon ranked first in Text Arena at 1,528, second in Agent Arena at a median cost of $1.80 per task, and fourth in Code Arena: WebDev at 1,768 ¹. On AA-Briefcase, Carbon scored 1,609 Elo—126 points above Argon—but behind Grok 4.7, Muse Wonder, and Kimi K3.1 Pro, which scored 1,678 ¹.

Analysis: “lead” here is task-, harness-, and metric-dependent. Carbon tops some boards and trails competitors on others in the same roundup. Match the comparison to your workload and test harness rather than extrapolating from a single index ranking.

Reasoning, domain work, and reliability

Carbon ranked second on the Vals Index and first on the Finance Index and Legal Research Benchmark, according to the roundup ¹. It also generated 64% more output tokens than Argon, concentrated in finance, tax, and legal tasks ¹.

Longer output can mean more thorough reasoning—or more text to review and a larger bill. Analysis: for long-form security work, measure whether extra output changes task completion rates; don't assume length equals quality. Track reviewer effort, latency, and cost per accepted result instead.

The roundup reports an 11% hallucination rate for Carbon versus 4% for Argon, while describing Carbon's rate as the second-lowest measured; Opus 5.5 and Astra are each reported at 50% ¹. The summary does not define what counted as a hallucination or establish that the rates are comparable across tasks ¹.

Analysis: the reported difference from Argon puts the measurement method front and center. Ask what counted as a hallucination, which tasks were tested, and how unsupported claims were scored. For security use, the errors that matter include missed vulnerabilities, misjudged severity, unsupported incident conclusions, and patches that introduce new defects—not just an aggregate hallucination percentage.

Gemini model access: what security teams can use now

Carbon first goes to “trusted cyber defenders” through the Fairwind Program, with wider developer access promised “as soon as possible” ¹. The source summary does not define who qualifies as a trusted defender, who decides, or how to apply ¹.

That makes access a procurement question as well as a technical one. The source summary provides no details on deployment options, tenancy and logging, supported environments, or support terms ¹. Any of these could determine whether the model fits a workflow, regardless of how the benchmarks read.

Before scoping a pilot, seek answers on eligibility and the application process, supported regions and environments, data-handling terms, model and tool restrictions, rate limits, pricing, incident procedures, and a fallback path if access changes. These are open questions, not established facts about Carbon.

Analysis: don't design a production dependency around an access promise with no date attached. If a pilot is viable now, keep it narrow and reversible—use representative tasks, limit the integration surface, and confirm access terms before code depends on them.

Cost and deployment trade-offs

The roundup carries two price quotes. Phil Schmid cites $2 per million input tokens and $10 per million output tokens; Artificial Analysis lists standard pricing of $4/$20 per million tokens with a 50% launch discount of unspecified duration, and cached input at $0.20 per million tokens ¹. The summary does not clarify whether the $2/$10 figures reflect the discounted price or how long any discount lasts ¹.

The output-token difference matters too: Carbon produced 64% more output tokens than Argon on the finance, tax, and legal tasks measured ¹. Analysis: per-token pricing alone won't tell you workflow cost when models differ in verbosity. Compare cost per accepted vulnerability finding, verified exploit, correct patch, or closed investigation—not cost per API call.

The roundup lists a 1-million-token context window and a matching maximum output, with text, image, video, and speech input and text-only output ¹. Multimodal input could matter for some investigations, but the listed modalities do not establish performance on any particular security task. Test that directly, and confirm which capabilities are available in the access tier your organization gets.

How to evaluate Gemini 4 Carbon for cyber defense

Build a small evaluation set from defensive work your team already understands—vulnerability triage, exploitability analysis, patch generation, code review, and incident-evidence summarization. These are proposed test categories, not tasks the roundup says Carbon was scored on.

Run Carbon against alternatives under matched prompts, tools, context, review rules, and budgets. The roundup's mixed leaderboard results and lack of protocol details make those controls useful for interpreting your own test ¹. Matched comparisons are simpler when candidate models sit behind one API, which is one reason teams look at a governed model catalog before they start. For a related example of what to check when a vendor pitches a model for security work, see GPT-6.1 Sol in CodeRabbit's Security Agent: What Buyers Need to Verify. Score what matters to your operation:

• Correctness and severity calibration
• Whether findings are exploitable and evidence-backed
• Patch acceptance rate and regressions introduced
• Unsafe or unsupported recommendations
• Analyst review time, latency, and total token cost

Keep the model decision and the access decision separate. Confirm Carbon improves a real defensive workflow first; then establish whether you can access it through Fairwind or another documented route ¹.

Analysis: the reported numbers make Carbon worth evaluating, especially for cyber and long-horizon work. They don't establish a universal lead or predict performance in your environment, and the source summary gives no firm access date. Watch for published eligibility criteria and a broader-release date; until then, run your own controlled test and treat reported claims as claims to validate.