OrcaCyber Zero 1.5: What a 100% Cybench Score Actually Tells a Security Team

Orca reports that OrcaCyber Zero 1.5 solved 39 of 39 Cybench tasks and 23 of 24 evaluable CVE-Bench tasks. The figures are vendor-reported; the source says no technical report was available and does not document independent replication of these results.¹ For a security leader deciding whether to pilot the model, that score is a reason to run an evaluation, not a reason to skip one.

Orca says Zero 1.5 was post-trained for vulnerability reproduction, exploit development, penetration testing, security auditing, and cyber reasoning.¹ That focus makes it worth testing. It does not establish how the model will handle a specific company's codebase, tooling, or review process.

What this cybersecurity AI model is built to do

The model has a 1-million-token context window, a 128,000-token maximum output, native function calling, and structured outputs.¹ Those features may suit workflows involving large volumes of code, logs, or scan output. They do not establish whether the model will interpret a company's material correctly or produce findings anyone should act on.

Access is through OrcaRouter's hosted API; the source says there are no downloadable weights or quantized variants.¹ The API is described as OpenAI-compatible, with a base URL of https://api.orcarouter.ai/v1 and model ID orca/orcacyber-zero-1.5.¹ Compatibility may help get an integration started, but does not guarantee that every existing client, tool call, or workflow will run without rework.

Access requires OrcaRouter's Security Research tier. The listed requirements are an engagement, a passkey, and accepted terms; the tier targets trusted security researchers, red teams, and authorized testers.¹ The source does not state an approval timeline or account, concurrency, rate, or usage limits.¹

Treat the access process and hosted-service terms as part of the technical evaluation. Before sending source code, vulnerability evidence, logs, or tool output through the API, establish the applicable data-handling and retention terms. The published source does not state them.¹

What the Cybench and CVE-Bench scores actually cover

The reported 100% Cybench figure means 39 of 39 tasks solved under unrestricted agent execution. Cybench contains 40 professional capture-the-flag tasks.¹ The CVE-Bench result is 95.8%, or 23 of 24 evaluable tasks, in a benchmark built on 40 critical-severity web CVEs.¹

The source does not explain why only 24 of the 40 CVE-Bench tasks were evaluable or how that subset was chosen.¹ Without that explanation, a buyer cannot tell whether the subset represents the full benchmark, let alone the vulnerabilities their own team triages.

The published material also does not provide the prompts, scoring rules, model settings, or agent configuration behind the scores, or explain task selection.¹ Those omissions make it harder to reproduce the results or determine what completion meant in practice. A high score may still leave the method underspecified.

These numbers justify a structured proof of concept. They do not establish that the model will find or reproduce vulnerabilities reliably in production. A capture-the-flag task is not a substitute for testing against a team's own languages, frameworks, tools, and findings. A benchmark result is not a deployment decision, a point also made in our look at Sakana AI's 73% core-claim result. Use representative, authorized cases to assess whether that gap matters.

How much to trust vendor-reported benchmark numbers

The source labels the results vendor-reported and says no technical report was available. It also says competitor figures come from vendor announcements and have not been independently replicated.¹ That does not establish that the scores are wrong; it means procurement should record them as supplier claims, not validated results. The same logic applies to any reported result that needs independent verification.

Other numbers on the model page need the same scrutiny, with each result tied to the right model. Orca reports 93.9% on HumanEval+ and 76.5% on SWE-bench Pro V2; the source identifies the latter as the model's lowest published score and says it is not directly comparable with standard SWE-bench Pro results.¹ A separate 98% CyberGym claim belongs to the earlier OrcaCyber Zero 1.0 and Orca's own test harness.¹

Keep a supplier record of Orca's reported benchmarks and their stated limits. Separately, record results from fixed in-house test cases with consistent scoring and acceptance criteria. The internal evaluation, not the supplier claims, should drive the adoption decision.

Cybersecurity LLM pricing for a real workload

Orca lists these prices:¹

Token type	Listed price
Input	$3.00 per million tokens
Output	$7.50 per million tokens
Cache read	$0.75 per million tokens

A per-token rate does not predict what a security workflow will cost. A request carrying source code, scan output, prior findings, and conversation history uses a different volume of input tokens than a short audit question. Generated remediation guidance and reasoning output drive output-token usage separately. Cache reads have a separate rate.¹

During a pilot, track input, output, and cache-read tokens as separate line items. Tie cost to the action taken, such as vulnerability reproduction, audit assistance, remediation drafting, or analyst triage, rather than raw request counts. Orca lists several of these as target uses, but the source gives no workload-level cost figures.¹

Set a spending cap before scaling access, whether in your enterprise AI gateway or your own billing controls. A long context window is not a reason to send more material than a task requires. Scope each task to the material it needs, then weigh the resulting cost against the value of the outcome. This is a control for the team to set, not a claim about actual use of the model.

What the latency numbers do and don't say

For the past seven days covered by the source, Orca reports time-to-first-token of 500 milliseconds at p50 and 2.36 seconds at p95, from 1,300 tokens of traffic.¹ The source also reports a 3.43-second p50 for Zero 1.0 over a much larger traffic window, while cautioning that the latency figures are not a clean comparison.¹

These figures describe the reported window. They do not establish latency at a buyer's prompt sizes, concurrency, or traffic pattern. Time-to-first-token also does not measure the duration of a complete security workflow.

Measure end-to-end time to an approved result instead, including tool calls, retries, full generation, analyst correction, and required review. Use latency as a pilot pass/fail criterion only after defining the request shape and traffic pattern.

How to test AI vulnerability testing before buying

Build an evaluation set from authorized, representative material: previously resolved vulnerabilities, synthetic reproductions, secure code-review tasks, and remediation exercises. Freeze test cases and scoring rules so results remain comparable across model versions.

Score more than whether a task finishes. Track valid reproduction rate, false positives, incomplete or unsafe exploit guidance, remediation accuracy, tool-call correctness, analyst correction effort, token spend, and time to an approved result. These are buyer-proposed evaluation measures, not results Orca has published.

Run the model against the team's current process or other candidates side by side. Ask whether it improves a defined security outcome at acceptable cost and review burden, not whether it matches a vendor benchmark. Test the operating model at the same time: confirm the Security Research tier process and API integration, and establish usage limits, data-handling terms, logging, and budget controls. The source leaves these details unstated.¹

Keep a human involved in reviewing findings and testing actions; our piece on what evidence is enough before an agent can take action covers how to set that bar. Start with bounded tasks where authorization and expected output are clear. Widen scope only after the model meets the team's own thresholds for quality, cost, latency, and operational fit.

What goes in the decision memo

Split the memo into supplier evidence and internal evidence. Supplier evidence should record Orca's Cybench and CVE-Bench claims, context and output limits, token prices, and reported latency sample.¹ Internal evidence should record results from the team's fixed test set, including cost and review effort.

State the open questions directly: benchmark configuration, the evaluable CVE-Bench subset, independent replication, access approval, usage limits, and data handling are not established in the published material.¹ Assign an owner to each: security engineering for technical performance, the AI platform team for integration and cost, procurement for hosted-service terms, and legal or privacy for authorized-testing controls.

Frame the first approval as a controlled pilot, not a production rollout. Set stop conditions before work starts. If the model misses the team's thresholds for security quality, cost, latency, or governance, do not extend its use on the strength of a published benchmark.

Next step: confirm the access process and request any further access and data-handling details, then run a bounded evaluation against representative, authorized cases with scoring rules fixed in advance.

FAQ

What is OrcaCyber Zero 1.5?

It is a hosted cybersecurity model from OrcaRouter, post-trained for vulnerability reproduction, exploit development, penetration testing, security auditing, and cyber reasoning. The source lists a 1-million-token context window, a maximum output of 128,000 tokens, native function calling, and structured outputs.¹

What score did OrcaCyber Zero 1.5 receive on Cybench?

Orca reports 100%, covering 39 of 39 tasks with unrestricted agent execution. The source says the figures are vendor-reported and have not been independently replicated.¹

What is the CVE-Bench score for OrcaCyber Zero 1.5?

Orca reports 95.8%, covering 23 of 24 evaluable tasks. The source does not explain why only 24 tasks were evaluable or how that subset was selected.¹

How much does OrcaCyber Zero 1.5 cost?

The listed price is $3.00 per million input tokens, $7.50 per million output tokens, and $0.75 per million cache-read tokens.¹ Actual workload cost depends on the input, output, and cache usage measured by the buyer.

How can a security team access the model?

Access is through OrcaRouter's hosted API and is gated to the Security Research tier. Orca lists an engagement, a passkey, and acceptance of terms as requirements.¹