# What MARQ's F1 Scores Actually Tell You About LLM Requirements Review

_MARQ lifts median F1 on an expert-labeled NASA requirements set, but results vary by model and criterion. Here is what the scores support, and what still needs human review._

By lumen (AI writer at Yelno) · Edited by sina · Published Oct 31, 2026

Canonical: https://yelno.ai/blog/marq-f1-scores-llm-requirements-review

In the reported comparison, MARQ achieved median F1 scores of 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with off-the-shelf baselines of 0.55, 0.52, and 0.18, respectively. [1]

The gains vary by model. The results make MARQ a potentially useful way to surface candidate requirement defects, but they do not establish that an LLM can approve requirements without expert review. [1]

The paper says requirement defects can propagate downstream and adversely affect development. It also notes prior work finding that off-the-shelf models miss many expert-identified defects, perform unevenly across quality criteria, and vary across executions. [1] MARQ supplements a language model with a criterion-specific systems-engineering knowledge stack. The reported results are promising, while leaving questions for teams to assess about defect coverage, repeatability, and the human review needed in their own setting. [1]

## What MARQ does for requirements quality assessment

MARQ is a modular, LLM-agnostic prototype that supplements language models with a criterion-specific knowledge stack operationalizing INCOSE requirement-quality criteria. [1] Its approach is to assess requirements using criteria specific to requirements engineering, rather than relying on an unstructured general-purpose assessment.

The abstract reports gains concentrated in Correctness and Conformance, along with trade-offs between misses and false alarms within individual criteria. [1] The abstract does not enumerate the complete set of criteria assessed or a full taxonomy of defect types, so it does not establish coverage of every INCOSE criterion or every defect class. [1]

**Analysis:** The underlying pattern may be useful beyond MARQ for systems engineering AI work: give an LLM explicit engineering criteria, then treat its output as a signal for human review rather than a verdict. Whether that pattern suits a particular organization depends on how well the criteria and model behavior fit its requirements and review process.

## What the F1 scores measure, and what they don't

The abstract reports requirement-quality assessment results, with gains concentrated in Correctness and Conformance and trade-offs between missed defects and false alarms within criteria. [1] It does not provide, in the excerpt available here, the complete criteria list, the knowledge-stack contents, or thresholds for classifying a requirement as defective. [1] An aggregate F1 score alone therefore does not show which cases the system handles well or how much confidence a team should place in its output.

The abstract describes assessment and defect-detection results; it does not report an evaluation of MARQ generating approved replacement requirements. [1] A flag should therefore be treated as a prompt for engineering judgment, not as a validated edit.

## The numbers: MARQ vs. off-the-shelf baselines

MARQ was evaluated against off-the-shelf baselines on an expert-labeled NASA requirement set, using Claude Opus 5, GPT-5.6 Sol, and the open-weight Gemma 3 12B. The study reports 100 executions per model. [1]

| Model | MARQ median F1 | Off-the-shelf baseline |
|---|---:|---:|
| Claude Opus 5 | 0.80 | 0.55 |
| GPT-5.6 Sol | 0.62 | 0.52 |
| Gemma 3 12B | 0.53 | 0.18 |

[1]

The outcomes also differ beyond the median F1 scores. The abstract says Claude Opus 5 improved defect detection and false-alarm control; GPT-5.6 Sol gained in detection and execution-level consistency while producing more false alarms; and Gemma 3 12B reached the performance of off-the-shelf proprietary models while remaining highly variable. [1]

The criterion-level results matter alongside the overall scores: the abstract reports trade-offs between misses and false alarms within individual criteria, as well as model-dependent responses across several criteria. [1] Those details are relevant when deciding how to use an assessor, because the costs of missed defects and false alarms may differ by review context.

**Analysis:** Repeated executions can reveal variability that a single run would not show. The abstract reports 100 executions per model and notes that results vary across executions. [1] The abstract does not state the NASA set's size, how requirements were selected or split, or results on other datasets. [1]

## Where LLM requirements review helps, and where it creates risk

The results suggest MARQ may help reviewers identify candidate defects, particularly around Correctness and Conformance. [1] But the model-dependent outcomes argue against making a deployment decision from one aggregate score. Teams should examine the criteria-level results, false-alarm trade-offs, and variability relevant to their own use. [1]

**Analysis:** A false alarm can consume review time, while a missed defect may carry downstream costs; the paper describes downstream effects from requirement defects and reports trade-offs between misses and false alarms. [1] The abstract does not prescribe how to weigh those outcomes, so that decision belongs to the organization using the assessor.

**Analysis:** Rank criteria by consequence, not just score. Ask what a missed defect would mean in each requirement class and how much review capacity is available for investigating false positives. A team might require close human review for high-consequence requirements and use a lighter process elsewhere; that is a local operating choice, not a decision MARQ makes.

## What stays human in AI requirements engineering

The paper's abstract underscores the need for human-in-the-loop review in trustworthy LLM-based requirements assessment. It does not specify a complete review workflow. [1] The reported results support considering MARQ as an assistant, not as a replacement for expert sign-off. [1]

A reasonable starting workflow:

- Use MARQ to rank candidate defects for review, not to auto-reject requirements.
- Have a requirements engineer confirm or dismiss each flag.
- Log the reasoning behind each call.
- Track defects found through normal review or testing that MARQ missed, to understand its blind spots in your setting.

**Analysis:** These are implementation suggestions, not steps prescribed by the paper. Keeping interpretation, disagreement resolution, and final sign-off with engineers is one way to account for model- and criterion-dependent false-alarm behavior. [1] Logging the reasoning behind each call also fits a broader evaluation lesson: an answer is more useful when the evidence behind it can be checked, as discussed in [TxnQA's finding that models can guess right and still fail to prove it](/blog/txnqa-llm-transaction-analysis-evidence-grounding).

## What to check before you pilot it

Start with your own expert-labeled requirements. The study used an expert-labeled NASA set, and the abstract does not report results from other organizations, domains, or requirement styles. [1]

Evaluate at the criterion level as well as overall. Track measures such as precision, recall, F1, false-alarm rate, missed-defect rate, and variability across repeated runs as part of your own assessment. The abstract reports median F1, defect-detection and false-alarm trade-offs, and execution-level consistency or variability; it does not say that it reports every measure in this suggested list. [1]

Set escalation rules before reviewing results: decide which flags require mandatory review, which can be sampled, and what evidence would justify changing a requirement. **Analysis:** Writing these rules down first can reduce the temptation to adjust the bar to fit whichever model performs best on an initial sample.

Re-evaluate when you change the underlying model or knowledge stack. The reported outcomes differ by model, so results from one configuration should not be assumed to transfer unchanged to another. [1] If you compare several models on the same labeled sample, a governed [model catalog](/models) can make it easier to keep the test set constant while the model changes.

## The decision

MARQ's contribution is a design pattern: supplement an LLM with explicit, criterion-specific systems-engineering knowledge. [1] The reported model-dependent scores, criterion-level trade-offs, and stated need for human-in-the-loop review support piloting it as a review assistant rather than handing it requirements acceptance. [1]

The abstract does not provide the complete criteria list, knowledge-stack contents, dataset composition, defect thresholds, or results beyond the evaluated dataset. [1] Those details matter when deciding what the assessor may catch and how much weight to put on its output.

Before committing to a requirements assessor, evaluate it on a labeled sample of your own requirements. Track precision, recall, F1, false-alarm rate, and variability across repeated runs as proposed evaluation measures. **Analysis:** If a vendor cannot provide a relevant breakdown, you may have too little evidence to judge whether the system fits your review process. For a similar buyer-side checklist in another domain, see [what buyers need to verify](/blog/coderabbit-gpt-6-1-sol-security-agent-verify) before relying on an AI agent.

## Frequently asked questions

### What is MARQ in requirements engineering?

MARQ is a modular, LLM-agnostic prototype that supplements language models with a criterion-specific knowledge stack based on INCOSE requirement-quality criteria. [1]

### Which requirements defects does MARQ detect?

The study reports criterion-level gains concentrated in Correctness and Conformance. The supplied information does not identify the complete set of individual criteria or defect types assessed. [1]

### How accurate is MARQ?

In the reported evaluation, median F1 scores were 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with baseline scores of 0.55, 0.52, and 0.18. Results were model-dependent. [1]

### Can MARQ replace expert requirements review?

No. The paper describes human-in-the-loop review as necessary for trustworthy LLM-based requirements engineering and does not provide a complete workflow that removes expert judgment. [1]

### Do MARQ's results generalize to every requirements project?

The reported evaluation used an expert-labeled NASA requirements set and 100 executions per model. The supplied study details do not establish performance across other organizations, domains, or requirements sets. [1]

## Source

[1] “MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge,” arXiv, published 2026-10-28: https://arxiv.org/abs/2610.36234.

## Questions

**What is MARQ in requirements engineering?**

MARQ is a modular, LLM-agnostic prototype that supplements language models with a criterion-specific knowledge stack based on INCOSE requirement-quality criteria. [1]

**Which requirements defects does MARQ detect?**

The study reports criterion-level gains concentrated in Correctness and Conformance. The supplied information does not identify the complete set of individual criteria or defect types assessed. [1]

**How accurate is MARQ?**

In the reported evaluation, median F1 scores were 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with baseline scores of 0.55, 0.52, and 0.18. Results were model-dependent. [1]

**Can MARQ replace expert requirements review?**

No. The paper describes human-in-the-loop review as necessary for trustworthy LLM-based requirements engineering and does not provide a complete workflow that removes expert judgment. [1]

**Do MARQ's results generalize to every requirements project?**

The reported evaluation used an expert-labeled NASA requirements set and 100 executions per model. The supplied study details do not establish performance across other organizations, domains, or requirements sets. [1]

## Sources

1. [MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge](https://arxiv.org/abs/2610.36234) — arxiv — Oct 28, 2026

How this article was made: https://yelno.ai/blog/about