What MARQ's F1 Scores Actually Tell You About LLM Requirements Review

In the reported comparison, MARQ achieved median F1 scores of 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with off-the-shelf baselines of 0.55, 0.52, and 0.18, respectively. ¹

The gains vary by model. The results make MARQ a potentially useful way to surface candidate requirement defects, but they do not establish that an LLM can approve requirements without expert review. ¹

The paper says requirement defects can propagate downstream and adversely affect development. It also notes prior work finding that off-the-shelf models miss many expert-identified defects, perform unevenly across quality criteria, and vary across executions. ¹ MARQ supplements a language model with a criterion-specific systems-engineering knowledge stack. The reported results are promising, while leaving questions for teams to assess about defect coverage, repeatability, and the human review needed in their own setting. ¹

What MARQ does for requirements quality assessment

MARQ is a modular, LLM-agnostic prototype that supplements language models with a criterion-specific knowledge stack operationalizing INCOSE requirement-quality criteria. ¹ Its approach is to assess requirements using criteria specific to requirements engineering, rather than relying on an unstructured general-purpose assessment.

The abstract reports gains concentrated in Correctness and Conformance, along with trade-offs between misses and false alarms within individual criteria. ¹ The abstract does not enumerate the complete set of criteria assessed or a full taxonomy of defect types, so it does not establish coverage of every INCOSE criterion or every defect class. ¹

Analysis: The underlying pattern may be useful beyond MARQ for systems engineering AI work: give an LLM explicit engineering criteria, then treat its output as a signal for human review rather than a verdict. Whether that pattern suits a particular organization depends on how well the criteria and model behavior fit its requirements and review process.

What the F1 scores measure, and what they don't

The abstract reports requirement-quality assessment results, with gains concentrated in Correctness and Conformance and trade-offs between missed defects and false alarms within criteria. ¹ It does not provide, in the excerpt available here, the complete criteria list, the knowledge-stack contents, or thresholds for classifying a requirement as defective. ¹ An aggregate F1 score alone therefore does not show which cases the system handles well or how much confidence a team should place in its output.

The abstract describes assessment and defect-detection results; it does not report an evaluation of MARQ generating approved replacement requirements. ¹ A flag should therefore be treated as a prompt for engineering judgment, not as a validated edit.

The numbers: MARQ vs. off-the-shelf baselines

MARQ was evaluated against off-the-shelf baselines on an expert-labeled NASA requirement set, using Claude Opus 5, GPT-5.6 Sol, and the open-weight Gemma 3 12B. The study reports 100 executions per model. ¹

Model	MARQ median F1	Off-the-shelf baseline
Claude Opus 5	0.80	0.55
GPT-5.6 Sol	0.62	0.52
Gemma 3 12B	0.53	0.18

¹

The outcomes also differ beyond the median F1 scores. The abstract says Claude Opus 5 improved defect detection and false-alarm control; GPT-5.6 Sol gained in detection and execution-level consistency while producing more false alarms; and Gemma 3 12B reached the performance of off-the-shelf proprietary models while remaining highly variable. ¹

The criterion-level results matter alongside the overall scores: the abstract reports trade-offs between misses and false alarms within individual criteria, as well as model-dependent responses across several criteria. ¹ Those details are relevant when deciding how to use an assessor, because the costs of missed defects and false alarms may differ by review context.

Analysis: Repeated executions can reveal variability that a single run would not show. The abstract reports 100 executions per model and notes that results vary across executions. ¹ The abstract does not state the NASA set's size, how requirements were selected or split, or results on other datasets. ¹

Where LLM requirements review helps, and where it creates risk

The results suggest MARQ may help reviewers identify candidate defects, particularly around Correctness and Conformance. ¹ But the model-dependent outcomes argue against making a deployment decision from one aggregate score. Teams should examine the criteria-level results, false-alarm trade-offs, and variability relevant to their own use. ¹

Analysis: A false alarm can consume review time, while a missed defect may carry downstream costs; the paper describes downstream effects from requirement defects and reports trade-offs between misses and false alarms. ¹ The abstract does not prescribe how to weigh those outcomes, so that decision belongs to the organization using the assessor.

Analysis: Rank criteria by consequence, not just score. Ask what a missed defect would mean in each requirement class and how much review capacity is available for investigating false positives. A team might require close human review for high-consequence requirements and use a lighter process elsewhere; that is a local operating choice, not a decision MARQ makes.

What stays human in AI requirements engineering

The paper's abstract underscores the need for human-in-the-loop review in trustworthy LLM-based requirements assessment. It does not specify a complete review workflow. ¹ The reported results support considering MARQ as an assistant, not as a replacement for expert sign-off. ¹

A reasonable starting workflow:

• Use MARQ to rank candidate defects for review, not to auto-reject requirements.
• Have a requirements engineer confirm or dismiss each flag.
• Log the reasoning behind each call.
• Track defects found through normal review or testing that MARQ missed, to understand its blind spots in your setting.

Analysis: These are implementation suggestions, not steps prescribed by the paper. Keeping interpretation, disagreement resolution, and final sign-off with engineers is one way to account for model- and criterion-dependent false-alarm behavior. ¹ Logging the reasoning behind each call also fits a broader evaluation lesson: an answer is more useful when the evidence behind it can be checked, as discussed in TxnQA's finding that models can guess right and still fail to prove it.

What to check before you pilot it

Start with your own expert-labeled requirements. The study used an expert-labeled NASA set, and the abstract does not report results from other organizations, domains, or requirement styles. ¹

Evaluate at the criterion level as well as overall. Track measures such as precision, recall, F1, false-alarm rate, missed-defect rate, and variability across repeated runs as part of your own assessment. The abstract reports median F1, defect-detection and false-alarm trade-offs, and execution-level consistency or variability; it does not say that it reports every measure in this suggested list. ¹

Set escalation rules before reviewing results: decide which flags require mandatory review, which can be sampled, and what evidence would justify changing a requirement. Analysis: Writing these rules down first can reduce the temptation to adjust the bar to fit whichever model performs best on an initial sample.

Re-evaluate when you change the underlying model or knowledge stack. The reported outcomes differ by model, so results from one configuration should not be assumed to transfer unchanged to another. ¹ If you compare several models on the same labeled sample, a governed model catalog can make it easier to keep the test set constant while the model changes.

The decision

MARQ's contribution is a design pattern: supplement an LLM with explicit, criterion-specific systems-engineering knowledge. ¹ The reported model-dependent scores, criterion-level trade-offs, and stated need for human-in-the-loop review support piloting it as a review assistant rather than handing it requirements acceptance. ¹

The abstract does not provide the complete criteria list, knowledge-stack contents, dataset composition, defect thresholds, or results beyond the evaluated dataset. ¹ Those details matter when deciding what the assessor may catch and how much weight to put on its output.

Before committing to a requirements assessor, evaluate it on a labeled sample of your own requirements. Track precision, recall, F1, false-alarm rate, and variability across repeated runs as proposed evaluation measures. Analysis: If a vendor cannot provide a relevant breakdown, you may have too little evidence to judge whether the system fits your review process. For a similar buyer-side checklist in another domain, see what buyers need to verify before relying on an AI agent.

Frequently asked questions

What is MARQ in requirements engineering?

MARQ is a modular, LLM-agnostic prototype that supplements language models with a criterion-specific knowledge stack based on INCOSE requirement-quality criteria. ¹

Which requirements defects does MARQ detect?

The study reports criterion-level gains concentrated in Correctness and Conformance. The supplied information does not identify the complete set of individual criteria or defect types assessed. ¹

How accurate is MARQ?

In the reported evaluation, median F1 scores were 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with baseline scores of 0.55, 0.52, and 0.18. Results were model-dependent. ¹

Can MARQ replace expert requirements review?

No. The paper describes human-in-the-loop review as necessary for trustworthy LLM-based requirements engineering and does not provide a complete workflow that removes expert judgment. ¹

Do MARQ's results generalize to every requirements project?

The reported evaluation used an expert-labeled NASA requirements set and 100 executions per model. The supplied study details do not establish performance across other organizations, domains, or requirements sets. ¹

Source

¹ “MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge,” arXiv, published 2026-10-28: https://arxiv.org/abs/2610.36234.