CleanRM Claims a Few Clean Labels Can Help Correct Noisy Reward Models—but Its Abstract Leaves Practical Questions Open

CleanRM proposes using a small set of clean preference feedback to calibrate sample-wise noise in reward-model training. The paper identifies subjective human judgments, annotation mistakes, and imperfect LLM judges as sources of noise in preference data. ¹ The abstract describes the method, but does not specify how clean feedback is verified, how many examples are needed, or the size of the reported gains. ¹

The paper, “CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF,” says it calibrates sample-wise noise ratios via optimal transport, then trains with a noise-corrected surrogate loss. It reports that CleanRM outperforms state-of-the-art denoising baselines and improves downstream RLHF alignment. ¹ The abstract does not name the benchmarks or baselines, or provide metrics and effect sizes. ¹ That leaves the abstract’s performance claims difficult to assess for a particular production setting. It is a familiar pattern: we saw something similar in three agent self-improvement papers that claim gains, where the strength of the evidence varied widely.

How CleanRM handles noisy preference data, as described

The abstract says standard reward modeling on noisy feedback produces a biased estimate of clean risk. It says classical denoising methods generally assume a uniform noise ratio across samples, while feedback noise can vary sample by sample. ¹

In broad terms, the described method is:

1. Use a few clean feedback examples to calibrate sample-wise noise ratios.
2. Use optimal transport to align the joint distribution of preferences recovered from noisy feedback with that of clean feedback.
3. Train the reward model using a noise-corrected surrogate loss. ¹

The abstract does not describe a protocol for collecting or verifying clean feedback, or explain how the authors determine whether an example qualifies as clean. ¹

Why “clean” human feedback is doing much of the work

Clean feedback is central to the method, but the abstract does not say who verified those examples, how disagreements were handled, or what distinguished them from the noisy feedback. ¹

The composition of the clean examples could also matter: if they represent only a narrow portion of the feedback, the calibration may not generalize to other examples. That is a concern to investigate, not a result reported in the abstract. The abstract does not report tests of selection bias, sensitivity to errors in the clean subset, or a curve showing how results change with clean-set size. ¹

What the abstract compares against

Approach	How it treats noise	What the abstract says
Standard reward modeling	No noise correction described	Yields a biased estimate of clean risk. 1
Classical denoising	Generally assumes a uniform noise ratio across samples	The authors say that assumption does not fit feedback’s sample-wise noise. 1
CleanRM	Calibrates sample-wise noise ratios via optimal transport and uses a corrected loss	Its objective is proved unbiased with respect to clean risk, and experiments are reported to outperform state-of-the-art denoising baselines. 1

The abstract does not name the baselines, datasets, or metrics. ¹ Without those details, readers cannot assess the scale or consistency of the reported advantage from the abstract alone.

An unbiased objective is not an empirical result

The authors say they prove the resulting objective is unbiased with respect to clean risk. ¹ That is a theoretical claim about the objective; by itself, it does not establish whether the method’s assumptions hold on a given dataset or whether it will improve downstream behavior in a particular application.

The abstract also does not report tests of how calibration responds to mislabeled or unrepresentative clean examples. ¹ Those are reasonable questions for an evaluation because the method uses clean feedback to calibrate noise ratios.

What the abstract does not report about reward modeling and RLHF feedback

The abstract reports extensive experiments, says CleanRM outperforms state-of-the-art denoising baselines, and says it yields more effective downstream RLHF alignment. ¹ It does not provide dataset names, model details, benchmark tasks, metrics, numerical results, effect sizes, clean-feedback provenance, computational cost, or integration requirements. ¹

A useful evaluation would include clean-set-size ablations, a documented verification protocol, tests involving mislabeled or unrepresentative clean data, and comparisons with named baselines on shared data. Reporting both reward-model and downstream RLHF metrics would help readers judge practical relevance. These are suggested evaluation criteria; the abstract does not report them. ¹

Before running a pilot, ask for

• A clean-feedback protocol: What makes an example clean, and who verifies it? The abstract does not say. ¹
• A sample-efficiency curve: How do results change as the clean set grows? The abstract does not report one. ¹
• Robustness tests: What happens when clean feedback is wrong or unrepresentative? The abstract does not report such tests. ¹
• Named baselines and evaluation details: Which denoising methods, datasets, and metrics support the reported gains? The abstract does not specify them. ¹
• Cost accounting: What additional training time or budget does clean-feedback collection and the optimal-transport step require? The abstract does not address this. ¹

The bottom line

The abstract describes a plausible approach: use a few clean feedback examples to calibrate sample-wise noise ratios, align preference distributions via optimal transport, and train with a corrected loss. It also reports a theoretical unbiasedness result and experimental gains over denoising baselines and in downstream RLHF alignment. ¹ But it does not explain how clean feedback is defined or verified, how many examples are needed, or which baselines and measurements support the reported gains. ¹

A controlled pilot could test the method on the same data as a team’s current approach while varying clean-set size and tracking relevant reward-model and downstream outcomes. Checking those outcomes again after rollout, in the spirit of drift checks rather than launch tests alone, would be a sensible addition. That is a proposed evaluation, not a result established by the abstract. The abstract supports interest in the method, but does not provide enough detail to judge its practical performance. ¹ The same caution applies to other claims that rest on an abstract alone.

FAQ

What is CleanRM?

CleanRM is a reward-modeling method that uses a few clean feedback examples to calibrate sample-wise noise in preference data, align distributions with optimal transport, and train with a noise-corrected loss. ¹

Why does CleanRM use clean feedback?

The paper argues that noisy feedback can bias the estimated clean risk and that common denoising methods may rely on a uniform noise ratio even when noise varies by example. ¹

How many clean feedback examples does CleanRM need?

The supplied abstract does not specify a number or provide a performance curve showing how results change as the clean set grows. ¹

Does CleanRM outperform existing reward-modeling methods?

The abstract says it outperforms state-of-the-art denoising baselines and improves downstream RLHF alignment, but it provides no baseline names, benchmark results, or effect sizes. ¹

Is CleanRM proven for production use?

No production conclusion can be drawn from the supplied abstract; it does not report the clean-label process, operational costs, sensitivity to clean-set errors, or detailed evaluation results. ¹