---
title: "In RLHF-shaped work, evaluation is the job"
description: "Model names get the headlines; judgment criteria get the results. When reviewing code-oriented LLM outputs, vague 'looks good' feedback trains noise. Explicit rubrics, correctness,"
canonical: "https://www.ayushrai.site/notes/eval-is-the-job/"
last-updated: "2026-10-16"
---

# In RLHF-shaped work, evaluation is the job

Source: https://www.ayushrai.site/notes/eval-is-the-job/
Markdown: https://www.ayushrai.site/notes/eval-is-the-job.md

Model names get the headlines; judgment criteria get the results. When reviewing code-oriented LLM outputs, vague 'looks good' feedback trains noise. Explicit rubrics, correctness,

Kind: observation. Published: 2026-08-08.

Model names get the headlines; judgment criteria get the results. When reviewing code-oriented LLM outputs, vague 'looks good' feedback trains noise. Explicit rubrics, correctness, edge cases, security smells, make human feedback something a training loop can use. Brand proximity to big AI labs is not a substitute for that discipline.

_Unverified award metrics are omitted. Wording matches the public site._