---
title: "AI engineering"
description: "Ayush Rai on AI engineering: LLM evaluation, preference feedback, and AI products, with the work and the limits named."
canonical: "https://www.ayushrai.site/ai-engineering/"
last-updated: "2026-10-16"
---

# AI engineering

Source: https://www.ayushrai.site/ai-engineering/
Markdown: https://www.ayushrai.site/ai-engineering.md

Ayush Rai on AI engineering: LLM evaluation, preference feedback, and AI products, with the work and the limits named.

Evidence level: Mixed: shipped pieces and explicit limits.

## In this section

- [LLM evaluation](/ai-engineering/llm-evaluation.md): Ayush Rai on LLM evaluation: rubrics, review loops, and reliability for model outputs, grounded in contract evaluation work.
- [RLHF and preference feedback](/ai-engineering/rlhf.md): Ayush Rai on RLHF-shaped work: human review and preference signals for model outputs, scoped to the experience record — not reward-model training.
- [LLM systems](/ai-engineering/llm-systems.md): Product systems where a model proposes and surrounding software decides. MedVoice is the clearest case. AGENYRA is the structured-data case.
- [AI agents](/ai-engineering/agents.md): Agent-shaped product thinking and the read-only agent interfaces on this site. Not a production agent orchestrator.
- [Retrieval and RAG](/ai-engineering/rag.md): An honest page about retrieval: one legal-text investigation mentioned it. I have not shipped a production RAG system.

## Scope

On this site, AI engineering means the practical work of making model-shaped systems usable: judging outputs, constraining what a model is allowed to decide, and shipping the software around that judgment.

The pages under this hub separate contract evaluation work from product work and from research notes. Those are different kinds of evidence.

## What I have actually done

From April 2024 to August 2025 I did AI engineering and technical evaluation at FoCDoT, reviewing model outputs against criteria and analyzing coding and STEM tasks. Since August 2025 I have contributed, through Outlier, to generative-AI evaluation workflows for frontier-lab programs. That is contract evaluation work, not employment by those labs.

The product work is separate. MedVoice is a voice-assisted wholesale billing workflow where speech proposes a line and deterministic rules decide identity, tax, and posting. AGENYRA is a distribution layer that starts as structured product data. The Open Framework is an early civil-society coordination prototype, not an LLM platform.

## What this hub does not claim

I do not publish benchmark scores, user counts, or a ranking against other engineers. I also do not treat model-serving, quantization, or a general inference platform as a shipped specialty. Those topics stay off their own URLs until there is a system to show.

- LLM evaluation and RLHF-shaped review: contract work, with detail limited by the engagement.
- LLM-shaped products: MedVoice and AGENYRA, both still building.
- Retrieval: an older legal-text investigation, not a production RAG stack.
- Agents: product and interface thinking, not a multi-agent runtime I operate in production.

## Evidence

- [FoCDoT, SDE — AI Engineer](/experience#focdot-sde-ai): April 2024–August 2025. Evaluation and STEM/coding-task analysis.
- [Outlier, AI evaluation contributor](/experience#outlier-genai): Contract work from August 2025. Frontier-lab programs via the platform, not direct employment.
- [MedVoice case study](/work/medvoice): Speech proposes. Rules decide medicine, price, and tax.
- [Note: evaluation is the job](/notes/eval-is-the-job): Why vague feedback is not a training signal.

## Related

- [LLM evaluation](/ai-engineering/llm-evaluation)
- [RLHF and preference feedback](/ai-engineering/rlhf)
- [LLM systems](/ai-engineering/llm-systems)
- [Agents](/ai-engineering/agents)
- [Retrieval](/ai-engineering/rag)
- [Software engineering](/software-engineering)


## Further reading

- [Hugging Face Learn](https://huggingface.co/learn): Public courses. A study resource, not a credential I am claiming.

## Gaps

- No public eval harness, preference dataset, or benchmark table from the contract work. The engagements do not allow that dump.
- No production inference, quantization, or model-serving write-up. I am not publishing an inference page to fill the gap.
- No claim of training a frontier model, owning a lab pipeline, or leading an RLHF program.

Contact: /contact

_Unverified award metrics are omitted. Wording matches the public site._