Role-aware rubrics: one prompt engine, thirty assessment shapes

The first version of any LLM assessment system is one big prompt. It works in the demo, then falls apart on the second candidate type. HireIQ assesses sales candidates for Apprento, and the range is wide: a career starter applying for an SDR seat and a twenty-year veteran going for a sales leadership role should not be scored against the same criteria. A one-size prompt either judges the junior unfairly or flattens the senior into noise.

The obvious fix is one prompt per role. With three seniority levels and ten role types, that is thirty prompts to keep in sync. Every wording improvement, every output-format change, every bug fix has to be replicated thirty times, and drift between copies becomes a silent quality bug. I have seen where that road goes and did not want to be on it.

Separate what you score from how you prompt

The design that held up was splitting the system into an engine and a rubric. The engine owns everything invariant: document handling, the prompt scaffold, the typed JSON report shape (executive summary, per-dimension scorecard, fit analysis, targeted probe questions), schema validation, and retry logic for malformed responses. The rubric owns everything that varies by candidate: which dimensions get scored, what good looks like at this seniority, and which role-specific signals matter.

Rubrics compose at runtime from two axes: a seniority framework (career starter, experienced salesperson, sales leader) and a role overlay (SDR, AE, BDM, CSM, and six others). The engine selects and injects the pair per request. Thirty effective assessment shapes, one prompt codebase. A wording fix lands once and applies everywhere.

Why typed output is non-negotiable

Everything downstream of the model — the report renderer, the recruiter review UI, the scorecard — depends on structure. So the engine demands a typed JSON shape and validates it on every response. When validation fails, it retries; when the primary provider degrades, a fallback path (Groq) keeps the workflow alive, because recruiters use this inside their daily flow and a provider outage cannot mean they stop working.

The validation layer is also what makes the rubric system safe to extend. A new role overlay cannot break the report contract, because the contract is enforced after the model, not assumed of it.

The gap I would close first

What this architecture still needs is an eval layer: a fixed suite of known CV-and-transcript pairs with expected report properties, run on every rubric change. I built the prompting iteratively against real example reports, which got quality high, but without automated evals each rubric change means manual spot-checking. When output quality feeds hiring decisions, that is the next thing to build, not a nice-to-have. If you are building anything similar, build the eval harness the same week you build the second rubric.