Hiring an 'Agent Engineer': JD and Skills Profile
The role exists, sort of. The skills, the interview signals, and the JD template, with the parts that should differ between platform teams and product teams.
The skills profile
Strong software engineering: the agent is a software system, not just a prompt. Engineering rigour matters.
Evals fluency: knows how to design test suites, score outputs, detect regressions. This is the rare skill.
Prompt engineering: can write and refine prompts. Less rare than evals but still uncommon at depth.
Production operations: knows what observability, on-call, and SLOs mean. Otherwise the agent ships without them.
Interview signals
"Tell me about an eval suite you built." Real candidates have stories; pretenders have generalities.
"How do you debug a prompt that started failing in production?" Real candidates describe a methodology; pretenders describe symptoms.
"What is the cost trade-off you would make between Haiku and Opus for triage?" Real candidates have an opinion with reasons; pretenders quote vendor pages.
Take-home: design an agent for X workflow. Real candidates produce a small but coherent design; pretenders produce a wishlist.
JD for platform team
Focus: build the agent platform that other teams use. Think infrastructure engineer with LLM specialty.
Skills: distributed systems, observability, API design. LLMs as one component among many.
Compensation: senior software engineer level, with LLM premium of 10-15%.
JD for product team
Focus: build agents for specific workflows. Think feature engineer with LLM specialty.
Skills: domain expertise, prompt engineering, eval design. LLMs as the core tool.
Compensation: senior software engineer level, with domain premium based on the product area.
Retaining them
Give them ownership of the eval suite. Without owning quality, they cannot improve it.
Give them visibility into the impact. Their work changes MTTR; show the chart.
Give them time for craft. Prompt and eval engineering reward iteration; under-resourcing produces mediocrity.