When Agents Move Faster Than Organisations Can Think
Episode 5
AI coding agents can turn an initial idea into thousands of lines of code before a team has discussed whether the approach is sound. Kesha Mykhailov explains how that speed is changing development at Fin: technical plans and the conversations behind them are disappearing, while expert reviewers face an unsustainable volume of generated code.
Fin’s response is to integrate more of the organisation’s intelligence into the agent workflow. Kesha describes how expert React review comments became shared guidance for both code generation and verification, why deterministic hooks load critical context, and how their code-review agent Shrek rewards small pull requests. He also recounts an evaluation failure in which a cheaper model preserved issue recall but produced damaging false positives—showing why offline benchmarks must be combined with human and production signals.
About Kesha
Kesha Mykhailov is an engineer at Fin working on its internal AI developer platform and Shrek, its code-review agent. His infrastructure background shapes an enablement-focused approach: capture expert knowledge, make it available across the engineering organisation, and build controls that help developers use coding agents safely without surrendering their increased velocity.
In this episode
- 00:49 — Fin’s evolution from Intercom and its AI customer-service agent
- 01:49 — The scale and distribution of Fin’s engineering organisation
- 02:44 — Why adopting AI coding agents became a competitive necessity
- 05:07 — Standardising on Claude Code to create a shared internal platform
- 07:01 — How fast code generation displaced technical plans and early alignment
- 09:22 — The hidden rationale that source code and agents cannot reveal
- 10:08 — Using auto-approval to encourage smaller pull requests
- 13:36 — Why agents struggle to divide work into valuable vertical slices
- 15:31 — Encoding expert context in skills and loading it with deterministic hooks
- 17:12 — A React migration exposes the cost of unconstrained AI-generated code
- 18:47 — Turning expert review comments into reusable guidance
- 21:54 — Why an AI agent is an evolving process rather than a finished system
- 22:22 — Measuring review quality through reactions, applied feedback and evaluations
- 25:18 — Shrek’s problem-statement, security and logical-correctness reviewers
- 27:16 — A cheaper model passes the evaluation but creates a false-positive incident
- 30:01 — How evaluation datasets inherit the biases of the models that produced them
- 32:12 — Using production signals and A/B tests to measure effects on engineers
- 34:54 — Why Fin built its own reviewer instead of relying on an off-the-shelf product
- 35:51 — One source of expert guidance for generation, questions and review
- 38:44 — Making organisation-specific knowledge available through shared skills
- 39:38 — Teaching the agent to ask for intent before creating a pull request
- 42:01 — Enforcing critical workflow guidance through hooks
- 42:56 — Detecting a model-driven increase in misleading code comments
- 44:18 — Building security controls into the common agent platform
- 45:58 — Cost, model choice and the difficulty of changing an established harness
- 48:35 — Why reliable evaluations matter more than post-training alone
- 50:54 — Bringing infrastructure enablement experience into the AI platform team
- 52:15 — Engineers shift from writing code to choosing and validating valuable work
- 54:27 — Why newcomers struggle to acquire the organisational context behind the code
- 56:47 — How Shrek’s personality helps Fin counter automated-review fatigue
Key ideas
- When implementation becomes nearly free, teams can turn the first plausible idea into code before discussion exposes its assumptions and hidden constraints.
- Technical plans were valuable partly because they recorded conversations that had already aligned people around an approach; preserving the document alone would not preserve that deliberation.
- Expert knowledge should guide code generation and verification from the same maintained source, while independent security, correctness and human checks challenge the resulting work from different angles.
- Deterministic hooks are useful when important context cannot be left to an agent’s judgement about which skill to invoke.
- Fin makes small changes the faster path by withholding automatic approval from large pull requests, using the workflow’s incentives to protect reviewability.
- Offline evaluations describe only the behaviours represented in their datasets. Shrek’s model downgrade preserved recall but increased false positives that disrupted real engineers.
- AI systems require continuous operation: model behaviour changes, prompts decay, costs shift and production signals reveal effects that static evaluations miss.
- Faster implementation moves engineering effort towards identifying valuable problems, validating customer outcomes and maintaining the environment in which agents work.
- Skills can distribute explicit standards and rationale, but they cannot completely replace the conversations through which newcomers acquire tacit organisational context.
