Reflecting on a project: LLM-driven development and addressing friction
July 2025
I had the opportunity to help shape a complex project whose primary developer and driver strongly embraced LLMs and agent-driven development. He wanted to move extremely fast. Others were worried we were abandoning basic engineering discipline.
The pressure to deliver, along with the optimistic stance, led to clashes with implicit assumptions. The tension wasn’t about intelligence or capability but about different risk tolerances and motivations.
It is not a new problem but is likely to get worse as the ability to write code outpaces other processes.
Fear leads to anger - yes, its human to get emotional at work
Raising my two kids teaches me a lot about myself and my interactions with others. It contextualises things. Here, naming the fears helped. A lot of folks were frustrated at that point, and I was mostly avoiding it because I was riding the high of just getting promoted to a staff level.
We are operating in a wildly different way - sure, but what exactly is causing the frustrations?
Fear of not being listened to - can imply that you are dumb and not worth listening to. It can trigger the impostor syndrome.
Fear of not being valued - an age-honed positive reinforcement loop taught you what good looks like. If that is not important (and you know it still is!), then what is my role here?
Fear of being let go - Q&A numbers going down led to uncertainty about the product and a fear of a new round of layoffs. We need to prove ourselves or even save the company wholly.
What reading Difficult Conversations thought me is that simply naming the fears helps. Take the fear of not being listened to. We can gently distance ourselves (or another person) from not agreeing with someone meaning that what you are saying is stupid. We can expose a likely mismatch in the implicit assumptions.
Identifying what different folks are hedging against
While we can’t do much about some fears there are others we can reframe as risks - technical things within our control. Risk might be another word for fear, but with the emotion removed. There are two common mismatches at a high level:
- The operational risk
- Development best practices
We broke down some of the concerns further and decided if we’re happy about how we’re mitigating them:
- Risk of disrupting the user experience (this is a key concern that deserves further exploration)
- Risk of exposing user information
- Risk of causing issues in other parts of the site
- Risk of overloading the system, which could lead to a complete failure of the feature
- Risk of slow response times
- Risk of excessive spending on cloud compute credits (this was particularly interesting for me as it was my first experience provisioning powerful ML nodes and realistically considering ingress and egress due to the large sizes of the feature stores we were transferring)
- Risk of creating an unmaintainable system
On the flip side: Risk of not delivering - trying out new things and improving the experience to surface interesting questions is what we believed in.
Explicitly stating what we cared about and agreeing on what the priorities were we broke the standstill and were able to move forward - tackle the risks.
Gaining confidence through observability and high level tests
Tacking the operational risk can be fairly simple - minimise the surface area by decoupling the systems, putting the integration point under a lot of scrutiny. (It still ended up with bugs, but with fallbacks the team got the confidence that even a broken system won’t be catastrophic).
Adding a lot of observability (p95 SLO for response times and error rates) as well as measuring a proxy for user impact (click through rate) further increased the confidence - we were not working blind.
Deployment verification tests were added - no deployment of the recommendation system was marked healthy unless a few smoke tests succeeded (we codified the few things we found ourselves doing manually after every deployment - for a few of our internal users we asserted a few well-known expected suggestions were appearing).
Closing thoughts
Now more than ever, going back to the basics leads to delivering with less frustration and with more confidence. While the project had other compounding factors, getting on the same page around goals and values will always be step 0.
Development good practices are a separate topic and the other key side of the coin - something to reflect on in the future.