Two Years of Learning to Code with LLMs
I started deeply coding with LLMs sometime at the end of 2022 and the beginning of 2023 when GPT-4 came out. We’ve come a long way. Back in the day (👴, circa 2022) I spent lots of time copy-pasting between GPT-4 and jetbrains. Working on a startup then, I immediately recognized the ability for this to speed up my workflow.
In a weird way, for a senior developer, I don’t feel that LLMs have really helped me all that significantly with speed more since about 6 months after GPT-4 came out and I was used to coding with them. My workflow has changed a lot and we have more powerful tools (note how I didn’t say better).
I could talk at length about where I think it’s going with engineering, but I wanted to write down some high-level lessons for people who coded before LLMs on how to use them correctly.
The most important lesson is that before you start a feature, you need to decide how it’s going to be coded. We never did this before, we just coded it. Is this something I need to hand write, LLM draft and then engineer in detail, or vibe code and throw into production with little consequences?
The key to this is understanding the impact and risk of what you’re coding. If it breaks, is it risky? How much will it be extended (ie how much do I and others need to understand it)? I’ve found it’s toxic to have core code in the codebase that no one understands, but also often your core engineers shouldn’t be spending key cycles thinking about the details of posting slack-webhook updates for your dev server. It’s a balance.
My current split of code written (heavily depending on the project and user imact) looks something like:
25% code that is strictly my own - I write it by hand
50% code than an LLM drafted and I reviewed and understand completely (basically a code review)
25% pure LLM code where my mental model is that if it breaks I’ll circle back later but it’s not a big deal
I enjoy using cursor - the diff view and agents are very useful, but I also think it has a lot of downsides. It’s easy to get stuck in a place where you can footgun yourself by getting intellectually lazy and not understanding code you really should. My guess is that we’ll have different splits of tools down the line with things like lovable being closer to former no-code tools, cursor being the VS-code alternative, and then I’m expecting there will be some jetbrains-like interface for power users who want full control over their workflow.
I’ve worked a bit on my own custom tooling too as I’ve played with things that I’ll post below. It’s all a work in progress but overall amazing for engineering.
Power-user tools I’ve explored myself
Yoink - cmd line tool for recursively pulling all context from a directory or website
git aliasing for giving agents credit in PRs