Coding with LLMs is all about knowing when to stop
My trusty AI sidekick and I have been working hard this week, burning tokens and blowing my high-model budget. I admit to having a bit of a crush on Claude 4.7 Opus - it really gets in there, solving some intractable problems, and making some beautiful UI while it's at it.
I've been making RAG pipelines, my current obsession. My first, which I built the old-fashioned way over two years ago, took me months. About three months ago, it took me about two weeks with the help of the previous-gen AIs. This time, I had a pretty good base in two days. Admittedly, it was a combination of my own hard-won experience and the leap forward in the "frontier" models, but it was in no small part thanks to the AI. To go along with the hard, back-end stuff, it made a gorgeous UI straight off the bat. Initial tests look good, things were fast, and I had high, high hopes.
The first sign of trouble was when I went from throwing in a couple docs to a couple thousand docs. I've never seen my computer work so hard – every core was max'd out, 32GB of memory disappeared, and I didn't even have enough capacity left to stop the Docker containers doing the damage.
A few hard reboots and some prompts later, and I'd sorted that problem out, but my pipeline kept getting stuck. Fixing this problem would either slow the whole system down to a crawl, cause a bottleneck somewhere else, or just freeze everything up completely. I figured this was a Hard Problem, so I just turned Opus 4.7 on and let it do its thing.
A day later, after I'd burnt through my Opus token allotment, I'd managed to take a mostly-working system and turn it into complete garbage.
Herein lies the most important lesson in using AI as my coding buddy, a lesson I apparently have to keep learning: Know when to stop.
Very, very good; Very, very bad
AI is very, very good, until it's very, very bad. The latest models have managed to successfully push this point out significantly, but the cliff is still there.
The problem used to be caused by attention. Once there's too much information in a working prompt – previous fixes, more files, chat history, etc., the AI can no longer distinguish useful information from junk. Likewise, it can be caused by the context window – it just can't keep all the info it needs in working memory.
Part of the fix has been to compress knowledge as it goes. Something like Cursor is constantly summarising its history, solving that pesky context window problem, but also losing nuance. So solving the context length and attention problem introduces a new problem - potentially losing the pieces of information that really matter.
If you go down a rabbit-hole with one of the big models, you'll see it starting to reimplement fixes that didn't work 30 minutes ago, trying things it's already tried, and getting itself twisted in knots. That, I think, is due to the compression rather than it revisiting old solutions.
Another common issue that leads it astray is building on mistakes that it doesn't think are mistakes. It makes a change, observes the outcome, sees that it has improved the system, and thinks: "More of that!" But just because something works better, doesn't mean that it was the right choice. If I have a sore toe, amputating my leg would definitely solve the sore toe problem. The AI, having chopped off one leg, might think a good solution to me now being somewhat lob-sided, would be to chop off the other, especially since that worked so well with fixing the sore toe issue.
The birth of the code golem
Typically a human brain would pick up on these mistakes, which usually involve logic or false assumptions or a misunderstanding of how things work at a deeper level. But that requires someone (me) reading the code being produced. And let's face it, I'm getting to the point where I'm not bothering. There's just too much being generated to keep up. When things are going well, who cares? When things start going badly, there's already too much code to get a full picture of what's happening. I've created a code golem and it's gone rouge.
By this point, I've missed the point I should have stopped using AI and started coding the old-fashioned way. I've spent a bunch of tokens, noticeably heated up the planet, and used about a year's supply of fresh water, so I really don't want to lose what you've done. I'm going to think that the only thing to do will be to get the AI to fix it.
This is classic sunk cost fallacy. And it is a real cost - this little project cost me $30 in tokens that I really shouldn't have spent; forget the time I personally spent testing, coaxing, prompting, begging and swearing. Or the fact that I had other, more pressing things to do that just didn't get done.
But eventually even the most stubborn person will realise that it's fucked up beyond recovery. It's time to take the L, and either try go back to a state where things were still getting better, or start from scratch.
Read The Fucking Manual
But all is not lost! There is another path, a path that I'd forgotten. First, RTFM – go read all the docs of the things that the AI implemented. I'll need this knowledge to find the logical problems. I might even find that the tools and libraries it happily picked for me were not the right ones.
Next, read the code. This is where it gets painful, because as good as the LLMs are, their code can still sometimes be embarrassing. But there will be good stuff in there, things worth saving. I'll hopefully see the places where it all went wrong, and it's usually the bits where I can see great complexity where there should be simplicity.
On to the refactor. This isn't gardening, this is weeding. I'm going to want to simplify, restructure things into logical order, finding the places where it made poor logic decisions along the way.
I can totally get the AI to help me, but don't give it epic plans – give it short instructions, use autocomplete, and actually read what it's doing for once. Code like it's 2024.
This work is going to suck. It's not glamorous, I'm not building anything new. So I must take what I've done, and write it down, because the things I fix will become a problem again, either in this project or the next. These become Cursor rules I need for future projects.
Finally, remind myself of the lesson of knowing when to stop with prompting and start with coding. Watch for that inflection point where things go from good to bad. I missed it in this project, hopefully I catch it in the next.