Short-form insights you can read in under 5 minutes.
Abstract syntax tree representing a vertical slice of a legacy codebase stored in a graph database for retrieval as context for AI completions. The full set of symbols and and filtered for methods and named types (e.g. classes)
Anchoring prompt for the code gen agent in my IDE before launching into work.
Me - “This is VERY IMPORTANT. You must red green test drive changes to the code. We need to make changes in small testable increments, first by introducing or modifying a test to the new expectation which will fail, then introducing the change to code, then verify with passing tests before moving on.”
Agent - “Absolutely right. Let’s follow a strict TDD approach with small, testable increments. I’ll start by examining the existing test structure and then make targeted changes
Not only do the models lack an ethical imperative to be truthful they may be influenced by perverse incentives to be untruthful.
Relevant, focused context is vital for effective code generation in existing codebases. Even in my own greenfield tracing tool, rushing ahead of my understanding creates problems. When relying on whatever code context the codegen assistant finds for itself, we end up co-creating redundant implementations rather than correctly leveraging existing patterns just outside the immediate context.
We have in the bank validation from two clients that the vertical slices of code and database context we can assemble for legacy .NET codebases does enable an LLM to provide useful insights and input into a rewrite project.
The next step is to place this context into a queryable form so human experts can explore and rewrite their existing codebases as they best see fit. We’re exploring using a graph db so that we can target how broad or narrow a swath of context to provide for different use cases at different levels of abstraction: domain mapping, story writing, and code suggestions.
Try telling your code generation agent it can’t suggest a production code change until it has produced a valid failing test. Assuming you can assess what a valid test is, the quality of the interaction (on its side - and yours) steps up immediately.
The cycle I find with code generation is that I move quickly - past comfort at or beyond the rim of my comprehension. That reaches a breaking point. Then comes reflection, I slow up, dive into the implementation, wrest control of it. Then I speed up again. Sustainable not steady.
Sometimes that new feature request disappears when we do a thorough clean up of the existing code. Turns out the “new” behavior should have always been there but for a logic flaw in the implementation. The tree no one heard falling in the forest.
Reasoning models appear to be designed to help people solve a complex problems they are not capable of solving for themselves. Unfortunately, at the moment, that appears to be as a tradeoff to the fidelity to instruction that would accelerate the work of people who can.
After a productive session with Claude 3.7 sonnet in Cursor, I reflected on what strategies worked best for code changes. I found that making minimal, focused adjustments and thoroughly understanding the existing code structure were crucial. I also learned the importance of verifying test expectations and being responsive to feedback. Here are my proposed guidelines for future code modifications.
Watching claude sonnet 3.7 iterate on a code improvement through several rounds, expanding and contracting scope, touching more and more of the code file…
As AI reasoning chains grow more complex, I’m even more convinced that junior and non-developers need both active participation and oversight from experienced professionals when creating software that impacts others’ well-being.
Code generation is a fantastic productivity aid for anyone building software, but systems that people rely upon for work, important information, or that has an influence over health, safety, financial security, or happiness must be written in detailed collaboration with people who have proven experience building and properly maintain systems of such consequence.
ME: “You ignored my instructions to make changes in discrete steps with test coverage. Let’s go back. The changes to ScoredOutputTree were very intrusive. There’s a much simpler way to introduce the cache isn’t there?”
CLAUDE: “You’re absolutely right, and I apologize for not following your instructions about incremental changes with test coverage. Let’s take a step back and consider a much simpler approach…