llm generated agent.md files are harmful. handwritten ones on average don’t have an effect on ability to complete a task
thread article:
agents keep signing off on commits wrong. we added it to the agents.md and now they don’t
benchmarks notoriously don’t measure the likelihood that the results would be merged by maintainers, just that they passed a test
the old style “repo map” agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly
“Writing code well” includes several relavant things:
Security implications
Stability and edge case handling
Performance
Quality of output (ie. for UI/UX)
Readability (Agents need to read to make changes too! Violating DRY/SOLID/etc. could still cause issues!)
As far as I’m aware you do currently still need a human to ensure stuff like this is followed. People use agents because they don’t care about the above, or because they can get close enough and intervene to fix any issues that appear.
34 comments
log in to comment.
Agents md file doesn’t actually do anything except using more tokens arxiv.org/pdf/2602.11988
I feel like you either didn’t read your article or didn’t read the post article
[deleted]
your research paper:
thread article:
benchmarks notoriously don’t measure the likelihood that the results would be merged by maintainers, just that they passed a test
the old style “repo map” agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly
Agents.md doesn’t exist to make sure generated code functions, it’s there to make sure it’s written well.
[deleted]
Agents for generation, humans for review
[deleted]
That’s just because I need to stop and appreciate my efficiency for 10 seconds every time it saves me 10 seconds :-)
[deleted]
I think you’re correct, the amount of rigor in your prompts seems orthogonal from whether you’re using them for architecture or implementation.
“Writing code well” includes several relavant things:
As far as I’m aware you do currently still need a human to ensure stuff like this is followed. People use agents because they don’t care about the above, or because they can get close enough and intervene to fix any issues that appear.