About

Language model (LM) agents have landed with a splash into both industry and academic research, but claims about agents have proven extremely difficult to validate. Many claims about the (purported) most powerful models and harnesses come from frontier AI labs, who have a vested interest in conveying the power of their agents, as it directly affects their financial status.

This is not an acceptable state of affairs for good science. I myself am an extreme user of frontier agents, but I find the lack of clarity disappointing: I believe that these tools are powerful, but I've found it difficult to verify or validate techniques, and feel inundated with claims from new agent system developers (skills, tools, etc) without rigorous investigation or justification.

I am someone who is familiar with the design and implementation of programming systems, and (as a graduate student) I've worked on the design and implementation of probabilistic systems and languages. Further, I've kept up with the often wild and sometimes inane ideas of popularizers of agents.

With this background in mind, this is a tiny project with the intent of investigating large scale engineering with frontier agents, with careful analysis and discussion.