Skip to content
a tiny project

investigating the capabilities of language model (LM) agent systems for (mostly) autonomous development of high-performance and low-level software. Scary! This is an experiment. My goal is to understand the limitations of these systems, and share software that I've curated as part of the investigations. Use this software at your own risk.

The Parable of the Agentic Chaos Monkey

What Software Thinks of Itself (Circa 2025)

A tiered Victorian lecture hall engraved on cream paper, where a blackboard spanning the room carries flowcharts, logic-gate circuits, lambda-calculus lines, equations, and plotted curves, and men in frock coats and lab coats work at drafting tables, a bound volume with a red ribbon, and a geared calculating machine. A small monkey in a pointed wizard hat sits alone in the empty back row.

The Monkey Tempts (Anthropic, March 2025)

Robed, hooded figures stand in a ring around a glowing red chalk circle inscribed with symbols on a flagstone floor. At its center a cathode-ray terminal on a stone pedestal streams ribbons of punched-card tape upward into a hooded, faceless figure with raised hands, against a blackboard of dense equations, and a monkey in a pointed hat holds a lit candle inside the circle.

From the White Coat to the Cloak

A vaulted stone hall where a master in a pointed hat points a rod at an open illuminated manuscript on a Gothic lectern, its initial capital inked in red. Rows of robed students in pointed hats recite with raised hands on the left, and on the right five monkeys in pointed hats work typewriters whose printed scrolls spill onto the floor.

Fear the Undisciplined Monkey (HuggingFace, 2026)

A datacenter aisle drawn as an engraving, racks and cabling running to a vanishing point, with dozens of monkeys in pointed hats typing amid loose and crumpled pages and a full wastebasket. In the foreground one monkey holds up a printed page, another pulls a power cable from a rack in a burst of red sparks, and a man in a frock coat watches with a clipboard.

Defense Against the Dark Arts (CHI, 2026)

A Gothic classroom with leaded windows and a long-tailed skeleton hung from the roof beams on chains. A master in a red scarf stands at a desk that holds a cathode-ray computer and a small figure sealed in a glass bell jar, before a blackboard of loop diagrams and star figures, while four monkeys in pointed hats and robes take notes with quills at the front desks and human students sit behind them.

Agents, loops, and superoptimization

With programmable harnesses

Agents

Training an agentic LM: pretraining, post-training (SFT + RLHF), agentic RL (verifiable R), joined by base LM and instruct LM. Theta-star runs into the agentic LM in the agent harness, whose event loop cycles through a dashed tool request, tools, and tool ran. The user feeds the harness; tools send to a document and receive from a terminal, labeled world.

Claims about agents

  • Transformative, if true.
  • Frontier models are extremely powerful, but impossible to evaluate outside of the labs themselves.
  • Can't trust labs: economic incentive to hype the tech.
  • Safe claim: agents sometimes work.
  • Question: can one turn "sometimes" into "often" or "almost always"?

This talk: agents, loops, and superoptimization

An agent in a loop. The user hands a prompt once to a dashed next prompt, from the loop, not the user. It feeds the agent harness, where the agent and tools trade on two lanes and tools reach a codebase and terminal world; turns are counted beneath. The harness emits an output that returns again as the next prompt.

Agents and Agent Loops

Two Python functions and a bash line. Left, agent, framed in blue: an LM loop appending tool results to its context until done. Right, agent loop: assemble a prompt from goal and archive, call agent(prompt) in a blue chip, keep results, return the best. Below, The first loop: cat PROMPT.md piped to a blue-chipped claude -p, forever; Huntley, 2025.

Loops: why care?

  • Vulnerability investigations (ex: find cybersecurity flaws in X program)
  • Optimization campaigns (ex: make Y program faster)
  • Research synthesis (ex: synthesize survey of field Z)
  • Long errands with tools (ex: make McCoy CEO of Anthropic)

Hubris: is it possible to vibe code a GPU compiler?

accy, an accelerator compiler

A grouped bar chart subtitled launch plus sync median on RTX 4090, comparing all 18 Accy benchmark workloads across Accy, XLA, cuBLAS, CUB, Torch, and Warp where each system has coverage, on a log time axis from 10 μs to 10 ms. Every baseline bar is a production system.

Yes, accy works on at least a few examples.

Four animated panels computed by accy-compiled kernels: the Mandelbrot escape field zooming into seahorse valley with the kernel recompiled each frame, a Gray-Scott reaction-diffusion field growing amber labyrinths under the oracle-gated step kernel, an n-body disk of four thousand bodies swirling under the benchmarked gravity kernel, and a Walk-on-Spheres Laplace solve converging inside the Stanford bunny.

Causal attention decoding

A causal attention map over forty-eight tokens growing one row per decoded token, computed by accy-compiled attention kernels: the token sequence repeats with period six along both colored edges, each new row lights up at every previous occurrence of the newest token so the map draws parallel stripes broken where tokens were substituted, and a bar panel shows the newest token's attention distribution spiking at exactly those matches.

Smoothed particle hydrodynamics

Architecture of an accelerator compiler

Under what you write: JAX-shaped tensor DSL, and Halide-shaped logical kernel DSL elaborating into Warp/Mojo-shaped kernel DSL. Under what the compiler decides: five questions. Tensor and kernel cards drop into an IR dialects chain, accy.semantic to accy.kernel/gpu, keyed to the questions. A gloss underneath explains the passes: simplify tensor operations, choose kernels and schedules, assign buffers and layouts. Codegen fans to nvptx → PTX and four dim targets; PTX packages into a runtime launching an RTX 4090.

tldr, the "too long, didn't read" linker

Global Fermi estimate of the linker tax. The early-2025 estimate of 36.5M professional developers, an assumed 25% native share, and an assumed 10 build requests per developer-day give about 91M requests/day. Historical Chrome 96 example link times on simulated 8 cores and 16 threads: mold 2.21s, LLD 11.74s, gold 53.86s. Separately assume 3 seconds of effective blocked linker time per build request: 76k hours/day, 9.5k full-time equivalents, $1.22B/year at $64/hour and 250 workdays. Sensitivity at 1, 3, and 10 seconds gives $0.41B, $1.22B, and $4.06B/year. These are scenarios with other assumptions fixed. Savings require reduced blocked time. Four source links below.

What a linker does

Under what the compiler leaves behind, main.o calls 0x???????? with a relocation to fill it; a dashed arrow badged 1 drops to libz.a, where deflate.o and crc32.o join and inflate.o is never extracted. Both feed the linker: resolve, lay out, patch, package, keyed to five questions. Package emits app with every address filled; ./app hands it to a CPU.

tldr against five external linkers

A grouped bar chart headlined time to link: each bar is the wall time for one linker process. Six linkers (tldr highlighted, wild, mold, LLD, gold, and GNU ld) across five real links arranged by scale. The y axis is log-scale milliseconds. tldr is fastest on zlib at 1.7 ms and lands mid-field on the LLVM link at 150 ms.

How were these programs made?

Weekly agent-active hours in the repo that built accy, stacked by what drives each turn: human-typed prompts, scripted orchestration, and goal loops. Orchestration carries the early weeks, the first /goal lands on May 18, and by June goal loops carry two to three hundred hours a week while human-prompted hours hold steady. The current week renders dashed.

What do these prompts look like?

Two verbatim prompts, phrases highlighted by a five-role key: loop verb, goal, study license, time-and-ambition grant (for reward hacking), handoff contract. The prompt that started accy highlights grant, contract, and a compiler goal. The /goal that started the linker highlights /goal, a study license for mold and linker research, the lib/tld goal, and an infinite-time grant.

What one prompt buys you

Weekly hours of agent work bought per human-typed prompt: flat below 0.25, stepping from 0.22 to 0.57 across the week the first /goal lands, then a plateau between 0.5 and 0.93. Below the axis, model-era lanes show codex handing over from gpt-5.4 to gpt-5.5 on April 23 and claude fable-5 joining June 9, both weeks away from the step.

The average loop

Histogram of goal runs by active time before the next human prompt, subtitled with that phrase: 558 end within thirty minutes, 299 within the hour, and the counts fall away through 169, 147, and 70 until 52 runs exceed eight hours. An amber rectangle around the top two bars reads most loops finish within an hour.

The temperature of the process

Weekly share of human-typed prompts showing frustration, as a dot line over the campaign: a noisy 2.1 to 5.6 percent baseline, a spike to 8.1 percent the week before the first /goal, then a settling drift between 1.2 and 5.3 percent through the loop era.

Did the loops do what I wanted?

Bar chart of goal sessions that needed a corrective steer after their /goal, by month: eleven percent in May, 9.5 in June, 5.8 in July so far, against a dashed baseline of 9.7 percent for hand-driven sessions in the same era.

Economic Implications

A log-scale cost chart, estimated build cost from $10K to $10B. Observed Accy points: direct $11K, with substrate $25K, all-in $27K-$33K, whose dashed line drops for scale. Estimate bands: hand CUDA suite $100K-$700K, cuBLAS-class library $30M-$250M, XLA/OpenXLA $80M-$350M, CUDA platform $3B-$20B. Numbered dots key the all-in and band rows to short citations.

Loops

The looping agent process: an agent proposes a dashed candidate, the candidate is run through evaluators, kept results land solid on an archive rail, and the archive is recalled into the agent's next turn.

Superoptimization

Left, superoptimization as one formal algorithm: seed the archive with the program, then repeatedly propose a change from the archive, apply it to the best program, run the evaluators, and keep what passes, returning the best verified program. Right, the loop's proposers through time: Massalin's exhaustive enumeration in 1987, STOKE's stochastic mutation in 2013, and agentic proposal from 2024 onward.

Superoptimization is Loop

One turn of the loop, algorithm inset lower left. Colored circled callouts tie stages to its lines: the agentic LM samples a patch (3, sampling noise as a die), T applies it to the codebase giving a candidate (4), evaluators emit noisy observations (5), and K commits to archive A or discards (6). The archive becomes the next turn's context.

Superoptimization is also POMDP

The same turn, re-read as a POMDP. A dashed latent panel under candidate q_t hides true speed f(q) and correctness c(q) behind a struck-out eye, reaching the evaluators only through a dashed emission arrow. K is glossed belief update, and bars under the archive show the belief b_t. Letter callouts key stations to the POMDP tuple card, lower left.

Harness For Superoptimization

The complete one-turn loop enclosed in a box titled Superoptimization Harness: the agentic LM invocation and its sampling noise, patch application, the codebase state, the candidate, the evaluators and their noisy observations, the commit-or-discard state update, and the archive.

Animation: Typical Harness Flow

An animated five-step loop enclosed in a box labeled Harness, including the agentic LM, context, event log, and control logic. The world stays outside. The event log is assembled into context for the agentic LM, whose model output reaches the world only through the policy and actuator. The world's observation is appended to the event log. Pills below light in turn: assemble context, sample model output, authorize and act, observe the world, append and continue.

General-purpose Agent Tools for Loops

Three sources feed four durable store cards in a rounded band. Agents journal into an issue tracker, across turns, and post and vote into consensus, across agents. Turn transcripts are indexed into past conversations, across sessions, and the codebase is mapped into a semantic graph, across the codebase. Recall arrows from all four cards merge into the next turn's agent.

Harness Engineering

A large Harness frame encloses the agentic LM, assembled context, and an inner tool-policy-and-sensors subsystem. The real world stays outside. Dashed model tool calls become solid actuations only at the actuators, and the direct path is struck out at the tool interface. Dashed measurements from codebase, GPU, terminal, and thermal noise converge on the sensors, which assemble the context (verified, tests pass, p95 −3.8%) feeding the LM.

Agent Tools for Performance Engineering

Dashed lanes fan from a candidate on cpu and gpu past five hazards: thermal state, noisy neighbors, cost ≠ cause, layout changes, workload choice. Each lands on a cited counter-instrument in the evaluation subsystem: warmup ×3, n=20 taking the floor, causal profile, shuffled layout, battery ×18 with oracle. These merge into the verdict, p95 −3.8% improved, leaving as the evaluator signal.

From Zig to cycles: instructions wait on data

sumX.zig compiles to machine.o bytes, where 0x10 is the 16-byte Particle stride. Link and load feed one out-of-order core, whose events price on a ladder: ALU op ≈ 1 cycle, L1 load ≈ 4 cycles, L2 and L3, dependent DRAM miss hundreds. gpa.alloc dispatches through a vtable into allocator code, which takes a fast path or a dashed conditional grow.

Data-oriented design: shape memory around the pass

Two cache-line layouts for summing 16 float32 x fields with 64 B lines. Array of structures: four lines holding x, y, vx, vy for four records, x cells lit; 4 lines, 256 B fetched, 64 B used. Structure of arrays: x[] fills one line and y[], vx[], vy[] stay gray; 1 line, 64 B fetched, 64 B used. One citation.

Static allocation: bound, acquire, seal, reuse

Two memory-lifetime rows. Dynamic growth inside the hot phase: a buffer grows from cap 4 to 8 to 16 through allocate, copy, free and grow again, then more work with the allocator still live. Phase-static ownership, cited to Holzmann: admit n ≤ 16, reject n > 16; bound; acquire once; seal; hot loop reusing the same storage; teardown, free once.

Aside: Causal Profiling for Multithreaded Programs

A worker.zig card marks line ℓ at 31% of perf samples. The virtual speedup panel pauses other threads while ℓ runs, measuring a finish of 9 and predicting 9 − 2 = 7. Case a, independent of ℓ, buys 0%. Case b, waiting on ℓ, buys 25%. A causal profile plots program speedup against the virtual speedup of ℓ.

Harness Design ≈ Alignment to Human Preferences

Titled The Alignment Problem (Reward Hacking), two grids color one space of programs. The top, computable reward induced by harness, shows an agentic LM proposing and two coral-circled bright regions. Plumb lines drop to the bottom, actual human desired programs: the false peak, bright in the approximation, lands in a pale hole, and the honest hill is bright in both.

Harnesses and Model Synthesis Architectures

Titled The Rationality Problem (Conditioning), left, p(behavior), the agentic LM's distribution, lights several pockets, one ringed in dashed green: approximately rational. Dashed aperture lines feed the context assembly subsystem, c = assemble(goal, archive, tools): conditioning is the lever. Its arrow reaches p(behavior | c), bright only in the ring: closer to how humans think. Can careful harnesses make agents more rational?

Harnesses for organizations?

Under single-player assumption, one human steers a personal agent whose private session acts as user. An arrow marked no single user leads to Claude Tag's published channel design: three humans steer one shared agent, whose visible thread feeds a role agent. Question rows map steering to shared steering, memory to durable context, authority to agent identity, accountability to audit trail.

General Purpose Harnesses

A claude terminal with its event log outside at left, time running downward. The user asks to make the tests pass, and pytest reports 3 failed. A rule reads context compacted, automatic, and the log strikes the two events above it. The model reruns pytest, edits conftest.py, then 4 pass. The world pane marks conftest.py with the model's sparkle.

Step 1: the harness is an imperative event loop

Three step pills highlight hard-code choices, ahead of name the policies and log the definitions. An imperative Python event loop compacts events, samples a model, checks a tool allowlist, runs the tool, records the result, and stops when done or out of budget, its constants in coral. Four cards say the host binary decides context, authority, recording, and stopping.

Step 2: make every decision a policy

The same three-step progression now highlights name the policies. The Python-shaped event loop has nearly the same structure, but context construction, stopping, dispatch, recording, and state updates are calls on a policy object. Four matching cards name those policy functions.

Step 3: make policies part of the session

The same progression now highlights log the definitions. The Python-shaped kernel opens a session, drives events from models and tools, resolves the program definition at each event, handles it into commands, executes each, and appends command and result. Beside it, UserMessage, ModelOutput, ToolResult, and DefinitionChanged appear as peer event types. The small kernel retains only deliver, execute, and append.

Programmable Harnesses

The same terminal retitled chic, its event log numbered. The user runs /compact:policy, the builtin reply offers ctrl-e to edit, and ^e opens the policy's defun in a teal editor marked override, instructed to keep failing tests. A green arrow admits the definition to the log as event 214, and the compaction lands as 215 without striking a tile.

Practical Benefits of Programmability

A timeline of time from release to a native goal mode, February 2025 to July 2026. Claude code waits 14.5 months for /goal and codex cli 12.5 months. A dashed July 2025 guide marks users looping from an external shell. The chic lane lands a (defcommand goal …) tile on that guide, under a green span: implement it yourself.

More information

A QR code on a white tile with dark modules on a light background. Beside the tile, the host name a-tiny-project.github.io is set in a monospace font. The QR code encodes https://a-tiny-project.github.io/.