Case study · n = 1
A retrospective trace study of evolving requirements and coordination in human-AI visual design
The user1, claude-opus-5-52, claude-fable-5-22, gpt-6-astra3
1 human, video artist and media technician 2 language models run in Claude Code 3 language model run in Codex CLI
Counted, coded and written by claude-fable-5-2 with helper agents, at the request of the user.
Affiliations
Not affiliated with Anthropic. Commissioned by the user it describes.
Not affiliated with or endorsed by Anthropic. Commissioned by the user it describes.
AbstractThis study traces from session logs how one user and AI coding agents turned a commissioned strawberry animation into a procedural fruit add-on and an intro film in two weeks. Counted by script, the user sent 3,958 prompts; an agent typically made 2 tool calls before the user wrote again, and 74% of the user's prompts arrived while an agent was working. Recounted from the logs by models, with episode lists: in at least 20 episodes an agent's check or reassuring report was followed by a fault the user found by looking; in at least 24 an agent reported a false status; in at least 37 the user stopped or objected to work the user could not see or had not asked for; messages reached the wrong agent at least 17 times. These are selected recorded cases, not failure rates, because correct checks and reports were not counted. Coded by models only, with no human coder, and less firm: requirements developed through proposals, results and failures and were not fixed in the first request. On the 60 episodes both coded, two model coders rank the sides the same way in all eight roles: the user more often requested, proposed, judged by looking, decided and stopped; agents more often built, measured and named causes. The coders disagree on many single episodes, with agreement beyond chance only minimal to weak for most roles, and on some counts, so this ordering describes two codings, not a measured split. These counts give no shares of the result, and miss some user work and agent design choices. This is one case without a control run, coded and written by models of the agents' own families and commissioned by the user, who offers the add-on to paying supporters ().
Every day of the project, with all 265 episodes, pictures and videos, is in Appendix A at the end of the study. Episodes linked in the text open as a short card in place; its link leads to the episode in Appendix A.
1Introduction
On Nov 5 at 10:19 the user sent a screenshot of a panel in Blender and asked for two buttons: one to create a strawberry and one to randomise its shape. He also suggested computing the inside of the fruit only when a cut needs it, with a toggle to switch it on. The agent called the toggle better than its own automatic plan. It pointed out that everything so far was loose scripts, and it proposed one switch with three settings: Off, Fast and Exact. At 10:24 the user asked whether this was a plugin or an add-on. The agent answered that it was only scripts and that an add-on was the right way. The user decided: “yes let's build the add-on.”
In these 28 minutes the user requested, proposed, looked, decided and stopped a test. The agent proposed, measured and named the cause of a slow step. Who made the decision depends on which of these acts one counts as making. This study is about 265 episodes like this one.
Where the question comes from. Eight days later the add-on and an intro film were released. When the user posted the result on Reddit, commenters disagreed about his part. Some said he contributed little or nothing and the AI did the work. Others said that giving orders is not making. The title comes from that dispute, but this study does not settle it. Whether directing, judging and choosing count as making is a matter of definition, and the logs test no definition. What they can show is how the work was organised: how the two sides took turns, where the work broke down, how the goal changed, who proposed, built, checked, kept and stopped what, and what was reused.
The project. It began on October 31 as a paid job: animate a strawberry for an ad. The user went on with a new goal: a strawberry built only from nodes, with no downloaded model and no textures. Nodes are Blender's building blocks that make shapes and surfaces from rules. The inside should follow the anatomy of a real strawberry when it is cut. Two weeks later an add-on and an intro film were released. The add-on included six fruit generators at different stages: the strawberry was the most developed, the kiwi was working or nearly complete, and four others were marked as work in progress. Anatomical plausibility and visual quality were judged by the user and not rated independently.
The record. The project left unusually full logs. What survives holds the user's prompts (messages), the agents' tool calls and more than a thousand screenshots. The analysis used extracts of these logs: the logs of helper agents were read only in part, and one event file that some checks need is missing (method tab, ). The logs do not hold what the user did by hand in Blender or what the user thought (). Each claim in this study links to the moment in the work that it rests on.
What the study found
Each claim says what kind of evidence it rests on: a count taken from the logs by script, a recount by a model with a list of episodes, an exploratory coding by models, or a proposed explanation. The chapters follow roughly that order, but the kinds belong to single claims, not to whole chapters.
The study asks four questions:
- The analysis is being written.
What kind of evidence this is. Five things hold for every number below.
One case. One user, one local workspace, two weeks, no control run; rented cloud machines and Blender instances shared between agents were used too. The record can show that something happened in this project. It cannot show that it had to happen, or what either side would have reached alone.
An uneven record. The logs hold the agents' work action by action. The user's work is mostly thinking and looking, and it is logged only when he acts on it. The small design choices inside the agents' tool calls are not counted either. says for each kind of contribution which side a naive reading would favour ().
Two sides, and others. The study speaks of the user and the agents. Others took part and are not counted: the makers of the harnesses and models, the user's standing setup of rules, skills and terminal, and the generators for pictures and music ().
Kinds of evidence. A script counted the messages, tool calls and stops without reading any text. That removes one kind of judgement, but it does not check the extraction, the exclusions or which session a message belongs to; the problems found with the extraction are listed in the method tab, . Five findings were recounted from the raw logs by models and come with a list of their episodes. They are readings that can be traced, not measurements. Everything else was coded or counted by models of the agents' own families, at the request of the user, who offers the add-on to paying supporters. No human coded anything, and where two coders coded the same episodes, they agreed only minimally to weakly on the codes of single episodes for most roles (). So this is not neutral ground ().
Counts of recorded events. A count says in how many episodes something was recorded. It is not a rate, because checks and reports that were right were not counted. It is not a share of the result, because nothing in the logs converts one role into another.
Out of reach for this record are each side's causal share in the result, whether any detour was necessary, a productivity gain beyond this case [12, 13], a comparison of model abilities, an independent rating of artistic or anatomical quality, and a legal finding on authorship ().
Three ideas from design and work research guide the reading. In design, problem and solution often develop together, called co-evolution [39]. In collaborative work, dividing tasks, keeping versions aligned and making results visible is work of its own, called articulation work [51, 52]. And thinking can be spread over people and things such as screenshots, reference pictures, scripts and the viewport [53]. The study uses these ideas to decide what to look at, not as hypotheses it tests.
How to read this study. shows the project at a glance and follows one morning, which introduces the units and the figure type used later. and rest mostly on counts and recounts, and mostly on coding, and each claim says which kind it rests on. Each chapter opens with its finding and its limit. says what was counted and who coded. lists the limits. Episodes linked in the text open as a short card in place, and all 265 are in . How the study itself was made, who set its questions and which corrections it went through, is told in the tab “How this study was made”.
2The project and one morning
The project ran for 14 days and went through five stages. One morning, on Nov 5, the sixth day, shows how the work went: who wrote, who built, who checked, and how this study counts it.
2.1The project at a glance
2.2One morning, step by step
This morning was chosen because it is easy to follow and because one idea from it returns a week later. It is not a sample: it went better than many others. It introduces the four units a reader needs: the prompt, the episode, the role and the stop.
08:28, a symptom. On Nov 5 the user found that scrubbing through the timeline was still slow. Even with the cut switched off he could not drag smoothly. The agent admitted that it had wrongly reported that the fruit no longer recomputed, and it measured: without the cut a frame took 21 milliseconds, with the cut about 5 seconds. Each message of the user is a prompt. Everything an agent does between two prompts is a run.
08:48, a proposal from each side. The agent suggested computing the whole fruit and the fully cut fruit once, so that the moving cut line only decides which part is visible, and asked whether it should test that. Four minutes later the user asked for a fast mode that shows the cut only roughly. The agent built its mechanism under that name. Two snags took most of the half hour; in the end a frame took 70 to 85 milliseconds instead of about 5 seconds. Such a stretch of work on one task is an episode. Each episode is coded for eight roles, such as proposing, implementing and judging. Here the proposal is coded for both, because each side has a recorded proposal of its own.
10:19, a decision. The user asked for buttons in the panel and suggested computing the interior only when a cut needs it. The agent proposed one switch with Off, Fast and Exact and said that an add-on was the right way, and the user decided to build one. goes through this episode role by role.
10:46, a choice between variants. The agent recommended prebuilt pieces (variant A). The user chose variant B, which keeps the finished parts and recomputes only a narrow strip around the cut line, and he judged that its edge did not look worse. The test gave about 2.5 seconds per frame instead of 5 to 6, well short of the agent's estimate of 0.5 to 1 second.
Eight days later, the idea returns. On Nov 13 an agent measured frame times for the final render and found a jump at frame 691, where 70 falling pieces were recomputed for every frame although their shape no longer changed. It computed the pieces once and froze them, and a frame took about 9 seconds instead of about 54 ().
So one morning holds many prompts and runs, four episodes, and one code per role in each episode. The fourth unit is the stop: the user interrupts an agent that is still working. He did so in the first and the third of these episodes.
TimeThe userAgents
Nov 5 · Morning: scrubbing is still slow
08:28judged by lookingscrubbing through the timeline is still slow
judged by measuring21 ms per frame without the cut, about 5 s with it
Nov 5 · A fast mode, from both sides
08:48proposedcompute the whole fruit and the cut fruit once
08:52proposeda fast mode that shows the cut only roughly
implemented, then measured70 to 85 ms per frame instead of about 5 s
Nov 5 · Decision: this becomes an add-on
10:19requested, proposedbuttons in the panel; interior only when a cut needs it
proposedone switch with Off, Fast and Exact; an add-on is the right way
10:24kept or dropped“yes let's build the add-on”
Nov 5 · Variant B: recompute only a strip
10:46proposedrecommends prebuilt pieces, variant A
kept or dropped, judged by lookingchooses variant B; its edge does not look worse
judged by measuringabout 2.5 s per frame instead of 5 to 6
Nov 13 · Frozen pieces make renders six times faster
21:31requestedasks how long a frame takes
judged by measuring, named the causea jump at frame 691: 70 falling pieces recomputed for every frame
implementedfreezes the pieces: about 9 s per frame instead of about 54
3Sources, units and blind spots
The study reuses logs that existed before it began. A script counted the units. AI agents cut the work into episodes and coded them. This chapter says what was counted, who coded, and where the logs are blind.
2.1Setup and sources
The user said what he wanted, did some work by hand in Blender and judged the results by looking, and the agents wrote and ran the code.
The user. A video artist and media technician who has made video for theatre and opera since 2011. He studied biology for some years, then camera. He has written code for media work for years and has built software products with AI agents since 2025. In this project, by his own account, he wrote no code and built no node trees by hand.
The user said in plain language what he wanted, mostly by dictation. He sent screenshots of the Blender viewport, the window that shows the 3D scene. The agents wrote Python scripts and ran them inside Blender. They looked at their own screenshots and reported back. The user worked by hand in the same Blender file: camera, light, materials, and looking.
Sources. The study uses the session logs of both harnesses, the git history, the user's screenshots and the project folders. A harness is the program that runs a model and gives it tools. The git history records each version of the code that was committed, that is, saved to it as a version. Nothing was recorded for this study: the logs already existed. The user writes and dictates in German, so quotes are translated.
: which model on which day
2.2What is counted
A reader needs four units. The first and the last were counted by script, straight from the logs and without reading any text. The episode and the role are codes: a model read the chat and assigned them.
- Prompt. One message the user sent: typed, dictated or only a screenshot. There are 3,958. They include the messages he sent while an agent was still working. The harness stores these apart from the conversation. They also include the 120 messages that hold nothing but a screenshot. The 158 messages passed on from other agents are left out. An entry stored twice is counted once. About 18,000 of the 366,000 characters counted as his prompts were written by agents ().
- Episode. A part of the work on one task. A reading agent, an AI agent that read the chat afterwards, cut each day's chat into episodes (). There are 265.
- Role. What a side did in an episode: proposing the approach, requesting the work, implementing it, naming the confirmed cause of a fault, judging the result by looking, judging it by measuring, keeping or dropping it, and stopping running work. Each of the eight roles is coded for the user, an agent, both, none or unclear (box in ).
- Stop. The user interrupts an agent while it is working. There are 246, 242 of them in Claude Code. 244 fall into 243 runs that began with a prompt, because one run was stopped twice. The other 2 fell into runs that began with a message from another agent. In Codex CLI the user steers by typing while the agent is still answering, and its log holds only 4 stopped answers in 388 prompts. Agents also ended work themselves (64 stop-task calls); those are not counted as stops.
All units, with their counts
- Tool call. One action of an agent, such as running a script in Blender or reading a file. Status reports to the terminal are left out. The main sessions made 17,453. Of these, 16,008 followed a prompt of the user. The other 1,445 followed messages from other agents or came before a session's first prompt. With the 3,440 calls of helper agents (see below) the total is 20,893.
- Session. One conversation with one agent in one harness. There are 42.
- Helper agent. An agent started by another agent. Agents in Claude Code started helpers 31 times, and 30 helper logs exist; they made the 3,440 tool calls above. Two other counts of the helpers exist and do not fully match (method tab, ).
- Run. Everything an agent does between two prompts of the user.
- Turn. Everything one agent does after a message until it hands back: its model responses and tool calls. An answer, used for tokens in appendix D, is one model response inside a turn.
- Stretch. The time between two messages of the user in one session. A stop counts as a message, a post from another agent does not.
- Model, agent, helper, harness. A model is the language model. An agent is one running session of a model with tools. A helper is an agent started by another agent. A harness is the program that runs the agent (Claude Code or Codex CLI).
Why the picture counts differ
Several counts of pictures appear in the data, because they count different things with different extractors. This page uses the study extractor's count of prompts with a picture; the pilot in uses the day-by-day check's extractor.
| Count | Value | What it counts |
|---|---|---|
| Prompts with a picture | 794 | Study extractor (messung.json), used for the totals on this page. |
| Image prompts and images | 787 / 869 | Day-by-day check extractor: prompts carrying an image, and the images in them. |
| Images sent, summed per day | 877 | Day records (tage.json), used in the day-by-day tab. |
| Screenshot files in the window | 1,104 | Screenshots taken on the user's machine in the two weeks, sent or not. |
| Older extraction | 713 | An earlier count in technik.json; not used on this page. |
3.3Who cut and coded
Episodes. AI agents read each day's chat and cut it into 265 episodes. Each episode got one of six labels (box). Sorting episodes into such fixed categories is called coding here. The episodes shown on this page were read from the complete chat. They are the second of two readings; the first had worked from a copy that lacked about half of the user's messages (method tab, ). The cut into episodes was made once and was not repeated by a second coder, so every count per episode depends on it; fixing the units is a coding step of its own that needs its own check [54].
Roles. One coder, a model of the agents' own family, then coded every episode for eight roles, an outcome and a scale, with a codebook written beforehand (box). A second coder of another model family coded 60 sampled episodes again without seeing the first coding. The two agree only minimally to weakly on most roles (; method tab, ).
No human coder. All labels and codes were assigned by models. How well a model codes varies from task to task, and its errors can lean in one direction, so coding by models needs a check against human labels [55, 56]. For simple categories on selected short texts a model has done better than crowd workers [57], and with a codebook written by experts a model reached kappa 0.38 and 0.61 against the experts' codes [58]. Roles in long work logs are not such a task, and the check against human labels is missing here ().
The six labels
went wellMuch became visible in a short time, or the user said he was satisfied. decisionA direction or a method was chosen or changed. Not defined in their instructions. routineOrdinary work that fits none of the other labels. Not defined in their instructions. misunderstandingThe agent did something other than what the user meant, or the other way round. dead endWork was thrown away. stuckA problem did not go away over many attempts.
The four definitions without a note are the rules the reading agents were given, together with the order not to embellish and not to dramatize. “Decision” and “routine” were only named in their instructions, so their meaning here is read from how the agents used them.
How roles were coded
Limits of the role coding: the threshold of three image reads for an agent's look was set, not calibrated. The main session of an episode is the one with the most prompts. Image views in Codex CLI are missing from the log pass. A role is coded per episode, not per event. “None” for keeping or dropping means that nothing was recorded, not that nobody decided.
Analysis. A further agent read all episodes, looked for patterns that repeat and counted them while reading, not by script. After a day-by-day check against the raw logs, five of these counts were recounted from the logs with a list of the episodes they include. The others still rest on the episode texts (). Each finding links to example episodes.
2.5What the logs can and cannot see
The logs record the agents' work much more fully than the user's. An agent's work is logged action by action. The user's work is mostly thinking and looking, and it is logged only when the user acts on it. Science [1, 2, 3], copyright law [4, 5], automation research [6, 7, 8, 9] and software research [10, 11] have met the same problem. is this study's own working framework, informed by the writings of these fields. It sets out seven kinds of contribution and, beside each, what the logs offer, where they are blind and which side a naive reading would favour. The sources do not validate these indicators or these directions. A last row on third parties is this study's own proposal.
Three rules follow. First, time is not treated as work. For the user the page shows only the gaps in which no agent turn was active, because thinking, looking, hand work and absence look the same in the logs. Second, roles are not converted into one another or into a share, because the literature sets thresholds for credit but offers no exchange rates (). Third, responsibility is not given as a number, because science and publishers assign it to humans by rule.
Two blind spots matter most. What the user did by hand in Blender left no log; that he wrote no code and built no node trees is his own account. And the coding agents never saw the pictures he sent, although about one prompt in five carried one. Two gaps were found later: some agent text is counted as his prompts, and the 33 helper agents were read only in part ().
3.5Who else took part
The study speaks of two sides, the user and the agents. Others took part too, and their part is not counted. Some were in the background of every episode, such as the harnesses, the models and the user's standing setup. Others took part only where they were called, such as the generators for pictures and music.
- The makers of the harnesses and models. A harness is the program that runs a model and gives it tools; it also comes with instructions of its own, which were not read for this study. What a model knows about Blender comes from its training. Neither shows in the logs as a contribution of its own.
- The user's standing setup. The agents ran in a tiling terminal that lets them message each other and claim ports, Blender instances and files (), and they kept rules in memory, skill and rule files (). Which of these existed before the project, and what they asked the agents to do, was not examined.
- Sources and generators. Agents read papers on fruit anatomy and measured photos of cut fruit. Reference pictures came from an image generator and the music from a music generator (, ).
So “the agents did” in this study always means the agents as their makers and the user had set them up. An act that the user did not ask for is not yet an act on the agents' own initiative.
4How the two sides took turns
The two sides took turns closely: an agent typically made 2 tool calls before the user wrote again, and three in four of the user's prompts arrived while an agent was still working. A script counted the numbers of the cadence table from the logs, and no model read or judged anything for them. Other parts of this chapter rest on readings by models: the working habits come from the episode texts, how the longest stretches of work alone began and ended was read in the chat, and for the comparison of fruits an agent chose which session belongs to which fruit. The counts show when messages and tool calls happened, not what the user was looking at.
4.1How often did the user write, and how often did he stop an agent?
The stop rate alone says little about how tightly the user led. A vendor's report on its own usage data, which is not peer reviewed, finds that experienced users of the same harness interrupt in roughly 9% of turns, more often than new users [59]. Its unit is the turn, not the prompt, and its users and tasks differ, so the two figures cannot be compared. The rate of 6.8 stops per 100 prompts neither shows nor rules out unusually tight control, and it says nothing about the quality of the agents' work.
: message cadence per day
: when the user stepped in, and how long runs were
How long the agents ran alone
How message cadence was counted
4.2How long did agents work without a new message?
: the fifteen longest stretches
That agents rarely worked alone for long in this project does not show that they could not. An outside benchmark of long software tasks reported in its first version, of March 2025, that the most capable models tested completed with 50% success tasks that take human experts about 50 minutes, and that this length had doubled about every seven months [60]. That is the length of a task for human experts, not how long an agent can work without a new message. It does not show whether these Blender tasks could have gone with looser supervision, and it says nothing about finding the goal.
How work alone was counted
4.3Working habits, and work the user could not see
The habits below are described from the episode texts, so they are a reading by models. One count among them was recounted from the raw logs with a list of its episodes: in at least 37 episodes on 12 of the 14 days the user stopped or objected to work he could not see or had not asked for. A field study of developers working with agents on open issues found that those who went step by step did better than those who gave one large request [61]; that fits the first habit and does not show that it was needed here.
4.4Two talks about working together
4.5The strawberry before the manifest, and the three fruits after it
Did the user work differently with the fruits that were built after the manifest of Nov 6? A script counted prompts, picture messages and stops per fruit in the raw logs. The strawberry took 1,011 prompts over two and a half days. Each of the three later fruits took 23 to 46 prompts in one afternoon and evening. The share of prompts with a picture stayed about the same: 27% for the strawberry, 20% to 26% for the later fruits. This is a comparison inside one case with strong confounds. It is not a speed test and not a control run.
At 15:06 on Nov 6 the user asked the add-on agent for a short manifest on how an agent builds a new fruit. The agent wrote a checklist of about 90 lines. From 15:57 fresh agents followed it for cherry, pineapple and pomegranate, each in a session of its own and each started with a brief that the add-on agent had written. The user then looked in on each of them: he asked for reference images, compared the fruit with them in the viewport and asked for changes to the look.
In the hours of the three builds the user's attention was split. From the manifest to midnight he sent 281 prompts to the sessions of this project; 99 of them (35%) went to the three new fruits, most of the rest (169) to one session that ran a juice simulation, renders and further work on the strawberry. Before the manifest, 1,011 of 1,449 prompts (70%) went to the strawberry. Per active hour the strawberry sessions got 26.5 prompts, the three later fruits 9.3 together (6.8, 14.9 and 7.4). Stops are too few after the manifest to compare: 6 in 99 prompts (0, 5 and 1). The strawberry's Claude Code sessions had 55 stops in 637 prompts (8.6 per 100). Its Codex session has no comparable stop line, as in .
What the counts can show is two ways of working in the same project: days of close turn-taking on one fruit, and fresh agents led by a written brief while the user looked in. What they cannot show is that the manifest caused the difference, or that a later fruit needed less of the user for the same result. The confounds are strong. The later fruits reused the add-on, its code and the rules learned on the strawberry, and the agents could read them. The user and his setup had three more days of practice. The later fruits are less finished: their build sessions ended that evening, and later work on them in shared add-on sessions is not counted here. A cherry is a simpler form than a strawberry with seeds and an interior. And while they were built, the user worked on other things at the same time.
How the fruits were counted
The script lesarten/vorher_nachher/zaehlen.py reads the Claude Code and Codex session logs of Oct 31 to Nov 13 once, in one process, and writes only counts. Every log line counts once: a resumed Claude Code session repeats earlier lines with the same identifier, and a Codex item counts once by its time, role and opening text. Helper agents are left out, as are the sessions that the study excludes as unrelated. A prompt is a message the user typed, also one queued while an agent worked; the same text again within 15 minutes in one session counts once. Briefs, peer messages and summaries written by agents are left out by the same rule as in , so the start brief of each later fruit is not a prompt here. A picture message is a prompt with at least one attached image. A stop is an interruption line in a Claude Code session. Active time uses the 30 minute gap of . The fruit sessions were chosen by an agent from the session list and a few prompts of each, before the counts were made: four sessions for the strawberry (one Codex, three Claude Code), one each for kiwi, banana, cherry, pineapple and pomegranate. The strawberry sessions also hold some work on fitting kiwi and banana into the add-on. Over the whole two weeks this script finds 3,498 prompts, 718 picture messages and 223 stops in 36 sessions, against 3,958, 794 and 246 in 42 sessions in . The event file behind those totals is missing, so the difference is not resolved; the numbers of this section come from this script alone. The results are in lesarten/vorher_nachher/ergebnis.json.
5Breakdowns and repairs
Checks that passed while a fault was there, reports that were false, work the user could not see, ambiguous instructions and agents sharing one machine: these are the breakdowns the record holds. Five of the counts in this chapter and the one before were recounted from the raw logs and come with a list of their episodes. They count recorded events. Checks and reports that were right were not counted, so none of them is a rate.
5.1How much of the episode time was trouble?
: what the routine episodes consist of
How this was coded
5.2What did a passed check establish?
In at least 20 episodes an agent's check or reassuring report was followed by a fault the user found by looking. Some of the documented checks tested a quantity that did not capture the fault.
One episode. On Nov 10 the user wanted the knife in the level video slanted, tip up. An agent tilted the knife in all levels and recomputed the falls. The user saw that the knife did not sit where the cut was. The agent measured the deviation and got zero. Only on his second complaint did it find two tilts applied in the wrong order, which turned the blade about 5 degrees out of the cut plane. Two hours later the user sent a picture showing that it was still wrong: the fix had checked the angle and not the position. The episode was chosen for clarity; it is one of the 15 labelled as stuck.
TimeThe userThe agent
16:26requestedthe slanted blade steeper, tip up
implementedtilts the knife in all levels and recomputes the falls
judged by lookingsees that the knife does not sit where the cut is
judged by measuringfirst check: measures a deviation of zero
judged by lookingcomplains a second time
named the causetwo tilts applied in the wrong order, blade about 5 degrees out of the cut plane
implementedsecond check: the fix is tested on the angle only
judged by lookingtwo hours later, a picture: still wrong
named the causethe position was never checked; both cut faces lie on the same side of the blade
20:22implementedshifts the falls by that offset and recomputes all six levels
How often. In at least 20 episodes on 10 days an agent's check, measurement or report said that all was fine, and the user then found the fault by looking. Two more episodes are borderline cases (first list below: 22 episodes, 20 clear and 2 borderline; two of them hold two such cases). Another example is identical exposure values in frames that were too bright because four lights were on. In several documented cases the agents had checked a number they had chosen themselves, and the fault sat elsewhere: in a position, a seam, a light or the strength of the scattering (how far light travels under the skin). That is not shown for every case. The list mixes measurements, visual checks and plain reports: for 11 of its 24 entries the role coding records no agent measurement. Read again by one model coder, 6 of these 11 are an agent's look at its own picture or render (2 of them the borderline cases), and 5 are status reports or diagnoses without a recorded check. Whether the measurements themselves were right was not checked, and a faulty check can also pass.
What this count cannot show. Correct checks were not counted, so these numbers give no failure rate. Faults that the user missed were not counted either, so the list shows that checks failed, not how reliable the user's look was. Research on people who supervise automation describes complacency as arising when other tasks compete for attention, and as not overcome by simple practice [62]. The user often ran two to six sessions at once (); whether this affected the user's checking was not examined. Similar patterns are reported outside this case. In a benchmark of graphics editing in Blender through code, the best vision-language systems tested still failed tasks that are easy for human Blender users, and the checking step was a bottleneck of its own [63]; it did not measure these agents or this user. In a study of software patches counted as solved, 7.8% failed the developers' own tests and 29.6% of the plausible ones behaved differently from the reference [64]; behaving differently is not the same as being wrong.
In at least 24 episodes an agent reported a status or a passed check that was false. Two more episodes are borderline (second list below: 27 reports in 26 episodes, 24 clear and 2 borderline; one clear episode also has a borderline report). One example is 62 finished frames that were really 5. The reports came from what a script returned or from an assumption, without a look at the result. Several confident answers that something was impossible were reversed only after he pushed back. With the sound, the agent said several times that it had measured levels and timing but had not listened ().
A report that sounds certain is no evidence that a result was checked. Stated confidence of language models has been found to be too high under the ways of asking that were tested [65], and the explanations models give can misstate the real reason for what they did [66]. So in this study a cause that an agent names counts as a recorded statement, and the 41 apologies below count as a habit of speech, not as a count of mistakes.
The other direction. The sound shows the reverse case: there the agents' measurements caught faults before anyone heard them.
With the sound the agents could only measure. In 23 messages over four days they said they had not listened and had checked only levels, spectra or timing. Measuring caught four faults before anyone heard them: two layers that were too loud, one by 12 dB, a sign error of 0.2 s in the agent's own placement of a track, and a beat the generator had dropped. Listening caught four that the numbers had passed: a re-cut track whose kicks sat 2 ms from the grid but “did not fit”, a guide with single hits where the user expected the movements, a track placed 8 s late, and an offset that was wrong twice until the user named the cause. Of 11 critical listening judgements by the user, 10 led to a change, most within minutes; the one without went to the wrong session. A count of onsets shows that the final track matches 73% of the guide's hits within 50 ms, against 33% by chance, so it follows the guide well above chance but not beat for beat. One coder counted this from the chat; that the user listened each time is inferred from his messages ().
Who caused the mistakes cannot be counted fairly. The coding names the agents far more often than the user (). But a mistake counts only where someone in the chat names it. Agents say “my mistake” or apologise in 41 messages, the user in 3. An order that can be read two ways is seldom called a mistake by anyone. No episode is coded as the user's mistake in both readings. Checks and reports that were right were not counted either, so these numbers give no error rate.
: nine misunderstandings, read again by three models in new sessions
How this was counted
How the replayed misunderstandings were chosen, prompted and judged is described in the method tab, .
The project had no agreed test for how a result looks. Fixed reference pictures existed from Nov 3, and image distances taken from deep networks match human judgements of the similarity of small image patches better than older measures [67]. Such a distance could be one part of a test calibrated against references. It does not judge anatomy, changing design goals or the quality of an add-on, so this does not show that an adequate test was available and left unused. Building and calibrating such a test is a task for future work.
Two pilot counts on looking. Two exploratory counts approach the same question from the user's screenshots and from the agents' own image reads. They are small or rest on unvalidated word lists, and they are not used in the abstract or in the list of findings.
Pilot (50 image prompts, coded by one model). 787 prompts carried 869 images. 381 of these prompts were sent while an agent was still working, typically 12 seconds after the screenshot was taken (772 matched a screenshot file). In a seeded random sample of 50 image prompts, coded once by a model of the agents' own family from the text around them without viewing the images, 25 were fault reports (95% interval 37% to 63%), 10 new wishes or instructions, 7 references or comparisons, 5 unclear, 2 approvals and 1 question. A keyword rule agreed with these codes in only 23 of 50, so no larger count is given. Per hour of episode window, misunderstanding episodes carried 6.1 images and stuck episodes 5.0, against 3.0 in routine episodes and 1.6 in dead ends.
More detail: and , the two pilot counts
| Purpose (sample of 50) | n | 95% interval |
|---|---|---|
| Fault report | 25 | 37% to 63% |
| New wish or instruction | 10 | 11% to 33% |
| Reference or comparison | 7 | 7% to 26% |
| Unclear | 5 | 4% to 21% |
| Approval | 2 | 1% to 13% |
| Question | 1 | 0% to 10% |
| Episode label | Images per hour |
|---|---|
| misunderstanding | 6.1 |
| stuck | 5.0 |
| decision | 3.8 |
| went well | 3.2 |
| routine | 3.0 |
| dead end | 1.6 |
Pilot (1,566 work turns in 31 sessions, counted by script). In the 31 main sessions in which agents ever read an image, 1,566 turns had three or more tool calls. 782 of them ended with a message saying done, fixed, checked or saved, and in 309 of these (40%) the agent had read no image in that turn. After the rule “check only with your own viewport screenshots” was first written (Nov 5, 14:50), 52% of work turns contained a look, against 59% before; claims without a look were 41% after and 38% before. A look is any image read, which can include reference pictures, and claims were found by a word list that was not validated. Tasks changed over the same days, so this does not show that the rule had no effect.
| Work turn | n | Next prompt with a screenshot |
|---|---|---|
| Claim, agent looked in that turn | 473 | 25% |
| Claim, agent did not look | 309 | 17% |
| No claim, agent looked | 392 | 26% |
| No claim, agent did not look | 392 | 18% |
Limits of these pilots
Both pilots were run once, by one coder of the agents' own family, on extracts of the raw logs. The purpose codes rest on the text around a picture, not on the picture. A fault-word list applied to the next prompt gave about half in every group and does not discriminate, so no fault result is given for the agents' looks. The counts are exploratory and are not used in the abstract, the lead sentences or the conclusion.
5.3Which problems recurred?
Seven problems recurred, from needless recomputation to short prompts read the wrong way. Five counts were re-read in the raw logs and come with a list of episodes; the others rest on the episode texts.
5.4What did parallel agents bring and cost?
Running several agents at once was fast and needed coordination of its own. In 98 of 171 hours in which the user sent prompts, they went to two to six sessions, and after a written manifest three new fruits were built in one afternoon and evening. Messages reached the wrong agent at least 17 times in 16 episodes. Once a passed-on approval was used by a helper to restart Blender. Agents shared files, Blender instances, ports, memory and the user's attention, and they coordinated through the tiling terminal: they sent messages, asked each other, and claimed and released ports, Blender instances and files (). Separate working copies, own ports and a shared state record were added only after collisions. On Nov 2 a lead agent held back its helpers' renders before the user made that a rule.
More detail: , coordination calls between agents
| Coordination call | Count |
|---|---|
| Messages sent to another agent | 262 |
| Replies to such messages | 103 |
| Questions to another agent | 23 |
| Inbox reads | 114 |
| Claims of a port, Blender instance or file | 131 |
| Releases | 97 |
| Messages to helpers in Claude Code (Nov 10 to 11) | 198 |
| Messages to helpers in Codex CLI | 69 |
| Relayed messages arriving in sessions | 424 |
Coordination research describes such work as the managing of dependencies; shared resources and things happening at the same time are two of its basic kinds [68]. Reading the collisions here as both kinds is this study's interpretation. A taxonomy of failures in multi-agent systems, developed from 150 runs and then applied by a model annotator to more than 1,600, has a whole group of failures in the coordination between agents [69]. A research agenda of twelve open challenges in the communication between people and agents names problems similar to several breakdowns of this chapter [70]. So similar kinds of breakdown are reported elsewhere; their causes and frequency here were not compared with those studies.
5.5How did objections become rules?Pilot
Pilot (214 writes to rule files, 14 rules, two cases read in full). Agents wrote 214 times to files that hold standing rules: 170 memory notes, 21 skill files, 18 memory indexes and 5 writes to the repository's agent rules. Of the 14 rules in today's agent rules file, the first write of each came a median of about three minutes after the user's last prompt in that session, and an explicit request to note it appears before only one of them. Two cases were read in full: on Nov 2 the user said never to work in Blender in the background, and a memory note followed within a minute; on Nov 6 he said that the stem must always join the berry the same way, and the line entered the rules file at 15:15, when one agent consolidated earlier notes. The rules did not cross harnesses: a Codex agent started four background renders unasked on Nov 5, when the rules lived only in Claude's memory files. Whether rules were kept later cannot be counted from patterns alone; about five clusters of renders without a request in the two prompts before are candidates, not proven breaches.
More detail: , the pilot list of standing rules
| Rule | First written | Minutes after the last prompt | Asked for |
|---|---|---|---|
| Reserve the shared Blender port first | Nov 5, 14:50 | 3 | no |
| Restart Blender only with the user's okay | Nov 6, 15:15 | 3 | no |
| A broad process kill once ended the agent's own shell | Nov 6, 15:15 | 3 | no |
| No bake job with a scene override | Nov 2, 17:58 | 1 | no |
| Self-test only after a big change or with an okay | Nov 5, 14:50 | 3 | no |
| Check only with own viewport screenshots | Nov 5, 14:50 | 3 | no |
| No final renders by agents; the user renders | Nov 4, 14:35 | 6 | no |
| View, light and camera are the user's | Nov 5, 15:14 | 7 | no |
| No photo on the cut face | Nov 6, 15:15 | 3 | no |
| No mass realisation of leaf hairs | Nov 6, 15:15 | 3 | no |
| Save the main file only if its check passes | Nov 6, 15:15 | 3 | no |
| Deleting needs the user's yes | Nov 2, 00:20 | 1 | yes |
| The repository stays local | Nov 1, 22:03 | 18 | no |
| Rendered viewport only on request | Nov 1, 19:10 | 14 | no |
Two cautions belong to this pilot. First, “unasked” means that the user did not ask in the prompts before. Whether the harness or the user's standing instructions asked agents to write such notes was not checked (). Second, a written rule is not yet a repair. A study of repository-level context files found that files written by developers raised the success rate of coding agents by only 2.4% on average, not a reliable gain over having no file, and that files generated by models fared differently, although the agents followed the instructions [71]. Another study, of 2,303 such files, found that they grow in many small additions and that functional instructions predominate [72]. In the pilot on looking in , the share of work turns with a look did not rise after the rule to check only with own screenshots was written, but tasks changed over the same days. Neither that benchmark nor this uncontrolled before and after count shows whether the rules of this project helped.
Two feedback rounds. Twice the user and an agent stopped to review how they worked together. On Nov 3 the user asked what he should do differently. The agent advised a reference picture or sketch for new looks, saying what comes first, judging only after it reported done, and saying what should hold after a stop; for itself it promised to report only checked results. The user gave no advice in return. On Nov 10 the user proposed the review. The agent asked him to name the window or object a screenshot meant and to say “now” or “later” with new ideas, and admitted two crashes, building before understanding and long reports; the user asked that the main task always come first and that side ideas go on a written list. Some of this was taken up. After the first round he worked with fixed reference pictures (explicit mentions of a reference rose from 1.5 to 3.3 per 100 prompts, though he had used reference photos the day before). After the second he briefly marked side ideas with “note:” (7 times, on Nov 10 and 11), and the agents kept a written inbox, which they wrote to more than 260 times in three and a half days and which held 71 entries by Nov 17. Other advice left no trace: image prompts named the window no more often than before, and misunderstandings per prompt did not fall after Nov 10. The agent's promise of shorter reports cannot be told apart from the normal drift. Only three and a half days of data follow the second round, too few to show an effect on the work. One script and one coder made these counts.
5.6In short, and what stays open
The firm part of this chapter is a pattern, not a rate. In at least 20 episodes a check or reassuring report was followed by a fault the user found by looking, and some of the checks tested a quantity that did not capture the fault. In at least 24 episodes a report was false. In at least 37 the user stopped or objected to work the user could not see or had not asked for. Messages reached the wrong agent at least 17 times. With the sound the direction was reversed: measuring caught four faults before anyone heard them, and listening caught four that the numbers had passed.
Open questions. How many checks and reports were right? A random sample of “done” reports, judged by a stated rule for the outcome, would estimate that for done-reports; it would not give the denominator for all checks. How many faults did the user miss? Did a written rule change what agents did afterwards? The pilots cannot tell.
6How the goal formed
A commission for one animated strawberry became an add-on and films through a series of recorded turns, and the final goal was not in the first request. Many turns came after the user had looked at a result or after a failure. This chapter rests on readings by single coders, each a model of the agents' own family. Its counts are interpretive, and no second coder repeated them. Some of its facts are plain in the chat, such as the step from a commission to an add-on, and are not weaker for having been described by a model.
6.1Was the final goal in the first request?
One turning point. On Nov 3 from 16:10 the user compared the particle strawberry with a reference photo for almost two hours. The agent measured and tuned the gloss and the tissue, and the user wrote: “hm. somehow we're not there.” In between, an agent declined to strip a watermark from a stock photo and generated new reference pictures instead; the user declared them the references for every agent. At 19:26 the user asked for a version without the particle spheres, and at 19:51 opened a new session with a second agent (Codex) to rebuild the strawberry from nodes, with the reference beside it and screenshots as the check. Two minutes later a first version stood. The restart was the user's step, prepared by a failed match and by references an agent had generated; one process study counts it for the user, another for both. It is chosen as a clear example, not as a sample.
Three exploratory readings reconstruct a shift from an animation commission to an add-on and films. Two agents (claude-opus-5-5) traced the turning points, each on its own, from the raw logs and without this page. A third applied linkography, a method from design research that links each step to the earlier steps it builds on [37, 38]. Each had one coder and its own rules for what counts as a turning point, so their counts are not three measurements of the same thing.
The three readings reconstruct a shift from an animation commission to an add-on and films. Neither process study found the final specification in the first request. Requirements were recorded step by step, often after the user had looked at a result. A release is first mentioned on Nov 9. The readings usually credit changes of purpose to the user, but they differ over where an event begins and when to credit both. Counting who first states a purpose tends to credit the person who commissions and approves the work. It does not show that he alone saw the possibility. Agents proposed some of the forms that were kept and many of the mechanisms: the procedural rebuild on the first day, their own particle solver, the precomputation behind the fast mode and the proposal to turn the scripts into a real add-on. The logs do not show whether an agent with a broader brief, or a different way of working together, would have reached a comparable result.
6.2Where did requirements and ideas come from?
The tally of the 25 selected ideas
Among the 25 selected concepts and working rules, the coder attributed 14 to the user, 2 to an agent, 7 to both sides and 2 to outside sources. The user voiced 19 of them first, an agent 6. Six of the user's 14 are working rules; for the add-on itself the split is 4 user, 4 both and 2 agent. Agents built 18 of the 25 alone and 7 together with the user. One model chose the 25 from visible decisions and features, so this tally measures the selection more than the project.
: the 25 ideas, one by one
Two things are missing from the list. Agents made many visible design choices at small scale. Examples are the camera path around the eaten fruit and the structure of the next level of the film; another is a first eating animation with 21 bites, which the user cut to 4. And the user also brought in technical concepts: a fast preview of a cut, whose mechanism the list credits to the agents, a rule for one global resolution, and parts of the fruit that can be selected one by one. Both points were found by two reanalyses of the same archive by another model family (method tab, ). There is also a general caution about first statements. In an experiment with an image generator, people held more closely to the first example they were shown and had fewer and more uniform ideas [73]. A first statement by the user that answers a result an agent has just shown may owe part of its form to that result.
How this was coded
6.3What did discontinued branches pass on?
Three lines of work did not end in the released add-on or film: the kiwi and strawberry trailer, water and juice simulation, and the level video. Together they take up to 86 of the 265 episodes, “up to” because some episodes count under two lines. The water count still rests on episode texts and includes some episodes that are not about simulation (). Whether to pursue them was a choice of direction, and the count of mistakes does not cover such choices.
Both process studies found much activity in branches without a release of their own. Their numbers are not directly comparable, because their corpora, time windows and ways of assigning work to branches differ (method tab, ). Several discontinued branches passed on references, features or ways of working, such as the juice, the cells, the bites, the precomputation and the default look. How much of each branch's effort survived was not measured. The second study informally estimated pure cost at roughly one fifth of the total effort; this was one coder's judgement, not a measured fraction or a cost-benefit calculation. Six fluid-simulation branches did not bring a simulation into the released film, but some contributed tools, requirements or the move to procedural juice. The linkograph finds 7 side paths with no later use by the cutoff, 5 of them started by the user and 2 by both sides (). The linkograph is one coder's map: of its automatic links that were checked, 43% were real building on earlier work, and a link that is missing does not show that no relation exists.
More detail: , the side paths and what they passed on
More detail: , the linkograph of all 265 episodes
How to read it: each dot below the line joins two episodes. Follow its two diagonals up to the episodes it connects. A dark dot means the later episode builds on the earlier one, a light dot that it meets the same problem again. Candidate links were proposed from shared words, and only 624 pairs were judged one by one by the coding agent, so a missing dot does not mean no relation. Hover an episode to highlight its links.
One chain of reuse. On Nov 5 at 00:38, for a water render on a rented cloud machine, an agent built on the user's request a small player that loops the finished frames and appends each new one. The water shot did not reach the release; the linkograph lists the player as what it passed on. Live players of this kind were then used for later renders. On Nov 6 the user saw in one that a bite render was wrong from frame 95. On Nov 12 a live preview of the full render showed the flesh glowing orange, and the user had the render stopped after 306 frames. This chain was chosen because it is easy to follow. It is not a sample, and it does not measure how much of the branch's effort survived.
7Who held which role
On the 60 episodes both coded, two coders rank the two sides the same way in all eight roles. The user more often requested, proposed, judged by looking, kept or dropped and stopped. The agents more often implemented, named the confirmed cause and measured. This describes these two codings; it is not an established ordering for all the work. The coders disagree on many single episodes: they give the same code in 55% to 93% of the episodes per role, and agreement beyond chance is minimal to weak for six of the eight roles [47]. They also differ on some counts, such as how often agents proposed (14 against 27 of 60). So this chapter sets both coders side by side and gives each count with its coder.
7.1One episode, eight roles
What a role is, and how two sides can hold the same one, is easiest to see in a single episode. This one was chosen because all eight roles occur in it. It is not a typical one: it is coded as a change of direction, and most episodes are small fixes.
On Nov 5 at 10:19 the user sent a screenshot of the panel and asked for buttons to spawn a strawberry and to randomise its shape (requested: the user). He suggested computing the interior only when a cut needs it, with an explicit toggle. The agent called the toggle better than its own automatic plan, pointed out that so far everything was loose scripts, and proposed one switch with Off, Fast and Exact (proposed: both, because each side has a recorded proposal of its own). The user added that cuts should still work without an interior, as a plain stand-in face, and decided: “yes let's build the add-on.” (kept or dropped: the user).
A test that he had interrupted with new messages (stopped: the user) still ran halfway and left Blender without an interior, and he saw that part of the inside was missing (judged by looking: the user). He also wanted to know why the exact cut is so slow. The agent's answer came with a number (judged by measuring: the agent): the join of skin and flesh at the cut edge is rebuilt on every frame and takes 3.35 of the roughly 5 seconds (named the cause: the agent). Implemented is coded for the agent. The episode ended at 10:47 and is coded as completed.
TimeThe userThe agent
10:19requestedbuttons to spawn a strawberry and to randomise its shape
proposedcompute the interior only when a cut needs it, with a toggle
proposedone switch with Off, Fast and Exact; an add-on is the right way
10:24kept or droppeddecides: “yes let's build the add-on”
stoppedinterrupts a running test with new messages
judged by lookingsees that part of the inside is missing
judged by measuringthe join takes 3.35 of the roughly 5 seconds per frame
named the causeskin and flesh are joined again on every frame
10:47implementedcoded for the agent
7.2The ordering, by two coders
One coder, a model of the agents' own family, coded all 265 episodes with a codebook written beforehand. A second coder, a model of another family, coded a sample of 60 again with the same codebook and without seeing the first coding. sets the two side by side for those 60 episodes.
: two coders side by side
| Role | Agreement | Kappa | Score moved towards the agents | Score moved towards the user | User held it, first / second coder | Agents held it, first / second coder |
|---|---|---|---|---|---|---|
| Proposed the approach | 65% | 0.39 | 18 | 3 | 52 / 44 | 14 / 27 |
| Requested the work | 93% | −0.01 | 3 | 1 | 59 / 60 | 1 / 3 |
| Implemented it | 87% | 0.38 | 0 | 8 | 3 / 11 | 60 / 60 |
| Named the confirmed cause | 70% | 0.49 | 14 | 0 | 3 / 5 | 26 / 44 |
| Judged it by looking | 63% | 0.46 | 15 | 5 | 45 / 52 | 22 / 38 |
| Judged it by measuring | 55% | 0.24 | 25 | 1 | 0 / 2 | 22 / 48 |
| Kept or dropped it | 65% | 0.41 | 10 | 10 | 32 / 41 | 2 / 11 |
| Stopped running work | 80% | 0.64 | 3 | 7 | 26 / 30 | 1 / 3 |
| Outcome | 77% | 0.52 | ||||
| Scale | 80% | 0.64 |
What both coders show. The user requested the work in nearly every episode. The tool makes this nearly unavoidable, because an agent starts work only on a message. Agents implemented in every sampled episode. Between these two fixed points, both coders have the user ahead in proposing, judging by looking, keeping or dropping and stopping, and the agents ahead in naming causes and measuring.
Where the coders differ. The second coder sees agents proposing in 27 of the 60 episodes, the first in 14. It sees them naming the cause in 44 against 26, judging by looking in 38 against 22 and measuring in 48 against 22. It also sees the user working by hand in 11 episodes against 3. So neither side held one kind of role, but how large the crossovers are depends on the coder.
What an ordering cannot show. A role is coded once per episode, however often it occurred, and a change of course counts the same as a small correction. A request is not yet a proposal: the codebook counts a proposal only where a side contributed essentially to the approach or form that was pursued, and the first coding gives the user one in 212 of the 265 episodes, not in all. Reading an image is not yet judging it: the log rule counts an agent as looking when it read at least three of its own images in the episode. A cause counts as named when the chat records it as confirmed, and part of that record is the agent's own explanation. The user's hand work is counted only where the chat records it. The small choices inside the agents' tool calls are not counted ().
Describing work with computers by roles and not by sums is an older idea. Earlier work sorts the ways a computer can be a partner in creative work [42] and describes creation in which human and computer both take the initiative [43]. One distinction from that work is useful here: in alternating co-creation both sides work on the same thing in turns, in task-divided co-creation each has its own tasks [74]. As a proposed analogy, not a coded result, this project looks mostly task-divided, with alternation where forms were proposed. The distinction also concerns whose requirements are met and who evaluates, which role frequencies alone do not settle. A framework for describing the interaction in co-creative systems, tried on 92 of them, exists [75]; the codebook of this study was not compared with it.
Four crossovers, one line each. On Nov 4 the user alone named a cause: the scattering under the skin had been switched off. On Nov 12 the user saw the effect of too strong scattering and the agent found the fault in the code, a joint diagnosis. On Nov 4 an episode text says the user had a run stopped; the log shows that the agent stopped it itself at 23:11. On Nov 13 the agent measured a jump at frame 691 forty seconds before the user named the falling pieces, so the cause is coded for the agent. The box below holds all nine cases.
The first coding for all 265 episodes ()
These are one coder's counts for all 265 episodes. They are the only full count. A second coder of 60 of them differs on several roles (), so read each count together with that comparison.
Disputed and illustrative cases
One of the 5 episodes in which the user alone named the cause: scattering under the skin had been switched off.
A joint diagnosis: the user saw the effect of too strong scattering, the agent found the fault in the code.
The ideas for varied shapes came from a helper, and nothing new was built in this episode, so proposed is coded for the agent and the outcome as unresolved.
The episode text says the user had the run stopped. The log shows that the agent stopped it itself at 23:11, so stopped is coded for the agent.
The agent measured the jump at frame 691 forty seconds before the user named the pieces, so the cause is coded for the agent; the proposal is coded for both, because the user had asked for a check before rendering.
The agent proposed the fast mode at 08:48, the user asked for it in his own words at 08:52, so the proposal is coded for both.
The agent refused to remove watermarks and generated its own reference pictures, which the user made the standard, so keeping is coded for both.
The agent proposed a viscosity of 0.3, the user the surface tension; in the end the user dropped the water.
Eight render starts in half an hour. The roles say what happened: the user proposed, stopped and decided, and the episode ended completed.
7.3How firm is the ordering?
Per role, the two coders give the same code in this share of the 60 episodes, with kappa, the agreement beyond chance, in brackets: proposing 65% (0.39), implementing 87% (0.38), measuring 55% (0.24), naming the cause 70% (0.49), judging by looking 63% (0.46), keeping or dropping 65% (0.41) and stopping 80% (0.64). By the scale of [47] that is minimal agreement for the first three, weak for the next three and moderate for stopping. On requesting they give the same code in 93%, but kappa is near zero (−0.01): the user requested almost every episode, so chance alone already gives a high match, and the few disagreements decide kappa. These bands are a convention for reading kappa. They do not turn agreement into correctness.
Counted per side, with “both” counted for each side, the second coder records more acts in most roles and for both sides. For the agents: proposing in 27 of the 60 episodes against 14, naming the cause 44 against 26, judging by looking 38 against 22, measuring 48 against 22 and keeping or dropping 11 against 2. For the user: implementing by hand 11 against 3, judging by looking 52 against 45, keeping or dropping 41 against 32 and stopping 30 against 26. Only for the user's proposals is the first coding higher, 52 against 44. The comparison file also scores each difference by a relative rule (user +1, agent −1, both and none 0); for proposing that gives 18 differences towards the agents and 3 towards the user. That score is not a count of acts: three of those 18 are changes from the user to none, where no agent act was added (method tab, ).
What holds under both coders is the ordering of the two sides in each role, and two counts: agents implemented in 60 of 60 episodes for both, and the user requested in 59 and 60. Other counts move with the coder. Neither coding was checked by a human or adjudicated. Each coder belongs to the family of one of the project's agents. The two set the threshold for a recorded act differently, and which threshold is nearer to the logs is open. For judging by looking and stopping, both codings also share one input: the events that one log pass added, which the second coder did not run itself.
Known leanings of models as judges could each have played a part: towards texts of their own family, towards the view of the person they answer, or towards longer texts. None of them was tested here, and none would show which coding is nearer to the logs. So neither coding is treated as the standard. The possible leanings and the full comparison are in the method tab, .
7.4Did the division change with the size of the work?
In the first coding, agents took part in the proposal more often as the work got larger: in 39 of 142 small fixes (27%), 35 of 102 features (34%) and 11 of the 21 episodes that changed the project's direction (52%). Examples of the last group are their own particle solver (Nov 2), the switch to the FLIP solver (Nov 1) and the step to an add-on (Nov 5). The second coder saw only 3 of those 21 episodes, and proposing is a role with minimal agreement. The rise is a pattern in one coding and is not checked.
The user stopped work in 32 of 48 episodes that stayed unresolved and in 72 of 188 completed ones. That is a co-occurrence, not a cause.
7.5The sound: a part where the user also made
In the sound the user also did much of the making. An agent's script made the first guide track from the keyframes and chose which sound marks which event (audio 1). Agents then built about 15 guide versions between Nov 10 and 12, and the user set their form: first clicks and warm sweeps, then “a different tone for each thing”, then fixed tones in one octave (audio 2). The music came from a generator. An agent used it once, through an API (four variants); the user generated tracks himself and brought six files back, and both posted videos use his music. He cut the sound of the first video himself, starting 26.8 s into a version of his track that no agent had opened. For the final track the agent found where the theme starts again, at 30.10 s, and cut it to the film (audio 3); the user exported the finished video himself.
7.6In short, and what stays open
Both coders give the same ordering: the user ahead in requesting, proposing, looking, deciding and stopping, the agents ahead in building, measuring and naming causes. Both also record crossovers in each direction, and they differ on how many. Each count in this chapter is given with its coder, and none is a share of the result ().
Open questions. Would a human coder, who can also see the pictures, side with one of the two codings or differ from both? Which codebook rules, with boundary examples added, raise agreement on proposing and measuring? Does the rise of agent proposals with the scale of work survive a second coding of all 21 direction episodes?
8Discussion
The record supports an account of how the work was organised and how its requirements formed. It does not support a share of the result.
8.1What the record shows about the work
The record answers two questions better than the one in the title. The first: which checks did each side perform, and which recorded faults got past earlier checks? In this project the user more often supplied the recorded visual judgements, and the agents more often the recorded measurements (). That is a tendency in the record, not a division of abilities. In the 60 episodes both coders coded, agents also judged by looking in 22 and in 38, and the second coder records the user measuring in 2; in the full first coding agents judged by looking in 105 episodes. An agent's look is counted from its image reads, which is only a sign of looking. With the sound the agents could only measure, because they could not listen. The second question: what coordination did running several agents side by side demand ()? The record shows that work, but not its cost in time or money for either side. Neither question needs a share of the result.
Design research holds that finding and reframing the problem is part of creative work, not a step before it [39, 40, 41]. This case fits that view: neither process study found the final specification in the first request, and many requirements were recorded only after a result had been seen. On the view that only building counts, this changes nothing. The evidence is rough. Each process reading had one coder, and the linkograph uses whole episodes as steps, which favours the user because episodes are organised around what the user asked for. Only part of its links were checked, fewer than half of the checked automatic links held, and a link that is not shown may still exist ().
The coordination itself was work. Splitting tasks over agents, keeping copies apart, routing messages and turning objections into rules took effort from both sides, and most of the recorded breakdowns sat there rather than in the code ().
Two cautions from outside this case belong here. Working with generative AI makes demands on the user's own monitoring: stating goals, splitting tasks and weighing one's own judgement [79]. That framework does not measure such work, and in this record it leaves little trace. And it cannot be taken for granted that steering improved the result. A meta-analysis of 106 experiments found that a human together with an AI was on average worse than the better of the two alone. Tasks of deciding showed losses. For tasks of making the estimate was positive but not reliably different from zero, although the two kinds of task differed significantly [80]. That gives no effect estimate for this case, which has no run without the user to compare with ().
6.2Roles, not shares
The logs give no single answer to the question in the title. Each role is counted in its own unit, the same interaction can carry several roles, and nothing in the logs converts one role into another. So the study gives no share for either side and offers no scale for weighing them. sets the roles side by side.
How to weigh roles is a convention the data cannot settle. Science and law have met the same question; they use roles and thresholds instead of exchange rates (). A reader who counts only implementation will find that the agents made the strawberry; a reader who counts direction and judgement will find that the user did. Both readings fit the same counts, and neither is tested here.
Other fields have met the same problem, and each has a convention of its own: prints name designer, engraver and publisher; workshops, film, music, courts and mathematics have their own rules for who is credited (). These conventions offer alternative role mappings, not evidence about this case. One mapping treats the agents as technical co-designers, since they proposed adopted forms and many mechanisms, and the user as commissioner, coordinator and judge who also did some hand work. Another puts the user near the inventor, producer and person in control and the agents near the engraver, workshop and session players. No field has a ready rule for hands that carry out the work, are software and make many choices of their own. How readers assign credit is a question of its own: in a study of attribution in human-AI co-creation, people judged the contributions of an AI differently from the same contributions by a human [14, 15], and in an experiment in four countries, participants gave a user more credit for AI-generated text when a scenario described the model as personalized on that user's own earlier writing [81].
6.3The strongest objection
The strongest objection is normative: commissioning, judging and rejecting is not making, and the agents manufactured. The ordering of the roles fits that description. It also fits the opposite one, that he shaped the work the way a director shapes a film; neither follows from the record. On Reddit, 83 comments from 45 accounts were coded as giving him little or no part; how many of them meant no activity at all, rather than no authorship, was not coded ().
Six further readings by agents with assigned briefs, two for the user, two against him and two neutral, each followed its brief (method tab, ). Their strongest challenge stays open: the agents' many small design choices are not counted (). This study, too, was made at the user's request.
What the comments measure is a further question. In experiments, work labelled as made by AI was rated as less creative, partly because the AI is seen as making less effort; in one set of studies the label lowered ratings directly for paintings, but not for advertisements or business ideas [82]. Another set found that the label lowered ratings even when a human was described as working on the piece too [83]. These studies do not show why the commenters here gave the user little part. They are a reason not to read the comments as a measure of roles. Points on Reddit are also not independent votes: in a field experiment on another site, a single early positive vote, assigned at random, raised later ratings clearly [84].
6.5Reach of one case
One case can show that this division of work happened, with these counts. It cannot show that it had to happen, because there was no run without the user and none without the agents (). Two readers tried something like it with prompts of their own, and neither result was rated here (). Nor does it show what a user without a trained eye would get. Responsibility cannot be counted either. The user published the result and answers for it, and the rules for authorship cited in place it with humans.
Two patterns seem more likely than the numbers to hold in other projects: checks that test the wrong value () and work that runs unseen (). If they hold, a person who directs agents needs one place where every result shows up at once. Agents working in parallel need their own copies and a shared record of the current state. In this project both were added only after collisions.
What kind of case is this? One densely logged project, with one experienced user, on a task whose quality was judged mostly by looking, though sound, running time and code mattered too, and in which the agents could operate Blender themselves. Naming the kind of case does not by itself carry its lessons elsewhere; a single case can teach something general when its conditions are stated and compared with other cases [85]. An observational study of programming in conversation with AI, whose recorded programmers were all experienced, found that they used their expertise to manage the context, judge quickly and choose between the AI and work by hand [86]. It did not test whether expertise is necessary. The logs show the user using expertise in similar ways here.
9Limits
This study has three main limits. It is one case, with no control run. Only models coded and wrote it, most of them of the agents' own family, and two coders agreed only minimally to weakly on most roles. And the user, who offers the add-on to paying supporters, asked for it. The limits below are in three groups. How each was found, and what was corrected, is told in the tab “How this study was made”.
Limits that bound every conclusion
- Methods not used. Other fields measure such splits in ways this study did not. No human coder repeated the coding, and agreement was measured only between models [46, 47, 48]. The user was not interviewed, which case study research recommends [44]. The only human who reviewed the study was the user himself, its subject and commissioner. This is a participant review (member check) [45], not an independent one; no outside human has reviewed the study. No outside experts rated versions of the fruit, as the consensual assessment technique, a rating by an expert panel, would [50, 18]. Process tracing, which reconstructs sequences and assesses evidence for proposed causal mechanisms [49], and linkography [37, 38] were used only in the small analyses of .
- Conflict of interest. This page was counted, coded and written by models of the same family as the agents it describes. The user it describes asked for it. He offers the add-on to paying supporters. On Nov 15 there were 4 paying members, who bring $19.67 a month before fees (). The counts made by script can be checked. The judgements cannot be fully neutral.
Limits that narrow single counts
Data and corrections
- Data availability. The raw session logs hold about 4 GB of chats, prompts, tool output and screenshots. They are not published, because they contain private material of the user and of unrelated projects. The download holds the coded data behind the counts: episodes, role codes, ideas, turning points, the linkograph and the lists of episodes for the recounted findings. Counts made by script from the raw logs (messung.json) cannot be regenerated from the download, and some scripts and intermediate files of the first pass are lost. Checking against the raw logs is possible on request, limited to this project.
10Open questions and further studies
Four steps would change what this study can say, and none of them needs a new project. Two further studies would need one.
- A human in the procedure. A human coder for the 60 sampled episodes, who also sees their pictures. The user coding the same 60 as a named participant. And an hour of going through ten episodes and the user's hand work with the user. That would add a human check of the two model codings and the user's own account of some of what the user did between messages. It would not by itself make the coding a verified account, and work between messages that nobody remembers would stay unrecorded.
- Denominators. A random sample of about 50 “done” reports, judged by a stated rule for the outcome, would estimate how often such a report was right. It would not give the denominator for all checks, measurements or faults nobody saw. The lists in cannot show either.
- The comparison inside the case, counted further. compares the strawberry with the three fruits built after the manifest of Nov 6, counted by script per fruit. It is a confounded comparison inside one case, not a control run and not a speed test. A next step would count the later work on those fruits in shared add-on sessions, check by a second reader which session belongs to which fruit, and reconcile the totals of its script with those of , which differ and cannot be reconciled while an event file is missing.
- Third parties. What the harness and the user's standing instructions asked the agents to do, for example to write rules down ().
The findings chapters end with their own open questions (, ). Three more belong to : Which of the user's first statements answered a result an agent had just shown? How much of each discontinued branch's effort survived? Would a second coder draw the same turning points?
Further studies. This case cannot be rerun as a control. An agent that fulfils the first, narrower request and stops has not failed a fair test. New studies could compare, for the same target, a fixed final specification with step-by-step refinement. Model, tools, budget and rating rules would stay the same, runs would be repeated, and experts would rate the results without knowing which is which. A second design would compare open-ended exploration under different conditions. A third would test what screenshot feedback adds, on matched repair tasks.
11Conclusion
Over two weeks one user and several AI coding agents turned a commissioned strawberry animation into a procedural fruit add-on and an intro film. The two sides took turns closely: an agent typically made 2 tool calls before the user wrote again, and three in four of the user's prompts arrived while an agent was still working. In at least 20 episodes an agent's check or reassuring report was followed by a fault the user found by looking; in at least 24 an agent reported a status that was false; in at least 37 the user stopped or objected to work the user could not see or had not asked for; and messages reached the wrong agent at least 17 times. With the sound, measuring caught four faults before anyone heard them, and listening caught four that the numbers had passed. Correct checks were not counted, so these are not error rates. The final specification developed through exchanges rather than being fixed in the first request. Some discontinued branches supplied references, features, tools or rules; seven side paths showed no later use by the cutoff. On the 60 episodes both coded, two model coders rank the two sides the same way in all eight roles, with the user ahead in requesting, proposing, judging by looking, deciding and stopping and the agents ahead in building, measuring and naming causes. Both record crossovers in each direction, and they disagree on many single episodes, with agreement beyond chance only minimal to weak for most roles, and on some counts. The user also proposed technical solutions, including one mesh for skin and cut face, a global resolution control and separately selectable parts, and worked on camera and light. Agents also proposed forms and mechanisms, including a particle solver and the fast preview. The logs omit some user hand work and agents' small design choices, and what third parties contributed was not examined. This single case gives no share of the result, proves no path necessary and predicts neither side's performance alone. Which roles count as making remains a matter of definition, subject to the study's limits ().
References
ADay-by-day record
Appendix A, the record of all 14 days with every episode and picture, is not part of this file. It is a separate PDF (“Appendix A · Day by day”) in the download menu of the page, and on the page it follows the conclusion and the references: https://strawberry-methodology.pages.dev/#sA
The project ran for 14 days. Figure A1 shows them side by side as one timeline; below it, each day has its own record with what happened, its pictures and every episode.
BReception on Reddit
What the comments are about
Comments about AI: what did the human do?
What viewers noticed in the picture
How this was coded
CResource use and cost
Most tokens in the logs are the same conversation sent again with each request and read from a cache. Counted that way, the logs account for about 8.9 billion tokens. Half of each monthly flat fee came to about €159.
C.1Where the tokens went
: context size per session
: tokens per day
: stops by context size
How this was counted
C.2What it cost, and who paid
How this was counted
DHow other fields assign credit
This appendix collects the examples behind . They give context, not evidence about this case. References [33] and [36] point to further examples from mathematics.
Other fields have the same problem, and each has its own answer. Old prints name each role on the plate: invenit for the person who designed the image, sculpsit for the engraver, excudit for the publisher [19]. Rubens let pupils paint large parts of his pictures. He priced a picture by how much of it he had painted himself [20, 21]. Koons and Hirst have assistants carry out their ideas, and the share of these hands is rarely named [22, 23]. Film counts words. The Writers Guild gives screenplay credit by share of the final script, and an idea counts only if it was handed in in writing [24]. In music, songwriting shares are negotiated. The producer, who directs the recording sessions and chooses the takes, gets a credit of his own [25, 26, 27, 28]. In the two U.S. decisions cited, courts tied authorship to control over the work and to giving it form, even where a camera or other people carried it out [29, 30]; they are single cases in one legal system. Mathematics is working it out now. It links roles to logged contributions and uses lists that sort AI help by how much humans took part [31, 32, 34, 35]. These conventions offer alternative role mappings, not evidence about this case. One mapping puts the user near the inventor, producer and person in control and the agents near the engraver, workshop and session players. Another treats the agents as technical co-designers, since they proposed adopted forms and many mechanisms, and the user as commissioner, coordinator and judge who also did some hand work. No field has a ready rule for hands that carry out the work, are software and make many choices of their own.
Visitors. This page counts visits, how far readers get, which part is being read and whether the simple words or another language are on, without knowing who anyone is. It sets no cookie and stores nothing in the browser for this. The server builds a short code from the network address and the browser type with a random key that is thrown away every day. The address itself is never stored.
Not affiliated with or endorsed by Anthropic. Commissioned by the user it describes.
Methods record · companion to the case study
What the agents did to build and check “Who Made the Strawberry?”, step by step, from the session logs.
Reconstructed on Nov 16, 2026. Times in CET (UTC+1).
AbstractThe case study “Who Made the Strawberry?” was built in about 24 hours of work over three days, Nov 14 to 16, 2026, by one main agent and the agents it started. This record lists what was done in 58 steps: counting scripts, two readings of the logs, analyses, reviews, consensus rewrites, cross-reads, adversarial readings, a blind test, independent studies by another model family, translations, restructurings, a recovery and publication. It shows how many agents of which model ran at once, how the output of one step fed the next, and what each check found. The checks found 33 errors in results or text; 23 were corrected. No second human took part, so every check described here is a check by models.
1What this record is
This tab describes the process that produced the study. It is not part of the study's argument. Its sources are the logs of the Claude Code session that built the study (from Nov 14, 14:36), its 201 subagent runs, eight Codex runs in the study folder, and the study's own briefs, reports and reviews. One model reconstructed the timeline from these logs and cut it into 58 steps. No second coder checked that cut. Private material in the logs was left out. The reconstruction ends on Nov 16: its steps, figures and totals, and the limits drawn from them, describe the process up to then, not the finished study of Nov 17.
The study was commissioned by the user, who approved its main steps and its publication; no other human took part.
One main agent did the orchestration. It wrote briefs, started subagents, merged their results, ran scripts and published. It ran on claude-fable-5-2 until Nov 15, 00:16, and on claude-opus-5-5 after that. Subagents were Claude models started by the main agent or by other subagents. Codex runs used models of another family, gpt-6-astra and gpt-6-luna, started from the same folder.
The study was started on November 14, about 14 hours after the result was posted on Reddit, so that it could be linked from there. The user and the same kind of AI agents that the study describes started it. So the study is not neutral ground, and its limits section says so (the study, ).
Since Nov 17 the two tabs are divided by one rule. The study holds what a reader learns about the project, plus what was counted, who coded, how far two coders agree, and the limits. This tab holds everything about how the study came about: the readings and codings and how they compare, what went wrong when reading the logs, the corrections, and the reviews. Sections 5 to 12 were moved here from the study under that rule.
2The steps
Nov 14. Within the first hour, 14 day readers cut the 14 days into 265 episodes, two coders labelled them, and the page went live; from then on every change was public. A fresh reviewer found seven problems in the afternoon. In the evening the counts moved into scripts, a second counting path found a gap of 138 tool calls, and an agent tracing ideas found that the digest lacked about half of the messages. So 14 new readers re-read every day from the complete chat. Two reviews, a blind test and eight adversarial readings followed before midnight.
Nov 15. The morning went into a plain-language mode (7 agents) and five translations (66 runs, at most 20 at once). In the evening a style audit led to three independent rewrites and a consensus, a check for changes of meaning by another model family, two cross-readers and two independent studies by another family. Two agents then built a shorter preview. Its review by another family stopped at the usage limit, and a same-family review replaced it.
Nov 16. The restructured version went live, and two process studies, a linkograph and 32 sources were added. A cleanup in an unrelated session deleted the local working folder; it was restored from the live site and the logs. The evening brought a plain-language pass checked file by file, a peer review by another family, a check of two listed limits, and an audit of all days against the raw logs.
Nov 17. A model of another family coded a sample of 60 episodes again for the roles, without seeing the first coding (). An overall review by a fresh instance of the writing model followed, and the study was rebuilt: findings grouped by the kind of evidence, roles as an ordering by two coders with both coders side by side, and this tab for the making of the study. An independent review by a model of another family then checked the build against its data and sources, and its corrections were built in. These steps are not in the 58 steps and the figures of this section, which end on Nov 16.
More detail: all 58 steps and what each produced
3How outputs flowed
Most steps took the output of an earlier step and passed their own on. Three patterns recur. draws one example of each.
Parallel and blind. The same task went to several agents that did not see each other: one reader per day in both readings, two reviewers of the same draft, two independent studies, six adversarial readers with only the data. Parallel runs made the work fast. Keeping them blind kept the outputs separate enough to compare.
Consensus. For the style rewrite, three agents rewrote the same text on their own and a fourth merged the three into one version. The merged text then went to a model of another family, which checked it field by field for changes of meaning, and to two fresh readers, one of each family.
Check, then fix. Most checks did not change anything themselves. A reviewer wrote findings, and a later step applied them, often the main agent and sometimes a corrector agent. Some loops reached far back. The day audit on Nov 16 corrected day texts written by the second reading on Nov 14, and the peer review sent section 6.4 back for a fix within an hour after it had gone live.
The links between the steps also show dependence. Almost every analysis after the first evening built on the episodes cut by the first reading and recoded by the second. Later checks that read the same episodes were therefore less independent than their number suggests.
4Effort per phase
The main agent made 1,769 tool calls, its 201 subagent runs made 5,991, and the eight Codex runs made 135. About half of the subagent runs went into simplifying, translating and restyling text, not into analysis. The phase of plain language and translations alone used 73 runs (), twice as many as both readings together. Checks were small in runs and large in effect: the first review was one agent and found seven problems, and the day audit used 16 runs for 292 corrections. The Claude models processed about 1.5 billion tokens, nearly all of them cache reads, and wrote about 10.8 million.
5The two readings of the logs
The episodes of the study were read twice. This section says why, what changed between the readings, and how far they agree. The study shows the second reading.
Episodes. For each day, one agent read all prompts of that day included in the initial digest and the first 500 characters of each agent reply. It split the day into episodes and gave each episode one of six labels. Sorting episodes into such fixed categories is called coding here. The labels are a judgement of the reading agents.
Revision comparison. The second reading kept the first reading's episode boundaries and could see its episode texts, but read the complete chat with another model and other instructions. On the 263 episodes both readings share, they give the same label to 80.6%, the same starter to 95.8%, the same deciding contribution to 79.8% and the same named mistake to 77.6%. By the scale of [47], agreement beyond chance is moderate for the label, weak for the deciding contribution and the named mistake, and minimal for the starter (box below). This compares two revisions. It is not an independent reliability estimate: shared texts can raise agreement, and changed evidence and rules can lower it. So the shift in is not due to the missing messages alone.
How agreement was measured
Agreement beyond chance is measured with kappa: 1 is full agreement, 0 is chance. For the label kappa is 0.74, for who started an episode 0.34, for whose contribution decided it 0.53 and for whose mistake the chat names 0.58. Who started an episode has the lowest kappa despite 95.8% agreement. The reason is that the user started almost every episode, so chance alone already gives a high match. The deciding contribution was later replaced by the role coding below and is kept only for this comparison.
: the two readings side by side
6Earlier codings
Before the role coding, the same episodes were coded in two other ways: by kinds of contribution, and by whose contribution decided an episode. Both stay in the data and are shown here for the record. Neither is used for a finding of the study. The role coding replaced the second one after a review by a model of another family found that its rule tested different things for each side.
Kinds of contribution (second reading)
What the old coding hid
The earlier coding asked whose contribution decided each episode. Its rule tested different things for each side and put the most common pattern under “both”. Read as roles, “both” is almost always the same division of work: the user proposes, an agent builds, the user decides. In 17 of the 32 episodes once coded “user decisive” an agent named the cause, and 11 of them ended abandoned or unresolved, which the old rule should have coded “none”. “Agent decisive” mostly meant that an agent found the cause.
How this was coded
How the misunderstandings in the study were replayed
7The role coding and its independent check
The study reports roles only as an ordering by two coders (the study, ). This section holds the full comparison of the two codings. The table and the chart of the comparison stand in the study ().
Roles. On Nov 16 one agent (claude-opus-5-5) coded every episode again for eight roles: who proposed the approach, requested the work, implemented it, named the confirmed cause, judged the result by looking, judged it by measuring, kept or dropped it, and stopped running work. Each role is user, agent, both, none or unclear, and each episode also gets an outcome (completed, revised, abandoned, unresolved) and a scale (small fix, feature, change of direction). The codebook was written before coding. Where a text did not decide, the raw log was read. After the text coding, stops and agents' visual checks were added from the logs by a fixed rule (box). The coder knew the earlier codes and is of the same model family, so this is not a reliability estimate. A coder of another model family, which recorded itself as “GPT-6 (Codex; exact runtime revision not exposed)”, then recoded a stratified random sample of 60 episodes blind, with the same codebook (). It did not run the log pass. The comparison script merged the first coding's log values into its codes afterwards, so for judging by looking and stopping the two codings share that input and are not two measurements of those log events. It agreed on proposed in 65% of the sample (kappa 0.39), diagnosed in 70% (0.49), judged by looking in 63% (0.46), measured in 55% (0.24), implemented in 87% (0.38), kept or dropped in 65% (0.41) and stopped in 80% (0.64), on the outcome in 77% (0.52) and on the scale in 80% (0.64). Requested matched in 93%, but its kappa is near 0 (−0.01) because nearly every episode was requested by the user, so chance agreement is already high. Counted per side, the second coder recorded more acts for both sides in most roles (). The comparison file also scores each disagreement by a relative rule: user +1, agent −1, both and none 0, and a change counts as towards the side whose score rose. By that score the differences were 18 towards the agents and 3 towards the user for proposing, 14 and 0 for diagnosing, 15 and 5 for judging by looking, 25 and 1 for measuring, 10 and 10 for keeping or dropping, 3 and 7 for stopping, and 0 and 8 for implementing. The score is not a count of acts: a change from user to none counts as towards the agents although no agent act was added (3 of the 18 for proposing), and a change from none to both counts as neutral although both sides gained an act. It is no evidence that either model family favoured one side. The independent coder named the parts of the codebook it found ambiguous: when a contribution is essential enough to count a proposal for both sides, when added work is substantial enough to count a request for both, whether a request to cancel that had no effect counts as a stop, which events belong to overlapping episodes, and how to code an episode with mixed outcomes. Both coders are models, and 60 episodes is a small sample.
The words for the agreement. By the scale of [47], kappa from 0.21 to 0.39 is minimal agreement, from 0.40 to 0.59 weak and from 0.60 to 0.79 moderate. So the two coders agree minimally on proposing (0.39), implementing (0.38) and measuring (0.24), weakly on naming the cause (0.49), judging by looking (0.46), keeping or dropping (0.41) and the outcome (0.52), and moderately on stopping and on the scale (0.64 each). Without the log rule, looking agrees in 43% (kappa 0.24) and stopping in 75% (kappa 0.54). Only 6 of the 60 episodes carry the same ten codes from both coders.
How to read the comparison. Four things limit it. Neither coding was adjudicated, and no human coded; the second coder's notes call its remarks “coding explanations, not adjudication against the first coder”. The second coder recorded more acts in nearly every role and for both sides, so the two coders set the threshold for a recorded act differently, and which threshold is nearer to the logs is open. Each coder belongs to the family of one of the project's agents: the second to that of its Codex agent, the first to that of its Claude agents. And the sample holds only 3 of the 21 episodes coded as a change of direction. Those three were checked, but they are too few to test the pattern for that group.
Possible leanings of the coders. Three known leanings of models as judges could each have played a part. None was tested here. Models have been found to recognise their own texts and to rate them higher [76]; that was tested on ratings of generated texts, not on which actor a coder credits in a work log, so it gives no direction for this comparison. Models adapt their answers to the view of the person they answer [77]; the user commissioned the study, so a coder could lean towards the user. Models as judges have preferred longer answers [78]. That is a preference in rating answers, which is not the same as unequal evidence: the logs hold more text about the agents' acts than about the user's, so a coder who looks for evidence in text can find more of it for the agents. Neither leaning is shown by the differences between the two codings, and each is only a reason to treat neither coding as the standard.
What holds. The ordering of the two sides is the same for both coders in all eight roles. Two counts hardly move: that agents implemented (60 of 60 for both coders) and that the user requested (59 and 60). The others move with the coder. For the user, the first coding is the higher of the two on proposing (52 against 44 of the 60 episodes) and the lower on hand work (3 against 11), looking (45 against 52), keeping or dropping (32 against 41) and stopping (26 against 30). For the agents, the first coding is the lower on every role but implementing.
An illustration of that movement, not an estimate. If the second coder's shares in the sample held for all 265 episodes, agents would have taken part in proposing in about 119 episodes, where the first coding has 85; in naming the cause in about 194, against 127; and in measuring in about 212, against 113. Resampling the 60 episodes gives ranges of 88 to 155, 163 to 221 and 186 to 238. The projection assumes that the sample stands for the whole. It does so only roughly: in the first coding, agents take part in proposing in 23% of the sampled episodes and in 32% of all. These figures show how far a count can move with the coder. They are not counts of the study.
Outcome and scale. The second coder called fewer episodes completed (37 against 46 of 60) and more revised or unresolved (22 against 12).
One episode as an example. The add-on decision of Nov 5 (the study, ) was in the sample. The two coders agree on nine of its ten codes. They differ on keeping or dropping: the first coder gave it to the user, who took the decision; the second gave it to both, because the agent had also dropped its own automatic plan. The codebook allows “both” when an agent discards one variant and the user approves another. Whether the agent's remark was such a discard is the judgement on which the two differ.
Two earlier small checks. Before the role coding, two further agents without access to the study each checked 16 random episodes and agreed with the old coding of the deciding contribution in 13 and in 16 of them. These were checks by models of the same family and are not reliability estimates either.
8The process readings compared
Three readings traced how the goal of the project changed: two process studies and a linkograph. They were made separately, each by one coder and by its own rules. The study shows the counts of both process studies side by side (the study, ). This section sets the three readings next to each other and says why their numbers differ.
More detail: and , how the three readings of the turning points compare
| Turning points | The user | Both | An agent | From outside | |
|---|---|---|---|---|---|
| Process study 1 | 30 | 22 | 5 | 1 | 2 |
| Process study 2 | 24 | 13 | 7 | 2 | 2 |
| Linkography | 9 | 9, the user or both | 0 | 0 | |
Why the process numbers differ
| Reading | Prompts | Sessions | Window | Effort measure |
|---|---|---|---|---|
| This study | 3,958 | 42 | Oct 31 to Nov 13 | not used for branches |
| Process study 1 | 3,928 | 40 | Oct 31 to Nov 13 | message share, main-agent activity, 15-minute bins; allocation by session and time window |
| Process study 2 | 3,708 | 37 | stated Oct 31 to Nov 13; its script runs to Nov 14, 23:00 UTC and includes 69 prompts of Nov 14 | occupied 10-minute prompt bins; allocation by keyword and carry-forward |
Thread shares differ widely: the intro path holds 16.7% of prompts in process study 1 and 2.8% in process study 2, the level video 9.7% and 3.5%. None of these measures is the user's attention. An inclusion ledger that reconciles the three corpora has not been made.
How the links of the linkograph were found. Its links were first found automatically from shared words, and only selected candidate pairs were checked one by one by the coding agent. Of the checked automatic links, 43% were real building on earlier work, and links that are not shown may still exist. The linkograph uses whole episodes as steps, which favours the user, because episodes are organised around what he asked for.
9Counts that differ
Some things were counted more than once, by different rules, and the counts do not fully match. The study gives the count it uses; the others are here.
Helper agents. Agents in Claude Code started helpers 31 times, and 30 helper logs exist; they made 3,440 tool calls. The count of work alone (the study, ) has 33 helpers, because it adds three sessions that agents had started, with 3,954 calls. A later count by file found 33 helpers with 4,259 calls; that difference was not resolved.
Prompts by model. The table of models counts prompts by the model that answered them (3,068 + 500 + 388 = 3,956; two prompts have no model there). The stop and cost tables count by the model of the run (3,073, 497, 378 and 10 unknown), so their numbers differ slightly.
Pictures. Several counts of pictures appear in the data, because they count different things. They are listed in the box “Why the picture counts differ” in the study, .
10What went wrong when reading the logs
Four things went wrong, or turned out to be traps, when models read the logs. Each left a limit in the study (the study, ). How each was found is told here.
A digest that lacked half of the messages. The first readers worked from a shortened copy of the chat. It lacked 1,931 of the user's messages, about half: 241 that began with a screenshot and 1,690 he sent while an agent was working. Nothing in the first reading's output showed the gap. An agent that traced ideas found it because it met messages it could not place. The second reading followed (). The first process study met the same gap: it found that the daily digests leave out about 43% of the user's messages, those sent while an agent was still working, and it took 8 of its 30 turning points from the raw logs for that reason.
A keyword rule that did not hold. For the purpose of the user's image prompts, a keyword rule agreed with the codes of a model in only 23 of 50 cases. So the study gives no count for all image prompts, only the sample of 50.
A log string with two meanings. The tool logs “The user doesn't want to proceed with this tool use” both for a refused command and for an interruption by a new message. 97 of 109 such events came within 10 seconds of a new prompt, so most are interrupts. Counts that rest on this string were redefined by what the user said and what the tool then did.
Agent text among the user's prompts. About 18,000 of the 366,000 characters counted as the user's prompts were written by agents. All nine long prompts that looked like agent texts were checked, and all nine contain such text. Only one of them did he paste on himself. Six are messages from other agents that the terminal put into his input line, five of them joined to a few words he was typing. Two are start orders for sessions that agents opened themselves. A second search found seven shorter cases of the same kind. All of them are still counted as his. In the role coding such text counts as the agents'.
11Corrections
The numbers of the study were corrected several times after it was first published. This section lists the corrections and the results that were not redone afterwards.
First published counts. The page was first published with 1,960 prompts, 272 stops, 74 context summaries and 17,966 tool calls. Three counting errors were found on Nov 14: continued sessions counted twice, prompts beginning with a screenshot left out, and messages sent while an agent worked never read.
The day-by-day check. On Nov 16 a check of all 265 episodes against the raw logs, day by day, left 117 unchanged and made 23 major, 117 medium and 152 minor corrections; it raised several counts of findings and changed some labels. It also found one helper counted as a main session and up to 10 prompts possibly counted twice through a Codex helper; the second could not be checked, because the event file is missing.
Results not redone. Three results were made before the day-by-day check and were not redone: the sorting of routine episodes, the token shares by label, and the comparison of the two readings. The sorting of routine episodes by main kind of work was done on the 100 episodes that carried the label then; 99 of them still do, and 113 carry it now. The token shares by label use the labels before the check and cannot be recomputed, because the event file they need is missing. They are shown below and no longer in the study.
12Reviews, adversarial readings and reanalyses
The study was read, argued with and redone in part by further agents. None of them was a human from outside. This section lists the reviews, six readings with assigned briefs, two reanalyses by another model family, and the user's own reading of the drafts.
Reviews. Two agents of the same family reviewed the study and checked samples against the chat. Their corrections are built in, and the wrong attributions they found favoured the user. A review by a model of another family led to the reframing of Nov 16, from whose contribution decided to who held which role. An overall review by a fresh instance of the writing model on Nov 17 led to the present order of the study. An independent review of that build by a model of another family followed the same day. It checked the cited passages of the sources added on Nov 17, in full text where it could open them; for two of them it could read only the abstract. The builder had read only abstracts or first pages of those sources. Sentences that claimed more than a source supports were narrowed.
The user's review. The only human who reviewed the study was the user himself, its subject and commissioner. He read the drafts, found errors of presentation and structure, and asked for independent checks and for corrections that went against him. This is a participant review (member check) [45], not an independent one. No outside human has reviewed the study.
Six readings with assigned briefs
Six further agents got the data without this page. Two were told to argue for the user, two against him and two to stay neutral. Their verdicts followed their instructions, which shows how much a reading depends on its brief. This is a test of sensitivity, not a check of the counts, which they took as given. Their strongest challenge stays open: the agents' many small design choices are not counted (). Two of the one-sided readings are in the box below, without weights or verdicts. This study, too, was made at the user's request.
Two advocacy readings
Deliberately one-sided. Two agents were each told to argue one side from the same data, and these texts are what they wrote to that brief. They are not evidence and not the view of this study; their numbers come from the earlier version of the page. The letters mark how firm each agent said its claim was: A follows from the data, B depends on a definition one can hold, C only works by leaving something out.
Two reanalyses by another model family
Two instances of a model of another family, gpt-6-astra in Codex CLI at its highest reasoning level, each made a small study of their own. They got the daily digests (summaries) of the chat, the raw logs and the code. They did not see this study or its readings. One drew a sample of 56 of the user's entries and followed 5 chains of decisions. The other drew 70 entries and followed 8 chains. They used the same archive and the daily digests from the main pipeline, so they are separate interpretations of the same material, not repeated measurements on new data.
Both came to the same answer. The strawberry was made jointly, under the user's direction and built in large part by agents, and the data allow no split in percent. The second study put it as “user-directed, jointly designed, and substantially agent-built”.
They weigh two things differently from this study. First, agents made many visible design choices at small scale. Examples are the camera path around the eaten fruit and the structure of the next level of the film. Another is a first eating animation with 21 bites, which the user cut to 4. The count of ideas in this study () does not catch such choices. Second, the user also brought in technical concepts. One is a fast preview of a cut, whose mechanism the list of ideas credits to the agents (). The others are a rule for one global resolution and parts of the fruit that can be selected one by one. All three can be found in the code.
Their limits are small samples, one coder each, shared access to the same digests, and neither of them watched the film. Their briefs are described in the tab “How this study was made”.
13What the checks caught
The process data list 33 errors in results or text. follows one number through three checks. sorts all 33 by what was wrong and by who caught them first.
Counts were the most common kind (12). Most were found by reaching the same number on two paths: a second script, a reviewer recounting, a cross-reader adding up parts. The largest error was silent. A filter in the digest dropped about half of the messages, and nothing in the first reading's output showed the gap. An agent tracing ideas found it because it met messages it could not place. It led to the second reading.
Claims and caveats came next (9). Several came from rewrites: shortening, smoothing and plain-language passes removed caveats or made scoped claims general. Each time, a check that compared the text before and after, or a reader of another family, found it.
Checks by the other family found six errors alone and five more together with a Claude reader; a sixth, first raised by the adversarial Claude readers, was confirmed by the other family's own studies. Among them are the wrong denominator in section 6.4 and the point that the replication and the prompted readings share data with the study. Four problems of layout and structure were noticed while looking at the live page, outside any agent check. Of the 33, 23 were corrected, 7 were disclosed or narrowed, and 3 wait for a reframing of the study.
Four problems were operational, not errors in results: the live page was blank for about 90 seconds after a build step, a translation agent wrote a helper script into the wrong folder, two other-family runs stopped at their usage limit, and the working folder was deleted once.
More detail: all 33 errors in the order they were found
14How disagreements were settled
When outputs disagreed, the process used four ways to settle it.
- Back to the data. When two counts or two agents differed, the count was rerun or the chat was read again. In the findings check, two agents disagreed about where one idea came from, and the main agent settled it by reading the chat itself.
- Measured agreement. Where two coders labelled the same items, the agreement was reported, not argued away: 78 to 81% between the two readings, kappa 0.86 for the subtypes of routine work. The second reading replaced the first.
- A consensus agent. In the style rewrite, a fourth agent merged three independent versions into one.
- Keep both. The adversarial and neutral readings were not merged. All eight agreed on the counts and differed on definitions, so they stay on the page side by side.
15Limits of the process
- One family checked itself. Most reviews were done by Claude models, the family of the agents the study describes. Three other-family runs ended without a result, two of them at the usage limit, and their tasks went to Claude models.
- Dependent checks. Later analyses built on episodes cut by earlier AI readers. The two independent studies used the same logs and the same episodes, so they replicate the interpretation more than the measurement.
- Not blind at the end. The blind test was judged by the main agent, which knew which model had answered.
- Briefs shape results. Adversarial readers were asked to argue one side, so their texts were extreme by design. Questions put to the agents could carry a preferred answer, and the agents tended to agree with the brief.
- Public while built. Drafts were public from the first hour and were linked under the comments the study later analysed.
- Drift. Numbers drifted between copies of the same text, and rewrites dropped caveats more than once.
- No version control. The working folder was lost once and rebuilt from the live site and the logs.
- This record. It was reconstructed and cut into steps by one model.
Archived from Appendix B: , reach and members
This figure stood in Appendix B of the study. It is kept here as a record, because the drafts were public and linked under the comments while the posts were read. Views and members are counted in different units and from different sources, and the figure does not show that views turned into members.
16Where human judgement was missing
Every judgement in the study was made by models or by the commissioner, whose reading of the drafts was a participant review (member check), not an independent one. No second human coded, rated, interviewed or reviewed. The 787 image prompts with 869 images were never coded by a human. A control run, in which a fresh agent gets only the first brief, was proposed and not done. lists where a second human would have been needed.
17What we would do differently
- Build and validate the complete corpus first, with counts per source and per message type, before any reading.
- Write the questions and coding rules down before coding, including rules that do not decide the main result by definition.
- Keep one source for every number and generate all texts from it.
- Draft privately, publish after review, and do not link from the threads under study.
- Put the study folder under version control with backups from the first hour.
- Have a human second coder for a random sample, a blind judge for the blind test, and at least one human reviewer outside the project.
- Use other-family models from the start for a subsample, and budget their quota.
- Plan a comparison that is fair to both sides: the same target with a fixed final specification against step-by-step refinement, with repeated runs and blind ratings. A run that only fulfils the first, narrower request and stops is no fair control (the study, ).
Data for the figures: methodik.json, derived from the process reconstruction without session identifiers or private content.
Not affiliated with or endorsed by Anthropic. Commissioned by the user it describes.