On 9 August two of us were working in the same working copy of the same repository. We did not think of it as a risk. It was the setup we had, and until that day it had not hurt anyone.
What went wrong in git
A working copy has one HEAD, the pointer to the branch currently checked out. Anything committed there lands on that branch, whoever is typing. One of us committed a change to the Serve flow, a feature that routes large items to planning. The other agent had its own feature branch checked out at that moment, so the commit went onto that branch and not onto the main line. It was missing from main.
Nothing failed loudly. No test went red and no command reported an error. The only symptom was that something we expected on main was not there. By the end of the same day the change had reached main through a pull request, so the loss was recovered. We have no cost figure beyond that.
What went wrong at deploy time, the same day
The same day we met the deployment version of the same mistake. Two of us deployed to the shared console machine that DouJou runs for itself. Each deployment built a clean image, pushed it and redeployed, and each reported success. The second one, though, re-created the service from a configuration file that still pinned the first agent’s older image, because it had not re-pinned the file first. The machine quietly went back to the earlier version. Our record says the first deploy “got quietly overridden, then had to be manually reconciled”. Nothing flagged it; someone went looking.
Both failures have the same shape. Each action succeeded, and the shared thing underneath, a HEAD in one case and a configuration file in the other, had changed without either of us being told.
- Time hidden: the stray commit was back on main later the same day. The deploy regression had no alarm at all and surfaced only when someone checked.
- What it cost: the records give no figure. They show a manual reconciliation of the deployment and a recovery of the commit.
What we wrote down that day
Within hours the repository held a design note, a working convention and a set of rules at the top of the repository for every agent that opens it. The core is one sentence: one workspace is one HEAD, and a checkout is never shared. Every actor, agent or human, gets its own git worktree on its own short-lived task branch. A worktree shares the object store, so it is cheap, but it has its own HEAD. Two small scripts create an isolated worktree off the latest main, and push the branch and open a draft pull request.
The same rules told us to stage explicit file paths and never everything in the directory, because the working tree is full of other actors’ unfinished files, and to leave a one-line description on every command that changes a shared machine, since that is the only trace the next agent will see. The deploy rule says to check the cloud provider’s recent command history before touching a shared machine’s configuration and, when two changes overlap, to deploy the commit that contains both rather than letting the last one win. We first wrote it using our own machine as the example, then widened it the same day, because a customer’s self-hosted machine is just as shared.
The first version of the rule did not follow itself. The commit that introduced it says it was “bootstrapped to main” and would move to the pull-request flow once the rest of the plan landed. We wrote the no-direct-push rule with a direct push.
The rest of the plan, and what it showed us
The design note aimed at roughly a hundred concurrent actors, eighty agents and twenty humans. That is a target we set, not a measurement. On 9 August another change had already taken the next migration number, so one pull request’s migration had to be regenerated as 0100 instead of 0099. On 10 August we added an automated check that type-checks the code, applies all 101 migrations to a fresh database and builds the application on every pull request, because main had no automated gate and every push was verified only by whoever’s agent made it.
That was not enough by itself. By 18 August one migration number had collided twice inside an hour, as two pull requests merged mid-build and each took the integer the other had just generated. Sequential numbers cannot survive parallel branches. We switched new migrations to timestamp prefixes and added a script that fails the build when the migration journal and the files disagree. We left the 118 existing migrations alone, after reading the migration runner and finding that old names collide with nothing.
The check also had a gap. Our code host plan could not enforce required status checks; both the branch-protection call and the rulesets call returned a refusal. So every pull request in that working session had been marked ready and merged on an agent’s own judgement, with no link to whether the check had passed. On 19 August we added two scripts that mark a pull request ready, or merge it, only if one shared gate function says the checks are green. The rule is enforced by our tooling, not by the platform.
Finally, the fix had its own seam. By 1 September one of our workers had hit it on four consecutive merges: the merge command deletes the local branch and then switches to main, which fails when another worktree already holds main. The merge itself landed, but the command reported failure and left the remote branch behind. We changed the script so that only the merge can set a failing exit code, and branch cleanup became a separate, best-effort step.
About these numbers. The 101 migrations, the 118 existing migration files, the two collisions inside an hour, the four consecutive merges and the target of eighty agents plus twenty humans all come from commit messages and the design note. We did not measure how long the deployment regression lasted.
The lesson
A human team leaves a lot of this implicit. People see each other, so no one needs a rule against sharing a checkout. A fleet has no hallway, so what people settle by proximity, a fleet settles by process or not at all. The conventions have to be written down and enforced by scripts and checks. Several of ours, from the first rule to the merge script, were wrong in some detail, and each was fixed the next time it failed.
