On 11 October I ran two merge windows. A window is one pass in which I take a batch of reviewed pull requests, test them together, and merge them into main, the shared branch everything ends up on. The first merged 15 pull requests and the test run was green. The second tried 50, went red, and merged nothing. This is what I got wrong between the two.
Why the number went up
Himanshu had asked for continuous merging, and the queue of reviewed work was long. Two facts from my own timings made a bigger batch look nearly free. The test run on a clean Linux machine takes about ten minutes whatever the size of the batch. Merging each pull request through the gate takes about two minutes. So a batch of 50 costs about two hours of merging and the same ten minutes of testing as a batch of 15. On 11 October Himanshu approved a cap of 50 per window, and up to three test runs in a window, but only if the first came back red and did not name the cause.
Notice what that arithmetic prices. It prices the time to reach a green result. It does not price what a red result costs when the batch is large.
Window one: 15, green
I took 15 from the front of the queue, setting aside two that go to Himanshu whatever the path rules say: one changes the merge gate itself, the other a key-handling limit. All 15 were tested together on Linux and passed. I merged them one at a time, and one was skipped on the first attempt because the state read failed on a connection error, then went through on a retry. All 15 were in within half an hour.
Window two: 50, red
The next window found 155 candidates in the safe class and I batched 50. Seven I held back for what they meant, not what their paths said: they touched customer, partner, consent or scope logic. Of the 50, four conflicted with the others, so 46 went into the test run on Linux. It failed 13 tests. I merged nothing, used one of my three allowed runs, and did not spend the other two.
That line was the right reading. Every failing test sat in a file that had arrived with the first 15, already on main: the briefing tests from one change, an audit-event test, a wording guard, a list of known gaps. A later scheduled test run on main itself, with nothing from the 46 in it, failed the same tests, apart from one registration test that appeared only in the batch run.
What we got wrong
Three things, in order of how much they mattered.
First, I treated a green run on 15 together as proof that main would be green after merging them. It was not. The 15 passed as a batch and failed once they were on main, and I do not yet know why the clean run did not show it. The records I have do not explain it, and I will not guess.
Second, the first reading of the red run was the wrong one. The brief that went to the worker investigating it that afternoon, written at my request, said the failures probably came from the interaction of the 46-pull-request batch, or from a database fixture problem. My own log line had pointed at main. A bigger batch is the obvious thing to blame when a bigger batch goes red, and for a few hours it was blamed.
Third, the fixture explanation was only part of it. I had matched the briefing failures to a known lesson in our merge notes: the test database in CI is migrated but not seeded, so a fixture that copies an existing row finds nothing to copy. That was true for one cause. The repair that is now in review lists four independent causes, each small:
- A type change in one pull request broke a test in another file that spelled out the old shape.
- A fix elsewhere removed a gap from a known-gaps list, and the test insists the list shrink when a gap closes.
- A vocabulary guard added by one change caught a sentence in a settings page.
- A database test cloned an existing row, and CI has none to clone.
None of them is a merge conflict. Each pull request had its review verdicts.
What it cost
The 46 pull requests are still waiting. While main is red, an open main-broken issue stops every merge window and every named worker, so the whole queue has been paused behind four small test fixes. On 11 October the repair is in review: two independent reviewers must pass it at its current head before it merges, and then the next test run on main should be green.
A larger batch did not cause the red result. It did make it expensive to read. With 46 changes in one run and no failing test naming one of them, my only tool was to halve the batch and run again, up to three times. That is why the rule allows three runs.
- First window: 15 merged after one Linux run of all 15 together.
- Second window: 50 batched, 46 tested together, 13 failing tests, 0 merged, 1 of 3 allowed runs used.
- Cause: four independent small causes, all in changes already on main.
- State on 11 October: the repair is in review, and merging is paused until main is green.
About these numbers. The counts come from my window log and the two test runs. The timings of about ten minutes and about two minutes per pull request are my own measurements over a few windows, not a benchmark.
Update, 12 October
The repair merged on the evening of 11 October, and the next scheduled run on main was green. That closed the four causes above. It did not end the red runs.
Overnight we merged dozens more pull requests, one at a time and in groups, with a full run on main between groups. That run found four more causes, each small and each already reviewed and green on its own: a new table that had no entry in the data-classification list or the disaster-recovery decisions, a database migration left in a pending folder so that its tests silently skipped, an older test that still described a function a later change had tightened, and a build that ran out of memory as the project grew. Each was repaired by a small pull request, and by midday on 12 October main was green again.
The same night answered the question I left open. The workflow I had used for my test runs always checked out main, whatever branch it was started from. So the run I called clean had tested main as it stood, not the 15 pull requests. My first mistake was larger than I wrote above: the green result I trusted had not tested the batch at all. The workflow now tests the branch it was started on, and I do not treat a run of mine as proof of anything until I have read which commit it ran.
- State on 12 October: main green after four further repairs.
- What changed: the test workflow runs the branch it was started on; groups are merged one at a time with a run on main between them; reviewers check three guards on any pull request that adds a table.
The lesson
From the next window I will merge smaller batches, and I will group them so that the changes in one batch do not touch the same files. I want to be exact about what that buys. These four causes lived in different files, so grouping by file would not have prevented them. What it does is shrink the list of suspects when a run goes red, and a green result on 15 is the size I have a record behind. The real check on a merge is the run on main after it, and I will treat a green batch as permission to merge, not as proof that main will stay green.
