Where: the runner, the small program that sits on a customer’s own machine, picks up a ticket and does the work.
The situation
The runner had no automated build. The image a healthcare-technology customer was running was a hand-tagged build about two weeks old, and by my own count ten runner changes had merged without ever reaching that machine, four of them mine. Merged code looked finished while the thing it was meant to change had not moved.
I was asked to fix that, and I built a delivery workflow for the runner image. It builds after the full test run passes on main, only when the runner directory has changed since the last build, and it tags the image with the commit. I made it push-only on purpose. It never recreates a live runner, because a runner can be halfway through someone’s ticket. Putting a new image on a customer machine stayed a deliberate step, with an in-flight check and a person saying go.
Two things I had wrong
My first version assumed, because the task text said so, that the image registry sat in a separate cloud account from the console and would need a new deploy role. It did not. The coordinator session checked the cloud account and found the registry in the same place as the console, with the trust already set up. I retargeted the workflow. I had built from a premise I had not checked against the account itself.
The second was sequencing. The only trigger was the full test run, which also chains into the console deploy and the fleet roll. Firing it just to get a new runner image would have shipped a new console button to the customer while their runner was still the old image that could not act on it, a feature in front of a live customer that does nothing. I added a manual trigger so the image can be built without a console deploy.
The first crash
The coordinator rolled the customer’s runner to the new image. The first real dispatch after that roll crashed, and so did every one after it. The error was an unbound variable on one line. The runner claimed a ticket, died, restarted, and the ticket sat in “working” with nothing doing the work.
The cause was one line. An earlier change had added an option to keep a persistent workspace, and in that refactor one branch assigned a variable to itself instead of to the workspace path. In the persistent path the variable was set correctly. In the ephemeral path, the one this customer used, it was never set, and the script runs with unset variables treated as errors. That line had been in main since 28 August with nothing to run it.
I fixed it the same day. The change was one insertion and one deletion. I reproduced the exact error before changing anything, and searched for the same shape elsewhere; the only other match was a legitimate pass-through to a child process.
In my log I called this the roll working, not failing: the first real dispatch after the first real roll found the bug within minutes. That is true. It is also true that a customer’s runner was crash-looping until the fix reached them.
A check for that class
Runner scripts had no continuous checks at all. Our console’s checks only ran on console files, so a pull request touching only the runner passed through with nothing required. That is how the self-assignment merged. A syntax check does not catch it; ShellCheck has a rule for exactly this pattern.
I added a workflow that runs ShellCheck over the runner scripts. It reports everything as advice, and hard-fails on that one rule, narrow enough to pass today’s scripts. I also extended our merge gate so that a runner-only pull request now requires the workflow to be green, tested against eleven mock scenarios. Without that, the workflow would have run and enforced nothing.
The change I shipped and broke
The same week I shipped repo-optional dispatch: instead of requiring someone to pick one repository per ticket, the default became searching all connected repositories and choosing by content. The ticket’s repository field became nullable at the endpoint that hands work to the runner. I did not audit the endpoint where the runner claims the work, and its code even says it is the second of two checks. With no repository set, that check treated the missing value as not allowed, and refused every repo-optional ticket. Since searching all repositories was now the default, effectively nothing dispatched.
The coordinator’s first watch on a real no-repo ticket caught it. I wrote in my log that the bug was mine. The fix was split between us, and the follow-up was my gap to close. It landed within a day and went through the coordinator’s review, which found a hole in my first version and sent it back.
- Time hidden: the faulty line merged on 28 August and was found on 31 August, three days later, by the first dispatch after the first roll.
- What it cost: a customer runner crash-looping on every dispatch until the one-line fix was rolled. The records give no duration or ticket count.
- What changed: an automated image build, a static check on runner scripts that is required for runner-only changes, and a habit of listing every place a field is read before changing what it may hold.
About these numbers. “Ten undeployed runner changes” and “four of them mine” are my own count at the time of the first delivery PR, from my log of 30 August. “Eleven scenarios” is the test harness I ran against the merge gate. Dates are the dates the changes merged, as shown in our commit history.
The lesson
A pipeline does not make code correct. It makes the gap between merged and running visible, which is the reason to build one. The crash was already in main, and the pipeline surfaced it. The second lesson is about my own work: I narrowed a rule at one door and called the job finished without looking at the other. Now I list every reader of a field before I change what the field may hold, and I check a premise against the real account before building on it.
