Where: the first pilot, a hand-run install of the DouJou console inside the cloud account of a healthcare-technology customer.
Symptom: none at the start. The install worked. Ten minutes after we recorded it as live, the second page a new administrator reached returned an error.
A rehearsal in our own account
On 6 June we wrote a plan to run the whole product in DouJou’s own cloud account as a live test site, and the same afternoon we chose its cost posture: lean sizing, and stop everything when nobody is using it. The commit message put the cost at rest at a small fraction of leaving it running all month.
That afternoon the rehearsal paid for itself. Four deployment fixes landed within a few hours. Two were about connecting securely to a managed database. In one, the migration tool exited with an error in the slim production image and put the container into a crash loop. In the last, redirects behind the reverse proxy pointed at the internal address instead of the public one.
The install day
The pilot itself happened on 21 June. We had drafted an install runbook earlier (the text is dated 19 June), but the first runbook commit, a stub for the customer’s record, and the entry recording the install as live are all from 21 June, about an hour apart by their timestamps. The customer had created a user for us in their account and had already switched on access to the managed model service. We verified that with a real call rather than a listing.
The design we ran was different from the plan for our own site. One virtual machine in the customer’s account, with the console, a bundled database, an embedding service and an automatic-HTTPS reverse proxy as containers. The database was bundled rather than managed because the runbook calls that the cheapest choice that is fine for a pilot. The embedder ran in the box so document embeddings stayed in the customer’s infrastructure. AI calls went to the cloud provider’s own managed model service through the machine’s role, so no model keys were stored anywhere. The hostname came from the machine’s public address through a wildcard DNS service, so the proxy got a real certificate with no DNS work for the customer.
What went wrong
Several things, in a row. Each one went into the runbook.
- With a bundled database, the migrator decided it needed a secure connection and failed on its first statement. The real cause was swallowed by the library. We had to set the connection mode explicitly.
- Calling the model service by its bare model name failed with an error about on-demand throughput. It needs the regional routing identifier for the model.
- On our operator machine the shell and the cloud command-line tool disagreed about file paths, so policy documents in a shell-style temporary folder could not be read.
- One public image lookup returned not found, so we searched for the image another way.
- The runtime image has no seed tooling, so the first administrator was created by copying a small self-contained script into the container.
The sixth we describe separately, because it was the one we had already written down as finished. The record said the install was live and a login had been verified. About ten minutes later, by commit timestamps, we fixed a server error on the onboarding wizard, the page a new administrator lands on straight after logging in. Its source-control step signs its connection state with a second encryption secret, separate from the first, and we had not set it. We added it on the box, recreated the console, confirmed every page returned a normal response, and put it in the runbook.
Our check was whether login worked. A customer’s first minutes are the login and then the next page.
Stopping when idle, and a button to show what a scan sees
Within about half an hour of the live record, the box was stopped to save cost. The public address was kept so the URL and certificate would not change, and the database lives on a disk that persists, so a restart brings the stack back. On 22 June we started it for a demo, confirmed it was healthy, switched to a name the customer had pointed at the box, redeployed the current build, and stopped it again once the demo ended. The runbook made the same point: a stopped box costs very little next to a running one.
Also on 21 June we added a “Debug” button beside “Scan now” on each cloud connection. It runs the estate scan as a dry run, writes nothing to the catalog, and prints every step, resource and error verbatim. Run against a real account, it listed seven resources. The commit says what it is for: to see exactly what a scan finds and what it does not. Later that day we extended it to the other two cloud providers. That commit says only the first provider’s path had been verified live and that the others share the same code path. We said so then and we say so now.
What changed in how we work
The runbook has a marker for each problem we hit on this install, and by our count from those markers there were six. The first pilot was not decommissioned because it failed. On 17 July the hand-run machine was replaced by an install through our automated deployer, and the lessons above were baked into its template rather than left as steps for an operator to remember. The runbook now opens with a banner saying it is superseded and kept for reference. It also notes that creating the first administrator was still a manual step at that point.
- Runbook commit to live record: about an hour, by commit timestamps, on 21 June.
- Live record to first fix: about ten minutes.
- Problems hit on the first pilot: six, by the runbook’s own markers.
The lesson
A runbook drafted before the first install is a list of guesses, and the install is what turns it into a procedure. Rehearse in your own account first, so the dull failures happen where nobody is watching. Then do not call an install live until the page after login works.
About these numbers. The timings come from commit timestamps, which record when we wrote things down, not when each step started. The count of six is our own count of the runbook’s markers for problems hit on this install.
