Where: a customer’s production console, self-hosted in their own cloud account.
Symptom: a customer with 29 GitHub repositories scanned, and 18 applications successfully inferred from their real code, asked Kai a basic question about those repositories. Kai answered as though it had never heard of them. The status line on the Catalog page read, calmly, “index not built yet.”
Everything upstream looked healthy. The scans had run. The evidence was in the database. The application inference, which reads package manifests, Dockerfiles, Kubernetes manifests and API specs, had produced 18 correct results visible on screen. The last mile was dead and nothing was complaining.
Three separate bugs, stacked
Layer 1: Kai asked the wrong source whether it was ready
Readiness was decided by reading a stored counter. That counter was only ever written by the full bulk-reindex path. Any code that inserted content surgically, which is how repository evidence gets in, never touched it. So the counter said zero while the index held real content.
Four independent places in the codebase made the same wrong check: the router that decides which engine answers, the retrieval step itself, the external API gate, and the Catalog page’s own status chip. All four were fixed by replacing the stored counter with a live count of what was actually there. We deployed it, re-tested on the live box, and it was still broken.
Layer 2: the index genuinely was empty, and no path existed to fill it
The code that indexes repository evidence into Kai only ran as a side effect of a fresh scan. The bulk reindex, the thing the “Reindex” button triggers, only pulled from catalog resources. It had no path to repository evidence at all. A customer whose repositories were scanned before that feature shipped could press Reindex forever and never get them into Kai’s retrieval.
Worse, the only automatic reindex trigger in the product was attached to cloud discovery. This customer has no cloud connection. Their catalog is curated and GitHub-only. So the one customer profile that most needed the repository index had no automatic path to it, and the manual path was broken.
We fixed it by backfilling every repository’s latest evidence during any reindex, and by triggering a reindex after any scan that collected anything, scheduled or manual. We deployed, re-tested, and it was still broken.
Layer 3: the database was missing two columns, and had been for three days
Clicking Reindex on the newly deployed version finally surfaced a real error instead of a shrug:
index failed: column "…" of relation "…" does not exist
Those two columns had been added weeks earlier by a feature that gave indexed content a type. They are created by a schema script that runs exactly once, when a database container initialises an empty data directory. That is standard behaviour, not a bug in the script.
This customer’s database was created two days before the columns were invented. The script never ran again, because from the database’s point of view there was nothing to initialise. Every reindex on that box had been crashing on a missing column since the day the feature shipped, and the only user-visible evidence was a status line reading “index not built yet.” We fixed it by re-applying the vector schema on every console boot, the same way database migrations already self-heal on every deploy.
The result
Reindex completed in about ten seconds: 29 of 29 documents, 198 chunks. Kai then answered a repository question with citations at 97% confidence, and correctly worked out that a name the customer asked about was not a standalone repository at all but a Helm chart living inside another repository, reading the README and deployment manifests to say so.
- Time hidden: three days for layer 3. Layers 1 and 2 had been latent for weeks, affecting an unknown number of installs.
- What it cost: roughly a full working day of senior engineering time, three production deploys, and a customer-facing feature that was silently dead on at least one box. The real cost is unmeasurable: nobody knows how many questions Kai declined to answer, because declining to answer is not logged as a failure.
- Why nobody noticed: every one of the three layers reported “not ready” rather than “I am failing.” Those two states are indistinguishable in a UI and opposite in meaning. A system that says “not ready” invites patience. A system that says “I am crashing” invites investigation.
The lesson we changed process over
“Not ready” and “failing” must never render identically. Any readiness check must distinguish “empty” from “erroring”, and surface the error. The fix for layer 3 generalised too: schema that lives outside the migration system will drift on exactly the installs you can’t see, so it must be re-applied on every boot, not on first install.
