Engineering Ask Architecture

Engineering Ask is the assistant on this site that answers questions about how I build software — "why SQLite?", "how does the Photography Assistant prevent hallucinations?", "what did you try and reject?" — with a short answer, the documented decisions it relied on, and links to the exact sections of my architecture articles that describe the implementation. Its premise is the same one that governs the rest of my AI work: a language model writes the prose, but it does not decide what counts as evidence, how confident the answer may sound, or which sources it may cite.

The problem

An engineering portfolio is mostly a list of outcomes. The interesting part — why one option was chosen over another, what it cost, what failed first — usually lives in tickets, reports and my own memory, where a reader cannot find it. I wanted that reasoning to be discoverable, not summarised by a chatbot that sounds informed.

That framing ruled out the obvious design for four reasons.

• The answers are about one person's undocumented-by-default work. No model knows why I chose a particular reranker. Anything not in my own records is, by definition, invented.
• Two kinds of truth matter, and they are easy to confuse. Why a decision was made lives in reviewed decision records. How a system works lives in published architecture articles. An answer that attributes one system's mechanism to another is wrong even if every sentence sounds right.
• Readers ask in English and German, and they follow up. "What about the frontend?" is only answerable with the previous question in mind — and dangerous if the previous subject leaks into the new one.
• Confidence has to mean something. A label that says "High — directly documented" on almost every answer is worse than no label at all.

What I built

Engineering Ask runs in production on the home page, behind the "Engineering" island. I designed and built the whole path: the decision knowledge base and its review workflow, hybrid retrieval and reranking, the independent architecture-evidence index, the generation contract, the citation gates, contextual follow-ups, the confidence and stance classification, and the offline evaluation tooling behind every change below.

It is a production system at a deliberately small scale: one author, a corpus of roughly 280 reviewed decision records and eight architecture articles at the time of the most recent measurements, and public, anonymous visitors.

Architecture at a glance

• Entry. Ten questions per minute per IP address and a global ceiling; questions between 3 and 500 characters; a pattern check refuses requests that try to extract the evidence wholesale before any retrieval runs.
• Context resolver and cache. Follow-ups are resolved by deterministic rules over a tiny server-authored context the browser echoes back. A planner model is consulted only when the rules cannot decide. Resolved questions that were answered in the last ten minutes are served from an in-process cache.
• Two retrieval branches. Engineering decisions and architecture sections are retrieved separately, from separate indexes, with different methods — for reasons the measurements below make concrete.
• Evidence pack. At most five decisions and three article sections. This is the only material the answer model and the confidence classifier see.
• Confidence before generation. The confidence label is resolved first, because it is part of the answer prompt.
• Generation and gates. The model returns a typed object; every decision it names and every article section it cites is checked against this request's pack after generation.

The stance indicator is not on this path at all:

Confidence feeds the prompt and therefore the answer; stance only decides whether a small "Lean yes / Conditional / Lean no / Uncertain" chip appears next to it. The section on confidence and stance explains why they are two separate calls.

The key architectural decision

The answer model is the last, narrowest step of a deterministic pipeline. Its job is to write a short answer and a few sentences of reasoning from evidence it is handed, and to say which of that evidence it relied on.

It does not control:

• which decisions or article sections exist, are reviewed, or are published;
• what is retrieved, how it is ranked, or how many items it may see;
• the confidence label, which is resolved before it runs and handed to it as a fixed instruction;
• the stance indicator, which it never sees;
• the structure of the response — a schema enforced at the provider and again in the application;
• which of its citations survive;
• how evidence is presented: decision cards are rendered by the application from the canonical records, in the visitor's language, keyed by id — not from the model's prose.

The model has no tools and no database access. What it can be wrong about is bounded by what the pack contains, and what it can cite is bounded by what the gates accept.

Two kinds of evidence: why and how

The knowledge base behind Engineering Ask is not a pile of documents. Its unit is a decision record: a title, a category, the situation, the options considered, the option chosen, the rationale, the accepted trade-offs, and a small set of controlled tags describing engineering style. Records can name the source they were derived from.

Records enter through an admin workflow. A model can propose candidate decisions from a document I supply — under rules that forbid inferring beyond the text and require supporting quotes — and candidates below a confidence threshold are discarded. Nothing proposed becomes evidence until I review it and save it as a record. Superseded decisions are tracked in a canonical supersession table and removed before ranking, so an answer is not built on a choice I have since reversed.

Architecture articles — like this one — are a different kind of evidence. They describe how a system is built, at a level of detail a decision record should not carry. They have their own lifecycle: an article is published on the site first; its sections are then extracted deterministically into a draft evidence revision; I approve or exclude each section (and may only remove text from it, never add); and only then is the revision embedded and activated, in one transaction, replacing the previous one. An unpublished article, an unpublished revision or an excluded section is invisible to Engineering Ask.

Keeping the two apart was a deliberate choice, not an accident of history. A decision says why; a section says how; an answer usually needs both, and a citation must say which one it is.

Finding the right decisions

Decision retrieval combines two candidate lists.

• Lexical. An in-memory BM25 over nine fields of each record, queried with the concepts extracted from the question (weighted by extraction confidence) plus the question's own words.
• Semantic. The question is embedded and ranked against the decision vectors; only vectors whose content hash matches the current record count, so an edited decision is never retrieved through a stale embedding.

The two lists are merged with reciprocal rank fusion, and the top ten are reordered by a reranking model (Voyage rerank-2.5). The five best become the decisions in the pack. If the reranker is slow or unavailable, the fused order stands.

Each step was measured before it shipped:

• The previous scorer turned out to have structural defects — substring matching that let one shared word count as a match, no term weighting, and a "recurrence boost" that had saturated to a constant for nearly every decision. Replacing it with the fused BM25 + semantic ranking cut answers with a material omission from 28.6% to 12.5% (16/56 → 7/56) in a 30-question, 180-answer comparison, graded blind by a model.
• Reranking the top ten raised NDCG@5 from 0.562 to 0.751 on 23 judged questions, with 17 improvements and one regression. Depth 20 scored higher (0.810) but with more regressions and latency; depth 10 had the best regression profile.

All of these judgments were authored by models, not by me or another person. The human disagreement review prepared for the reranker has not been done, and I treat the numbers accordingly: as evidence that the direction is right, not as precise effect sizes.

Architecture sections as separate evidence

Architecture sections are retrieved from their own vector index, which shares no table with decision embeddings. The question is matched against every approved section; split parts and the English and German versions of one heading collapse into a single candidate; the top fifteen are reranked by the same Voyage model; at most three reach the pack. Like the decision reranker, it fails open.

The interesting result was what didn't work. On a frozen set of 46 questions, semantic retrieval reached nDCG@5 0.628 on the supported questions, BM25 0.339, and an RRF hybrid 0.539 — below semantic on every slice. German was the clearest case: lexical retrieval scored 0.157, because the tokenizer is built for English. So the hybrid that works for decision records was rejected for article sections. Fusion is not a default; it has to win on the corpus it serves.

Two findings shaped the integration:

• No retrieval score separates "unsupported" from "weakly supported". Neither cosine similarity nor the reranker produced a threshold that rejected questions about systems I have not published without also rejecting legitimate ones. Refusing to answer therefore has to come from generation and confidence, not from a retrieval cutoff.
• Names collide. The engineering knowledge base and the AI-native repository's "knowledge layer" share vocabulary, and answers conflated them. A prompt rule that similarly named systems are not the same system reduced, but did not eliminate, that error (see limitations).

Measured end to end on 63 questions (134 generations, graded blind by a model on a 0–2 scale), adding reranked article sections raised completeness from 0.92 to 1.70 and accuracy from 1.54 to 1.87, and cut answers citing the wrong source from 13% to 2%. Decision retrieval, decision citations and confidence were byte-identical between the arms, so the gain is attributable to the new evidence.

Grounded generation and citation contracts

The answer model receives, in trust-labelled blocks: the extracted concepts and the resolved confidence (trusted), a deterministic style profile, the decisions, the architecture sections with request-local labels A1–A3, and finally the visitor's question (untrusted). Article text is neutralised before it enters the prompt, so markup or instructions inside an article are data.

It returns exactly four fields: a short answer, a few sentences of reasoning, the titles of the decisions it relied on, and the labels of the article sections it relied on. Output is generated as native structured output and validated again in the application against a contract that rejects unknown fields, markdown and HTML. The answer streams to the page as provisional text while it is written; only the validated response replaces it.

After generation, two deterministic gates run:

1. Decision attribution. Every claimed decision title must exactly match a decision in this request's pack. Others are dropped; the accepted titles are mapped to canonical ids, which is what the evidence cards are rendered from.
2. Architecture citations. Every label must resolve to a section in this request's pack, and that section must still belong to the published revision it came from. Unknown labels, duplicates, a section from a different article, and a section whose revision has since changed are dropped. The URL — including the heading anchor, read from the same renderer that builds the public page — is constructed by the server. The model never writes a link.

Both gates drop and record; neither retries. A dropped citation costs a reference, not the answer. In the integrated evaluation, 230 architecture citations were claimed and 230 accepted.

It matters to be precise about what this guarantees. Every decision card and every article link shown under an answer was retrieved for this question and is published and reviewed. What it does not guarantee is that every sentence of the prose is true: the prose is constrained by the evidence it was written from, the contract and leak redaction, but it is not fact-checked sentence by sentence. And the confidence label — discussed next — measures the evidence, not the answer.

Follow-up questions

Engineering Ask is conversational, but the server keeps no session. After each answer it returns a small context: the question that was actually executed and the ids of the decisions it used. The browser echoes that back with the next message. It is treated as untrusted input — strictly validated, checked by the same escalation gate, and filtered to decisions that still exist — and a malformed context degrades to a standalone question rather than an error. No previous answer text, confidence or stance is ever sent back.

Deterministic rules resolve the turn first: a refinement ("what about accessibility?") is composed with the previous question; an explanation ("why?") is scoped to the previous decisions; a new subject starts fresh. A planner model (Claude Haiku by default, one call, temperature 0, a closed set of four actions) is consulted only for corrections, references back ("that one"), and subjectless or pronoun-dependent turns. Its output must name words that actually appear in the conversation, and on any failure the deterministic plan stands. The planner can never choose which decisions are in scope.

The measurements made the trade-off concrete. On 65 agent-labelled follow-ups, deterministic rules alone resolved 66.2% correctly; with the planner on ambiguous turns, 87.7%. But the production planner, like rules alone, failed both "what about his voice interaction system?"-style topic switches: it concatenated the old subject onto the new one ("How is the frontend architecture of the Photography Assistant built?"). The only experimental arm that got one of them right was the one with an added deterministic rule, and a later prompt rule did not stop the planner. So an explicit new subject is now resolved deterministically as a fresh question and never reaches the planner — a case where removing the model was the fix.

Two more safeguards address contamination. On a refinement, the composed question can carry the previous thread's evidence for a subject the visitor has just introduced; a deterministic confidence recomputed from the current turn's own concepts caps the final label, so inherited evidence cannot vouch for a new subject. And the cache key includes the visitor's own message on such turns, so two different follow-ups to the same thread are never served the same answer.

The "Explore further" suggestions under an answer are built in the browser from fixed English and German templates over the cited evidence — no model call. They only fill the composer; the visitor still chooses to ask.

Confidence is not stance

Engineering Ask shows two labels, and they answer different questions.

Jev is a small hosted classification model: it answers a closed-choice question about a supplied state and returns a label with a probability for each option, not prose. Both tasks use it, with separate contracts, separate requests and tests that hold that neither outcome moves the other.

Confidence is evidence-pack sufficiency. It does not judge the generated answer — it runs before there is one. It answers: could these decisions and sections answer this question?

It got there in three steps:

1. A rule that labelled almost everything HIGH. The original deterministic rule scored a question against the whole corpus, so some decision matched almost any concept, and one shared word was enough. In production, 73 of 81 answered questions were labelled HIGH; in a 30-question sample, 27. Scoring against the pack and requiring every core concept to be documented brought the sample to 10 HIGH, 17 MEDIUM, 3 LOW. A manual review of the sample found no remaining HIGH over-calls, but about five well-documented, paraphrased questions were now under-called.
2. Jev, with a regression. In an offline experiment on 50 questions, Jev labelled 42–43 correctly versus the rule's 23 — but it produced five false HIGH labels to the rule's three, all on partially documented questions. A better average with more of the most damaging error was not shippable.
3. A pre-registered floor and an independent holdout. The false HIGHs all sat below a probability of about 0.66, and that probability drifted between runs. I fixed the rule before looking at new data: Jev's label, with HIGH kept only at p(high) ≥ 0.70 (the smallest value with zero false HIGH in all four development runs). On a frozen holdout of 62 new questions, in both repetitions:

The resolution can only lower Jev's label: a HIGH below the floor becomes MEDIUM; an empty pack is never HIGH; a gap the deterministic rules detect caps HIGH at MEDIUM; and on a contextual follow-up the current-turn ceiling applies. If Jev times out (1.5 s), is unavailable, or returns anything off-contract, the deterministic label is used.

Why awaited, not concurrent. Starting confidence in parallel with generation would be faster, and an earlier experiment considered it. But the label is in the prompt: it tells the model how strongly to state the answer. Running it concurrently would mean generating under one label and showing another. So every reasoning answer waits for it — about 0.3 s at the median (p50 277 ms, p95 374 ms over eight end-to-end questions) on answers that take around five seconds.

Stance is optional opinion. For a question like "does Vikram prefer deterministic safeguards?", the evidence can lean one way. Most questions are explanatory ("how does…"), and there the honest stance is "not applicable" — no chip. Stance used to be computed by rules and fed into the prompt; removing it from the prompt made no measurable difference to answers, and moving it to Jev reduced chips on 40 explanatory questions from 40 to 2. In 48 production-path runs Jev never finished after generation, so stance adds no visible latency. Its failure costs the chip, never the answer.

What failed, and what the evaluations showed

Hybrid everywhere. Hypothesis: the BM25 + semantic fusion that improved decision retrieval would improve article retrieval too. Result: on 46 frozen questions it scored below semantic alone on every slice, and German lexical retrieval was near-useless. Decision: semantic + rerank for sections, hybrid for decisions.

A cheaper planner, a wider boundary. A GLM planner would have saved one or two cents per thousand follow-ups; it failed a predeclared gate and was slower at the median on the follow-up set, so it was not adopted. Sending more turns to the planner recovered some follow-ups but also bound some fresh questions to the previous thread. A narrower boundary shipped instead, alongside the deterministic topic switch.

Prompt fixes that weren't. Two prompt changes intended to improve answers about architecture were measured and reverted: neither improved them.

A post-hoc threshold. In the first confidence experiment, a p(high) < 0.6 rule happened to remove every false HIGH. Re-running the same questions showed the probabilities drifting enough that it would not have held. The threshold that shipped was chosen on development data, registered with a timestamp, and only then tested on questions it had never seen.

On the labels. Every quality number in this article comes from offline evaluation on frozen inputs, and none rests on human ground truth. Relevance judgments, follow-up expectations and confidence labels were written by AI agents and are provisional; answer quality was graded blind by models. Where I could, I separated authoring from judging, pre-registered rules and used fresh holdouts. Human review of the borderline confidence labels and of the reranker's disagreement cases is the most valuable outstanding work.

Security, caching and cost

• Trust boundaries. The question and the echoed context are untrusted; retrieved text is data; only application-computed values are trusted prompt context. Requests that try to extract evidence wholesale are refused before retrieval, the output contract forbids markup, and leaked internal identifiers are redacted from both the streamed and the final answer.
• Public payload. The browser receives the answer, the confidence, the optional stance, decision ids and validated article citations — not the retrieved items, concepts or the internal confidence source.
• Caching. Answers, extracted concepts, rerank orders and the published article view are cached in process for ten minutes. The answer key includes a corpus version that any decision or article-evidence change bumps, so an edit is never masked by a stale answer.
• Observability. Each request records span attributes — candidate counts, rerank and citation outcomes, confidence source, classifier durations — without visitor text, plus a first-party log of public questions and outcomes.
• Cost. The two Jev calls together were measured at about $0.00023 per reasoning answer; the decision rerank at about six to nine cents per thousand questions. Neither is a reason to remove a safeguard.

Trade-offs and limitations

• Conservative HIGH under-calls. The floor costs correct HIGH labels: on the holdout, raw Jev would have labelled 9–10 more questions correctly. Ten of the shipped rule's 19 under-calls are HIGH shown as MEDIUM. I accept the occasional modest label to avoid a confident one the evidence does not support.
• Partial coverage reads as LOW. Eight under-calls are partly documented questions ("…and what does it cost per month?") labelled LOW rather than MEDIUM. That understates coverage, never overstates it, but it is a real imprecision.
• Confidence is not factuality. A HIGH label says the evidence documents an answer, not that every sentence of the answer is correct.
• Every answer waits for confidence. About 0.3 s at the median, bounded at 1.5 s.
• Name conflation persists. German questions about the engineering knowledge base can still borrow vocabulary from the AI-native repository's knowledge layer.
• Generation failure is an error, not a fallback. There is no deterministic answer to fall back to; a failed generation shows "No grounded answer available". The only retry is a provider fallback chain, where one is configured.
• Single instance. Rate limits and caches live in process memory: correct for one instance, wrong for several.
• No human ground truth yet, as described above.

What I learned

The biggest improvements did not come from a better answer model. They came from deciding what the model may see — five decisions, three sections, from two indexes that each earned their method — and from separating questions that had been blurred together: why versus how, evidence sufficiency versus answer truth, confidence versus stance. Once the fix was to take a model out of a decision (topic switches); once it was to put one in, but only behind a threshold registered before the data that tested it. The pattern underneath is the one I keep returning to: let the model do the language, and make everything it depends on checkable.

Further reading

Engineering Ask cites the other architecture articles as evidence. The closest relatives are the Photography Assistant, which applies the same grounding principles to a photo archive, and the AI-Native Repository, whose knowledge layer is a different system from the decision knowledge base described here. To try it, open the "Engineering" island on the home page.