Observability Architecture

My portfolio answers questions about my photographs and about how I build software. When one of those answers is wrong, slow or missing, the cause can sit in very different places. It might be ordinary application code, a retrieval rule, a model provider, a model that wrote something its evidence doesn't support, a validator that rejected it, a speech model running on the visitor's own laptop, or a decision I made months earlier for reasons nobody wrote down. This case study is about how the system makes each of those places inspectable, and where it can't yet.

It is not a tour of monitoring tools. The application has no dashboards, no alerting and no error-reporting service. What it has is a small number of deliberate observation channels, each built for a particular kind of failure, joined by one identifier where joining them is useful. Several of its blind spots are as important as anything it can see.

The problem

A conventional web application fails in a few recognisable ways: an exception, a slow query, a bad deploy. A request log with status codes and durations covers most of them.

A compound AI application fails in more ways, and most of them return HTTP 200:

• Retrieval returns nothing, or the wrong photographs, while every component works as designed. A strict rule doing exactly its job looks identical to a bug from the outside.
• A model provider is unavailable and a fallback model answers instead. The visitor sees an answer, and nothing says it came from the second choice.
• Generation produces valid JSON that makes a claim its evidence doesn't support. A validator rejects it, the system retries, and the final answer is fine. The rejection is invisible unless something records it.
• Validation refuses to answer at all. A refusal looks like a working feature.
• Speech recognition runs in a Web Worker in the visitor's browser. When it falls back from the GPU to the CPU, or fails, the server never knows.
• Quality regresses slowly. A ranking change can improve an average and hurt the queries that matter, and no runtime signal can tell you which photographs a person would have wanted.

So the question this architecture answers is narrower than "is the system up". It is: when an answer is wrong, can I find out which layer made it wrong, without having been watching at the time?

Why request logs are not enough

Every tRPC procedure emits one timing line: procedure name, duration, success or failure. That tells me a request to the Photography Assistant took nine seconds and succeeded. It doesn't tell me whether the nine seconds went into embeddings, the reranker or a second generation attempt. It doesn't say whether "succeeded" means an answer or a refusal, or how many candidate photographs existed before a filter removed them all.

Adding more log lines doesn't fix that, because the missing information has structure. A retrieval pipeline is a tree of stages, some of them running in parallel. A generation can have two attempts, each passing or failing four separate gates. A useful record has to keep that structure and attach it to one request. It also has to do that without copying the visitor's question, the prompt or the model's prose into a third-party system.

Five planes, not one pipeline

The repository doesn't have one telemetry pipeline. It has five places where evidence lands, each answering a different question, plus one important place where evidence doesn't land at all.

Three planes observe a request while it runs, and one id joins them:

The other three stand apart from any single request:

• Offline evaluation: frozen database snapshots, judged benchmarks and a tie-neutral protocol, run before a change ships.
• Engineering record: reports, decision records and validator output in CI, explaining why a change shipped.
• The browser: the speech model, its GPU or CPU backend, fallbacks and playback. All of it stays on the device and is reported nowhere.

The first three planes are runtime observability. Evaluation and the engineering record are about decisions: they explain why the system is the way it is, not what happened on a given request. The browser is the deliberate gap.

Runtime signals

The runtime layer is plain and partly incomplete, and that is how this article treats it.

What exists:

• One timing line per tRPC call, from a middleware every public and admin procedure passes through, plus one line per pipeline stage wrapped in the shared timing helper. Each line carries the stage name, duration, success or failure, the kind of external call involved (Claude, OpenAI, Cohere, the database), a cache status where one applies, and the request's trace id.
• One usage line per model call from the shared structured-generation layer: which feature called it, prompt version, model, provider, input and output tokens, prompt-cache tokens, cost where the provider reports it, and the fallback route if one was taken.
• Process metrics every 60 seconds: event-loop delay and memory, from inside the Node runtime.
• A liveness probe that the hosting platform calls every 30 seconds. It deliberately touches nothing (no database, no rendering, no provider), so it stays responsive while the event loop is busy with a long AI request. It answers "is the process alive", not "does the application work".

All of this goes to standard output. The hosting platform keeps it for a while. Nothing aggregates it, and no alert reads it.

What doesn't exist:

• No error-reporting service and no structured logger. Log lines are prefixed text, not JSON.
• In production, an error thrown from a tRPC procedure is not logged centrally: the route's error hook is enabled in development only. The timing line records that the call failed, but not why.
• No custom error page and only one React error boundary, around the 3D avatar, which renders nothing on failure and reports nothing.
• The post-deploy smoke check fetches the homepage and is allowed to fail without failing the deploy.

For a single-instance portfolio this is a defensible baseline, but it is a baseline. The richer observability described below exists only where the failures are hard to see, which is in the AI paths.

One identifier, three places

Every tRPC request gets a random UUID when it enters the middleware. The id is held in Node's async context, so any code running within that request can read it without receiving it as a parameter. It is derived from nothing about the visitor (not the IP address, the cookie or the question), so two requests from the same person share nothing.

For the two public AI surfaces, the Engineering Ask and the Photography Assistant, that same value is used in three places:

The root span of each AI trace takes the request id as its own id instead of generating a fresh one. That small decision is what makes the join work: the trace id on a log line, the trace id column in the database and the root run of the exported trace are one value, not three values connected by a mapping table that could drift.

An admin-only view lists recent public questions and can build a query artifact for any of them on demand: the first-party record joined with its exported trace, as one versioned JSON document. Nothing new is written while a request runs to make that possible. Generation details (model, tokens, fallback route, cost) are read back from the trace rather than stored twice.

This is the strongest part of the architecture, and it has clear limits:

• It covers two procedures. The Curator's smaller questions, voice narration, cover-letter and CV generation and photo enrichment all call models too. They produce runtime lines but no trace and no query record.
• The two retention periods differ. The query record is kept for 90 days. Exported traces expire after 14. For an older question the artifact returns the application's half and reports, as a typed status, that the trace no longer exists.
• It ends at the server. The id is not returned to the browser, so a visitor's error report can't be matched to a trace.

What an AI request records

Within a traced request, every stage that already had a timing boundary becomes a span: a named, timed, nested unit of work. Events mark points in time inside a span. The interface has three operations (start span, end span, record event), and attribute values may only be strings, numbers or booleans.

A real Photography Assistant trace, captured while the feature was built, has about twenty runs. It shows concept resolution and query expansion calling the embedding providers, lexical and semantic retrieval side by side, the rerank call, evidence-pack construction, and two generation attempts. Between them, the events show that the claim gate rejected one claim on the first attempt and that every gate passed on the second. Before tracing, the same diagnosis meant matching warning lines in a log by timestamp.

Token counts, prompt-cache counts and cost come from one place: the structured-generation layer that every model call passes through. So no feature can forget to report them. When a provider is unavailable and the fallback chain moves to the next tier, the route is recorded as an attribute: which tier was tried, why it failed over, which provider finally answered.

What a trace deliberately does not show is any text. That is the subject of the next two sections.

Retrieval: counts and reasons, not candidates

The retrieval question that matters most in practice is why didn't this return anything? Photography Search and the Assistant interpret a question into lexical filters, semantic queries, location and time scopes, and required tags, then reduce a merged candidate pool through several stages before the model ever sees it.

In the Assistant, each of those stages reports itself as a candidate funnel entry: its kind (source, merge, filter, truncate), how many candidates went in and came out, and a reason drawn from a closed list that mirrors the code's real branches. Examples are "every requested tag facet must match", "location diversity cap" and "evidence-pack size". Every count is read from an array that already existed. No stage computes anything new to report it. Before it shipped, eight real queries were run against a frozen snapshot with and without the funnel, and the result ids, order and scores were byte-identical.

Plain Photography Search runs the same stages, but it is not a traced surface, so its funnel is computed and never exported.

The funnel has already explained real cases. Three production questions about boats, mountains and rivers returned nothing. Lexical retrieval had found about 47 candidates and semantic retrieval about 23. The trace showed the merged pool of 64–68 candidates dropping to zero at one stage: the rule that a candidate must satisfy every independently requested hard tag. No published photograph carried a river, a boat and a mountain together. The result was correct, the rule was working, and the trace said so in one line. Without it, the honest answer would have been a reproduction session against a database snapshot.

Two decisions shape what retrieval observability is not:

• Aggregate, not per candidate. Recording each candidate's identity and fate was considered and rejected. The attribute type can't carry lists, by design. And in the motivating case every candidate had been removed by the same stage for the same reason, so a per-candidate list would have repeated one line 64 times.
• No per-photo scores in production. The trace doesn't hold ranking scores, fusion contributions or the reranker's per-photo deltas. Answering "why did this photo rank fourth?" or "why didn't that one appear?" for a specific photograph is an offline question. It is answered by replaying the request against a frozen database snapshot, where every internal value is available.

Visitors get their own explanation, separate from all this: each search result shows one true reason it matched, derived from which fields actually matched. That is a product feature, not a diagnostic, and it can't express rankings or exclusions.

Validation is evidence only if it is recorded

The Photography Assistant checks every answer before a visitor sees it. The output must satisfy its contract. Each claim must be supported by the evidence pack. Each cited photo must be one the model was given. Each per-photo evaluation must refer to a real photo. Prose that looks like leaked instructions is redacted. A failed claim check triggers one bounded retry with the photo selection held fixed.

Those gates stop wrong output. On their own they don't make failures visible. A system that silently refuses or silently retries has prevented a bad answer and left no evidence of why. So each gate emits an event with the attempt number, whether it passed and how many items it checked or rejected. The request records whether a retry fired and why, and the final outcome (answered or refused) carries a refusal reason.

The evidence is split by sensitivity:

| What failed | Where it is visible | |---|---| | Which gate, which attempt | trace event | | How many claims, citations or evaluations failed | trace event | | Whether a retry ran, its reason and its result | trace attributes | | The rejected claims and photo ids themselves | runtime warning line only, tagged with the trace id | | A contract failure's stage (empty, unparseable, transform, contract) | trace attribute and error class name | | The validator's own message | runtime warning line only |

The same split applies to the Engineering Ask, whose answer records how many sources the model claimed, how many it was allowed to keep and how many were dropped.

The content boundary

Observability in an AI system is a data flow, and this one is designed as a trust boundary first.

The visitor's question is stored only in the application's own database, kept 90 days and pruned on write. The table has no column for an IP address, cookie, session, account or user agent, and the trace id that joins it to anything else is random per request. Rate-limited requests are not recorded at all, because counting them would come close to tracking individual visitors.

The exported trace is metadata only, and the guarantee is structural rather than a matter of discipline:

1. The observability interface has no input or output parameter, and attribute values are scalars. Domain code has no way to hand over a document.
2. The exported run record has no input or output fields at all, so there is nothing to redact.
3. Errors are exported as a sanitised error class name, never a message, because a message can contain whatever its author interpolated into it.

Traces are exported to LangSmith through a small hand-written adapter, not the vendor's SDK, so no domain code imports anything vendor-specific. The exporter is off unless deployment configuration turns it on explicitly, and it accepts only LangSmith's EU-region ingest endpoint. Any other host disables it with a startup message instead of exporting silently. It has been enabled for the deployed application, and traces of real production questions have been read back from it during development.

One boundary is weaker than the others. Runtime warning lines are free text, and a few of them include rejected claims, which are model output. They are tagged with the trace id and stay in the application's own logs. They are not covered by the same structural guarantee, and there is no written policy for what runtime logs may contain. That is a real limitation.

When observability fails

The rule is that observability failing must never become a visitor's request failing. Every call into the exporter goes through one containment wrapper that catches anything it throws and writes a capped number of warnings, so a broken exporter can't flood the logs it was meant to improve. Upload happens on a deferred timer after the request has finished, never on the request path. A failed upload is dropped, not retried. In a forced test with an exporter that threw from every method, both public AI surfaces still returned complete, validated answers. The measured overhead of tracing on the heaviest request shape was about 0.017 ms at the median.

The most instructive failure in this subsystem was one that every in-process signal missed. When the exporter was first switched on, the server logged that tracing was enabled, every log line carried its trace id, and both AI surfaces answered normally. No trace reached the exporter. The bundler compiled the observability module separately for the startup layer and for the route layer. Startup registered the real exporter in one copy, and every request read the other copy, which still held the do-nothing default. Nothing looked wrong. Startup was logged by the layer that had worked, correlation never crossed a layer, requests succeeded because the exporter was fail-open by design, and unit tests run in a single module graph where the bug can't exist. The only check that could tell "working" from "discarding everything" was asking the trace store for a run id the server had just logged, and getting a 404. The fix moved the shared state onto a process-global registry, and the verification step now asks the destination, not the source.

The admin view relies on the same principle. When the trace store rate-limited reads during verification, three of six query artifacts still built correctly. Each carried the full first-party half and reported "request failed" instead of an empty trace.

Evaluation: observing quality before it ships

Runtime traces say what happened on one request. They can't say whether the system is good: whether the photographs were the ones a person would have wanted. That needs labels, and labels belong offline.

The Photography Search benchmark is a frozen, human-judged set: the current version has 106 queries, 381 photographs and 4,266 graded judgments, with its file hash and freeze date recorded. Evaluation runs against a snapshot of the production database, computes NDCG, precision, recall and MRR, and can compare against a stored baseline and fail on regression. The Assistant has its own suites for evidence recall, validation reliability and conversation behaviour. Engineering Ask retrieval and the repository's context routing were each measured on labelled task sets.

None of this runs continuously or in CI. It is run by hand, per change, and the reports record the results. The value lies in what the measurements caught:

• A tie artifact. Half of the eligible candidates sat in score ties. Ties were broken by photo id, and every judged photo happened to have a lower id than every unjudged one. So the benchmark was quietly rewarding whichever ordering kept low ids on top. Production beat all 200 random tie orders. A ranking variant that had looked like +4.16 points of NDCG turned out to be −1.70 points once ties were broken neutrally, and better than production in one seed of 200. Every absolute number before that point had to be re-baselined.
• An average that hid harm. A retry policy was rejected because mean precision showed no consistent direction. Counting relevant photographs instead showed the retry destroying ten and gaining one. The policy that was rejected on the average was later shipped on the count.
• A gate that admitted harm. The gate deciding when visual fusion runs admitted 42 of the 49 queries it hurt. Through the gate, the net effect was −1.9 points per query against never fusing.
• A model judge that wasn't safe. An attempt to let models pre-screen benchmark pairs as obviously irrelevant was rejected. No arm reached 99% safe-negative precision at useful coverage, and the vision judge called a helmeted cyclist "obviously irrelevant" to "bicycles and cyclists" with 0.90 confidence. The same exercise surfaced some old human labels that looked wrong, which is its own kind of observation.

Evaluation also has a cost model. A naive tie-neutral protocol would have needed nearly two thousand provider calls for one run and still failed its convergence rule. The protocol in use runs 200 seeds with zero provider calls, and confirms with the real reranker only on request.

The browser is dark to the server

Voice input runs Whisper in a Web Worker on the visitor's device, on WebGPU where available and WebAssembly otherwise. The Voice Interaction Architecture explains why. For observability, the consequence is simple: no server trace can see any of it.

The worker reports a fair amount to the page. It sends model download progress in bytes and the backend it actually selected. If WebGPU fails, it sends a fallback event with a reason from a closed list, and it marks the GPU unhealthy for the rest of the worker's life. It sends a typed error for a model-load failure or a transcription failure. The voice session state keeps the backend and the fallback trail as diagnostics.

None of that leaves the browser. It is not logged, not displayed and not sent to the server, and the code intentionally never tells the visitor which backend they are on. What the visitor gets is recovery: a permission prompt that comes before any download, a fallback that happens without asking them, a short failure label, and a text box that always works. There is no handler for a worker that crashes outright, and inference time is not measured in production.

Where browser behaviour mattered, it was measured in local experiments: harnesses with performance marks in a controlled browser for transcription timing, and probes comparing WebGPU and WebAssembly. Those results are dated measurements, not a live signal.

That is a real limitation, but collecting browser diagnostics would mean sending data about visitors' devices to my server, and the voice design's whole premise is that the microphone path stays local. If I add it, it should be aggregated, opt-in and limited to the failure class (backend chosen, fallback reason, failure kind), never audio or text.

Frontend performance: measured, not monitored

The frontend has no real-user monitoring: no Web Vitals reporting, no performance observers in production, no bundle analyser. Its performance evidence is engineering measurement:

• React Doctor runs on the changed React surface after a change and compares the result with a base snapshot. It is an agent and developer workflow, deliberately not a hook or CI gate.
• Rendered-state verification drives a real browser through a route and checks readiness, overflow, occlusion, accessibility and broken images, writing the evidence to disk. It is run locally.
• Source-level tests pin specific performance decisions (a deferred blur, an incremental gallery mount) so a later change can't silently undo them. These do run in CI.
• One-off browser probes measured layout shift, animation frames and render counts for specific experiments. Their results are dated experiments, not continuous signals.

Engineering decision observability

The last plane observes how the software changed, not how it runs. That matters here because a lot of this code was written with AI coding agents, and "why is it like this?" has to stay answerable.

What can be reconstructed from the repository:

• Why a change shipped or didn't. Changes that involve a real decision have a report with an explicit verdict, the evidence behind it and the commands that produced it. There are about 180 engineering reports, almost all with an explicit verdict, and two dozen decision records, many of them named for their outcome ("…-rejected", "…-adopted").
• Which rules applied. Architecture boundaries, the context routing manifest, the knowledge layer and the agent instructions are all versioned with the code, so the rules in force at any commit can be read from history.
• Which checks gate a merge. Static analysis, unit tests and four repository validators run in CI on every pull request and again before deploy. Their output is CI output.

What cannot:

• What context an agent actually received. The routing manifest says what an agent should load for a kind of task. Nothing records what it did load. Session transcripts live outside the repository, and the local checkpoint records which files changed, not what the agent read.
• Local check results. Git hooks run more checks than CI, and their output isn't kept. A bypassed hook leaves no trace.
• Which lines an agent wrote. Commits are mine and seldom carry agent attribution.

This is engineering decision observability, not production telemetry: it explains why the system is the way it is, and depends on the discipline of writing the report. It is the plane I rely on most when coming back to a subsystem after a month.

Maturity, plainly

| Plane | Detection | Diagnostics | Correlation | Persistence | Reproducibility | |---|---|---|---|---|---| | Application runtime | Partial: timing lines, liveness only | Weak: production errors not logged | Partial: trace id on lines | Weak: log buffer | Weak | | Retrieval | Strong in the Assistant, absent for plain search | Strong at aggregate level | Strong | Partial: 14-day traces | Strong offline (frozen snapshots) | | LLM generation | Strong on public AI paths, weak elsewhere | Strong: model, tokens, route, failure stage | Strong on public AI paths | Partial | Partial | | Validation | Strong: per gate, per attempt | Partial: counts traced, items only in logs | Strong | Partial | Strong offline | | Evaluation | n/a | Strong | Run-level: snapshot, benchmark hash | Strong: reports in git | Strong | | Local inference | Visitor-facing only | Weak: kept in browser memory | Absent | Absent | Local experiments only | | Frontend performance | Absent in production | Strong locally | n/a | Reports | Partial | | Agent workflow | CI validators | Strong via reports | Weak | Strong for decisions, absent for context | Partial |

What is not observed, on purpose and otherwise

Deliberately not observed: the content of questions, prompts, evidence and answers in any third-party system; per-candidate retrieval detail in production; anything that identifies a visitor across requests; rate-limited attempts; which speech backend a visitor's device used.

Gaps worth closing:

1. Error capture for the generic runtime. Production tRPC errors should at least be logged with their trace id and error class. It is a small change and the largest gap.
2. A written policy for runtime log content, matching the structural one the trace already has.
3. Trace coverage for the remaining model calls: the Curator's smaller questions and narration first, because they are public.
4. Aligning retention, or snapshotting the artifact for questions worth keeping, so a 90-day record doesn't point to a trace that expired at 14 days.
5. Recording locale for Engineering Ask questions, which the query record currently leaves empty.

Not justified at this scale: metrics backends, dashboards, alerting, SLOs, sampling and distributed tracing. There is one server process and one database, and every interesting failure so far was found by reading one request's evidence, not by a threshold on an aggregate.

Principles

• Separate the question from the evidence. Visitor text stays first-party; exported traces are metadata only, enforced by types, not by care.
• One identifier, reused. A second id for the same thing is a new way for correlation to be wrong.
• Record the gate, not just the outcome. A validator that refuses without a record prevents a bad answer and hides the reason.
• Aggregate in production, detail offline. Runtime records say where candidates disappeared, and frozen snapshots say which ones.
• Observability must fail open. It sits next to the request path, never on it.
• Verify at the destination. Every in-process signal can be green while nothing is recorded.
• Treat measurement as a claim. Benchmarks have artifacts, averages hide harm, and model judges need their own evaluation.

Conclusion

Observability in this application isn't one system. Runtime lines show that something happened and how long it took. A metadata-only trace explains an AI request stage by stage, joined to a first-party record of the question by an id that identifies nothing about the visitor. Evaluation on frozen, human-judged data says whether a change is better before it ships. Engineering reports say why the system looks the way it does. The browser stays dark, partly by design, and the generic runtime is thinner than it should be.

The decisions I'd keep are the boundaries: content stays first-party, the trace stays structural, observability stays off the request path, and verification asks the destination. The weaknesses are where the article names them: errors outside the AI paths, the speech model on the visitor's device, and the context an agent worked from.

Further reading

The subsystems observed here have their own case studies: Photography Assistant Architecture for the evidence pack and its validators, Photography Search Architecture for retrieval and ranking, Voice Interaction Architecture for local speech recognition, and AI-Native Repository Architecture for the guardrails and context routing the engineering plane relies on.