Photography Learning Architecture

The Curator can show a visitor photographs. I wanted it to be able to teach with them as well: open a photograph, ask how it works, and get a short lesson about what is actually in that image, then follow up, or ask for another example from the same archive. The problem is the familiar one of keeping an explanation tied to evidence. The surprise was where the errors came from: not the model's world knowledge, but my own stored descriptions.

The problem

A general-purpose model can explain leading lines or backlighting perfectly well. That isn't the hard part. The hard part is explaining them in this photograph, and finding another photograph in this collection that shows the same thing.

• The lesson has to be true of the image. "Notice how the shallow depth of field isolates the heron" is useless if the reeds behind the heron are sharp.
• The stored text is not ground truth. Every photograph carries a caption and tags that a vision model wrote when it was uploaded. An audit found that some of them were wrong about exactly the things a lesson relies on: which areas are sharp, where the light comes from, what is in front of what. A lesson built only on that text repeats those mistakes fluently.
• A topic has to exist before it is taught. A "Follow the leading lines" button on a photograph without leading lines is itself a false claim.
• Examples have to come from the archive. "Show me a better example" has to return real, published photographs, and the answer about them must not claim they show the technique unless their evidence does.
• Follow-ups are short. "Why is that one better?" means nothing on its own. The system has to know which lesson it continues, without carrying an open-ended conversation history into every prompt.

What I built

Learning is not a separate product, agent or model. It is a small set of additions to the Curator, the grounded assistant described in the Photography Assistant case study, and it reuses that system's conversation, transport, rate limits, generation layer and validators.

It has three parts:

1. Look closer. In Focus view, a photograph offers a general lesson ("Read this photograph") and at most two technique lessons, such as "Understand the silhouette" or "How focus directs attention". Pressing one produces a short lesson of three to five sentences and a "Look for:" phrase pointing at one concrete detail, written by a vision model that sees the photograph.
2. Bounded lesson context. The lesson leaves one small, typed fact in the conversation: which topic was just taught. The follow-ups that continue the lesson use it, and the follow-ups that start something new drop it.
3. Two follow-up paths. A question about the open photograph is answered from its pixels. A request for another example becomes a similarity search in the space that matches the lesson, answered through the Curator's normal evidence and validation path.

There are nine technique topics, from selective focus to repetition, plus the general lesson, in a closed registry. No model ever creates a topic.

Architecture at a glance

A lesson, from the press of a button to the answer the visitor reads:

And what happens to the next typed message:

The key architectural decision

The model explains. The application decides what is being explained, and on what evidence.

Each lesson has exactly one model call, and the model chooses the words. It does not choose:

• which topics a photograph offers (decided by the server before any button is drawn, and decided again on every request);
• which photograph it is looking at (the image is chosen through a signed record and checked against its digest);
• which settings were recorded (it may only use the ones it is given, and an unknown one stays unknown);
• what a follow-up is about (a deterministic resolver decides, with a narrowly bounded planner);
• which space an example search runs in (a fixed table maps each topic to a lens);
• which photographs are examples (retrieval finds them; the answer model can only cite what it was given).

A lesson sees one photograph and can reference only that one. The request carries a numeric id and a topic from a closed list, so it contains no free text at all.

Deciding what can be taught

The button is a claim that the photograph shows the topic, so a single function decides what is offered, in a fixed order. The button list and the server-side check on every lesson both use it:

1. An evidence hold wins. A handful of photographs have stored descriptions that I verified to be wrong. They offer no lessons at all until the data is fixed, because a lesson prompt carries the description as a hint and a follow-up answered from text would repeat the error.
2. An image-grounded record decides next. An offline job, started by me in the admin and never by a visitor, sends each photograph to a vision model once. The model assesses the topics against the image. A deterministic selector then keeps only topics the model judged clearly supported and worth teaching, at most two per photograph. A valid record with no topics also counts as a decision: a topic the image check rejected must not come back through a weaker rule. A record that leaves out one of the concepts it must always assess is marked incomplete and does not decide anything.
3. Otherwise, stored-text rules decide. They cover four topics: selective focus, silhouette, layered depth and monochrome tone. A rule requires agreement between stored signals, such as a subject-isolation assertion together with a tag that names the blur. The other topics have no rule because text alone measured too unreliable: for leading lines, backlighting, atmospheric perspective and repetition, only 4 to 7 of 16 probed photographs whose text suggested the topic clearly showed it.

The general lesson asserts no technique, so it needs no topic evidence. It is still refused for a held photograph.

When nothing qualifies, the visitor sees no technique buttons. If a request arrives anyway, for example from a stale page or a forged call, it gets a fixed, application-written answer, "There isn't enough reliable information about this photograph to explain that", and no model is called.

A lesson grounded in the image

The lesson prompt is built around an explicit evidence hierarchy:

• What is visible decides what the photograph shows: subjects, positions, light, sharpness, colour.
• Recorded capture settings are the only source for camera settings. The model may not infer a focal length, "compression" or a time of day from how the image looks.
• The earlier description is a hint, never a fact. The prompt names the things it most often gets wrong (sharpness, position, the direction of light) and tells the model to check each visual statement against the image, and to follow the image where they disagree.

The prompt also excludes the failures I saw in evaluation: no guessing at the photographer's intentions, no advice to crop, no generic praise, and no mention of captions, metadata or "evidence". In German it uses informal du and established terms such as Schärfentiefe and Gegenlicht.

The model returns a typed object, the lesson and the "look for" phrase, with hard length limits. The application rejects it if it fails the schema or uses forbidden vocabulary, and redacts internal identifiers. Unlike the Curator's answers, a lesson is never retried: any failure, including an unreadable image or a provider timeout, becomes a fixed "please try again" limitation. There is never a text-only lesson in place of a vision lesson. A silent fallback would quietly bring back the text-only error mode the vision lesson exists to avoid. Lessons are cached under a key that includes the evidence fingerprint, image digest, model and prompt version, so a cached lesson never outlives its inputs.

Lessons run on Claude Haiku 5.5 through a direct provider call. The Curator's usual text chain starts with a model that cannot accept images, and sending a vision request into it would either fail or silently drop the image.

Pointing at the photograph. A lesson can also mark what it describes on the photograph itself: a region, a focus point or numbered depth planes, tied to the sentence that describes it and following the narration when the lesson is read aloud. A model proposes the positions, and a deterministic check decides whether they are shown. A second call places the same sentences on a mirrored copy of the image, and a marker appears only if both placements agree. Line traces, edge traces and light-direction guides are deliberately not drawn: no verified geometry exists for them, and drawing one from a rectangle would mean inventing evidence.

Lesson context: one typed fact

After a lesson, the conversation has to remember one thing: which topic was taught on which photograph. I added exactly that as a field in the small context the server writes and the browser echoes back on the next turn. The field is learningTopic.

• It is a closed enum. The browser sets it when a lesson is shown, because which lesson appeared on screen is a UI fact. The server re-validates it on every turn, and an unknown value fails the parse. Then the whole context is dropped and the question is answered as a fresh single turn.
• It is advisory. It can select a similarity lens and add one line to the answer prompt. It cannot reach a claim, a citation, or the set of photographs in evidence.
• It has a defined lifetime. A question about the photographs on screen keeps it. A similarity search keeps it, because that is the example search the lesson motivated. A new search or a refinement clears it.

It is not memory in any longer sense. The server keeps no session, and nothing survives the conversation.

I rejected preserving the topic across a new search. It would rescue the rare turn where the planner's output failed validation and the deterministic fallback ran a new search, at the price of carrying a composition lesson into "find photographs from Switzerland".

Follow-ups: pixels or examples

Follow-ups use the Curator's normal router: a deterministic resolver, and a planner model consulted at most once when the resolver defers, whose output can only select one of a fixed set of plans. The lesson adds two small rules.

"Show me a better example." A request verb while a lesson is carried is not enough on its own, so the resolver marks the turn for the planner rather than running a search on five words. The planner sees the topic and has one learning-specific instruction: a request for an example is a similarity search on the lesson's photograph, and a question about the examples already found is a relationship question, not another search.

The planner decides that the visitor wants an example. The application decides where to look, from a fixed table. Composition topics (leading lines, layered depth, silhouette, selective focus, repetition) search a text space that describes structure. Light topics search one that describes light and atmosphere. Tone and colour topics search the pixel embedding space. The general lesson uses the general space. Retrieval returns at most eight photographs, using vectors already stored for each one and making no provider call.

The answer is written by the Curator's normal generation path, with one extra prompt line: the visitor was just taught this topic on this photograph, and "never state that a photo shows the technique unless its evidence above shows it."

"Why does that make the person look anonymous?" When routing has already settled a turn on exactly the photograph that is open, and it isn't a question about recorded settings ("which lens?"), the question is answered from the photograph's pixels by the same vision model, with the same image binding and the same output gates as the lesson. This decision is made after routing and changes nothing in it. Settings questions stay on the text path, where the recorded values live, and a held photograph keeps its text answer.

The result is a short exchange that holds together:

Visitor presses Understand the silhouette. The vision model writes the lesson. Visitor: "Why does that make the person look anonymous?" — answered from the pixels of the same photograph. Visitor: "Show me a better example." — a similarity search in the composition lens, up to eight photographs from the archive, answered through the full validation chain. Visitor: "Why is that one better?" — a relationship question over the examples now on screen, with the lesson named in the prompt.

What the validators guarantee and what they don't

Example answers use the Curator's existing validators, unchanged by learning. Lessons have their own, smaller set:

• Example answers pass the full gate chain: schema, relationship claims, citations (a cited photograph must be in the evidence), evaluations and result scope, with one bounded, informed retry.
• Lessons and pixel answers pass a strict schema, a vocabulary check and leak redaction, with no retry.
• The visitor's own free text reaches a vision prompt only on the pixel path, fenced as untrusted input, and only after the same escalation checks every question passes.

The deterministic guarantees are about identity and structure. Every photograph shown is real and published. A lesson describes the photograph it was asked about, using the exact verified image. A topic button appears only after the offer rules pass. A forged topic cannot enter a prompt. Relationship claims about examples correspond to computed facts.

What remains probabilistic is the prose itself. No validator checks that a sentence about light or sharpness is true of the image. The "never say a photo shows the technique unless its evidence does" line is a prompt instruction, not a check. The image makes lessons much more accurate, not infallible.

What failed, and what the evaluations showed

Every number in this section comes from an offline experiment on frozen inputs. None is a production metric, and no measurement of whether visitors actually learn exists.

A text lesson would have been "good enough". Problem: I didn't know whether a teaching answer needed the image at all. Experiment: 384 answers across four arms on 24 photographs, rated by two blinded model judges, and blind by one person (me) for 269 of them. Result: everyone agreed on the big step. I rated the Curator's ordinary answers useful once in 66, and all 67 text lessons I rated from the main text arm. On text versus vision, the judges and I disagreed. The judges, checking each claim against the pixels, preferred vision 38 to 9. I preferred text 14 to 8, with 46 ties, and I rated text lessons useful even where they repeated a wrong caption ("lorikeets" for pigeons). Consequence: the first shipped Learn was text-only, with the photographs whose captions were known to be wrong held back. The disagreement turned out to be the important finding. A visitor can't see a factual error any more than I did, so perceived usefulness can't detect it.

Vision fixed what correction couldn't. Correcting captions only fixes the errors you have already found, so the next step moved the lesson to the image. On the same model, the vision lesson carried over no errors from wrong captions where the text lesson carried seven. Answers both judges fully accepted went from 0 to 11 of 17 in English and from 0 to 13 of 18 in German. The comparison is not purely about vision, because the vision arm also had a revised prompt, but the inherited errors are the image's doing. Vision still errs on tone in dark photographs, calling a night sky "pale" and a grey bird "near-black". A 1280-pixel image cost 31% less but wasn't shown to be as accurate, so the 1920-pixel image stayed. Extending vision to follow-ups about the open photograph raised judged accuracy from 3.53 to 4.41 out of 5 on 33 answers. Answers both judges called materially wrong went from 4 to 1.

Lesson continuity. Before the topic was in the context, "show me a better example" after a lesson ran a fresh search on its own five words; in the traced case it found nothing. On a fresh holdout (22 conversations, 3 repeats each), learning follow-ups went from 0 of 12 trials correct to 10 of 12. Across all follow-up families in the same holdout, correct trials went from 102 to 136 of 153, with no invented photograph ids and no example misroutes. A later, larger holdout scored learning 12 of 12 and the "don't carry the lesson" controls 6 of 6.

Asking the image to verify the topic. A dedicated image check for "selective focus" removed every unsupported offer that the text rules made (precision 0.69 to 1.00). It failed a recall gate that I had set before the run: 0.81 against a threshold of 0.85, on 36 positive cases. I recorded that as a rejection rather than adjusting the threshold. A calibrated version later reached 0.97 precision and 0.83 recall on the same held-out set and was made permanent. One known false positive remains, and its labels are agent judgments, not mine.

A classifier I didn't add. To decide which marker style fits a lesson sentence, I compared a dedicated judge model against a deterministic classifier on 188 held-out sentences. The model scored slightly higher (83.0% vs 80.3%), but not significantly (p = 0.53), and it produced more technical false positives. The deterministic classifier stayed.

Trade-offs and limitations

• Learning is started by a button, not recognised from language. Typing "teach me leading lines" goes through the ordinary Curator path as a search, and a general photography question has no retrieval-free answer path. Starting lessons from typed text needs a different design, not another regex.
• "Better" is not ranked. An example search returns the photographs nearest in the lesson's space. Nothing ranks them by how well they demonstrate the technique, and nothing diversifies them. "Another example" does not exclude examples already shown.
• The lens applies only on the planner path. A request the resolver handles itself, such as "more like this", uses the general space even after a lesson.
• Example answers are text-grounded. The answer about examples knows their stored evidence, not their pixels. Only the open photograph is ever seen.
• One vision call per lesson. That is accuracy bought with cost and latency. Measured on the evaluation runs, a vision lesson cost roughly two and a half times a text lesson on the same model and took a median of 3–4 seconds. A failed lesson is never quietly downgraded to text.
• No human validation since the first review. The vision lessons, the topic calibration and the markers were judged by models and by an agent against labels. Blind review pages exist, but no person has rated these answers yet.
• Coverage depends on the archive. A photograph whose image checks fail, or whose description is held, simply teaches less.

What I learned

The most useful thing this work taught me was how the evaluation could be wrong. The text lessons were fluent enough that I, the photographer, rated wrong ones useful. That one disagreement is why the design checks evidence before it relies on taste: the image decides what the photograph shows, recorded settings decide how it was taken, and my own descriptions are treated as hints. The rest follows from the same principle. A button is a claim and has to be earned. The lesson context is one typed fact with a defined lifetime. The model writes the explanation, and the application decides what is being explained.

Explore the Curator

Open any photograph on the Photography page and choose Look closer. Then ask about what you see, or ask for another example. The Photography Assistant and Photography Search case studies describe the grounded assistant and the retrieval that learning builds on.