The gap between clicking and understanding

Put a typical AI agent on your app and it opens a page, clicks a few buttons, notices a misaligned element or a console error, and calls the run done. That's useful the way a smoke test is useful. The agent has no model of what your product actually does.

A QA engineer worth hiring doesn't click around. They learn the product first, then reason about it: this is a hotel system, so I need to test a single booking, a group booking, a cancellation that frees the room back up, check-in, check-out, and the night audit, and I need to confirm each one actually persisted on the server instead of trusting that the UI looked happy.

What happened when we gave it a real app and no instructions

We pointed Sentinel at a working full-stack hotel PMS (a property management system): a Next.js frontend, a separate API service, a Postgres database. The PMS is KaribuKit, our own product, which is why we could hand an agent admin credentials for a disposable test tenant. That's also all it got — the repo and those credentials. No test plan, no list of flows.

It read the code, concluded the product was a boutique and safari hotel PMS, and derived nine critical business flows on its own: the full reservation lifecycle, group bookings, the lead-to-proposal pipeline, rate management, guest self-service, mid-stay room changes, the night audit, payment through invoice to refund, and the AI copilot. Cancellations are in there too, though this run folded them in as edge cases and backend checks inside the other flows rather than deriving a standalone flow. That's close to the list a human QA lead would write on day one, and nobody handed it to the agent.

The run itself: plan derived from the repo, top two flows deep-tested twice each, then the vision pass over every screen it visited.

Then it ran the top two of those flows, twice each, and the trace read like watching a person work:

• It called GET /api/availability, got a 400, worked out the params it was missing, and retried with adults=2&children=0 to get a 200.
• It created a real reservation with POST /api/reservations (201), then fetched it back to confirm it had persisted with the right room and rate.
• It walked the status lifecycle, found the check-in endpoint by trial (/checkin gave 404, /check-in gave 400, then a valid call returned 200), and checked the folio.

The folio it was verifying: three room-charge nights at $178, balance due $534. The red toast in the corner is a bug being caught live: "No rooms available" on a reservation that already held its room.

The bugs it surfaced are ones you can't find from the UI alone:

• Confirming a reservation came back NO_AVAILABILITY, even though that same reservation already held the room. A backend state-machine bug the UI turned into a misleading "no rooms available" toast on one attempt, and into no feedback at all on another.
• The calendar showed a room as available after a booking already existed for it. The API and the UI disagreed, and only checking both layers caught it.
• Check-in returned 200, but the guest's registrationStatus stayed NONE on the server: a state transition that only half-completed.

The calendar mid-run, filled with reservations the agent created itself, tentative and confirmed side by side.

How it works

The QA agent's deepest engine, flow, is a pipeline:

1. Read the repo. A deterministic recon pass (grep and find, no model calls) extracts the structure: frontend routes, API route modules, services, database entities. The model reasons over that digest instead of crawling a monorepo blind, which is slow and misses things.
2. Derive the flows. Mimo, the Xiaomi model we run for decisions through the pi agent harness, turns the digest into a prioritized list of end-to-end business flows, each with UI steps, backend assertions, and edge cases. The plan is cached per commit, so it only re-derives when the code changes.
3. Run each flow as an agent loop: the model decides the next action, Playwright drives the browser, and a first-class api_request tool checks server state at each step by running fetch inside the page, replaying the Authorization header the frontend itself sent. The model only ever gets browser and API tools, never your shell or filesystem, and a hard budget bounds each attempt. Past 90 tool calls the action tools refuse to act, and the only move left is finish.
4. Run it more than once. Autonomous agents are non-deterministic. We measured it on one flow, where one attempt found zero bugs and another found five. So each flow runs twice by default (FLOW_ATTEMPTS, a knob) and the findings are unioned: a bug found by any attempt makes the report, and each flow keeps its worst verdict across attempts.
5. Grade the design. Every distinct screen the agent visits (deduped by URL, up to eight per run by default) also gets a vision pass from a multimodal model, scoring visual hierarchy, spacing, text likely to fail WCAG contrast, typography, and broken states. That's the layer a DOM-only check can't see.

Recon and the report are plain code; the model sits only where judgment is needed: deriving the flows, deciding each next action, grading the screens.

None of it knows anything about hotels. The recon pass digests any repo on the common JS stacks (Next.js routes, Express and Fastify route modules, Prisma, Drizzle, or plain SQL schemas) and everything downstream reasons over the digest, so the hotel PMS was just the demo we had handy; other stacks are a recon patch away.

All of it runs on a schedule: a launchd tick every fifteen minutes checks each repo and runs whatever is due, review on every new commit and QA on whatever cadence you set (every 12 or 24 hours, in ours). The QA agent has three siblings: a code-review agent that reads each new diff, a docs-sync agent that keeps your Markdown matching the code, and a brain-sync agent that distills what changed in each repo into a shared team knowledge repo. The two that write work in isolated worktrees and open PRs; neither ever merges its own changes.

How we got here

We started on Mimo for the least strategic reason possible: the pi agent harness on the machine was already pointed at it. The open question was whether a cheap model could drive a browser well enough to matter, and for version one the answer was no.

Version one was a deterministic Node loop. The script owned control flow and called Mimo as a toolless one-shot brain, once per step: here is the DOM, name the single next action, and the loop ran it through Playwright. It worked on something simple. Against a small product-search app (upload a photo, get visually similar items) it found real bugs for pennies. A three-cent run caught prices truncated mid-value, like "₹4,19", and product cards cut off; the cheapest run came in at $0.0044.

But the loop was the ceiling. The model saw one step at a time with no memory of why it took the last one, so it never accumulated enough context to test a flow with more than a couple of moves. It couldn't get past clicking around a single page.

So we gave the loop to the model. Version two, which we called pi-native, registers Playwright-backed browser tools as a pi extension and lets Mimo drive them inside pi's own agent loop, with full session memory. It went deeper on a single goal. But it still needed a goal, and writing goals by hand is the exact chore we were trying to delete.

Version three is the flow engine described above: read the repo, derive the flows, run each one against the browser and the backend.

Why Mimo, and what the first runs broke

The model choice got more interesting when we added the design review. We sent screenshots to Mimo for a UI/UX pass and got nonsense back, because the model we were running, mimo-v2.5-pro, is text-only. The fix was to read what the Xiaomi API actually serves. Alongside the text models it has mimo-v2-omni, which is multimodal, so vision calls omni directly over the plain OpenAI-compatible API, one screenshot per call; a rubric-scored one-shot doesn't need an agent harness.

And mimo-v2-omni is a reasoning model, so it kept cutting off before it emitted its JSON verdict until we raised its budget to 6,000 max tokens. A small thing that ate an afternoon.

We do run other models, just not in the hot loop. claude makes the surgical Markdown edits for the docs-sync and brain-sync agents inside guarded worktrees, where precision matters more than price and it runs rarely. codex, on gpt-5.5, sits in as an optional read-only review engine for when you want a slower, deeper second opinion. What kept Mimo as the default for the always-on work is volume: the premise is a fleet re-reviewing every new commit and re-running QA on a cadence across every repo you give it, and at that volume cost is a design constraint. A Mimo decision is a fraction of a cent, a shallow QA run is a few cents, and the deepest multi-attempt run on the hotel PMS was 364 steps for $1.95.

We haven't benchmarked the field, though. Mimo is the default because it was already there and turned out to be good enough, not because we proved it optimal. The whole thing is provider-agnostic (the Try it section has the knobs), and the number we want field reports on is bugs found per dollar. If a different model finds more for less on the same repo, open an issue with the numbers.

The first full run against a real full-stack app is where the integration bugs lived, and none of them were the model's fault:

• Our boot wrapper detached the dev server's session and next dev quietly died. We went back to a plain background boot, with a teardown that reaps the whole process tree.
• Playwright filled the login form before React had hydrated, and the empty submit came back a 400. A hydration-safe retype that verifies the field actually holds its value fixed it.
• CORS blocked every call because the frontend spoke to localhost while the agent used 127.0.0.1. An hour lost to a one-line config.
• The backend assertions returned 401 until we stopped reconstructing the auth token and simply reused the Authorization header the frontend was already sending.

Unglamorous, all of it. That was most of the actual work.

The hard one: QA an app you can't even log into

Plenty of apps don't have a login form. They have a Connect Wallet button, and everything past it is gated behind MetaMask or Rabby. A headless agent can't click a browser-extension popup, so for a while those apps were out of reach.

The way in was cleaner than we expected, and it never touched the app's code. A web3 app talks to its wallet through a standard interface: window.ethereum, or for newer apps an EIP-6963 announcement the wallet broadcasts to the page. So before the page loads, Sentinel injects its own implementation of that interface, backed by a throwaway private key it keeps in Node. As far as the app can tell it's an ordinary MetaMask, but it's a wallet the agent can drive. The app connects, reads the chain, and signs exactly as it would for a real person, and the key never crosses into the page.

The app sees a standard provider; the key and the method filter stay in Node, and send methods stop there.

The catch is that these are real apps on a real chain with real money. Point a funded wallet at a live exchange and "QA" can quietly become "opened a leveraged position." So the wallet is a freshly generated, unfunded burner, and we made spending impossible even under misconfiguration: no method that submits a transaction is ever forwarded to the network.

We had an adversarial pass go hunting for holes in that promise, and it paid for itself. The first cut checked the wallet's balance and bailed if it held funds, but the check passed silently whenever the balance lookup failed, which is precisely the moment you'd want it to stop. We fixed it to fail closed, then stopped trusting the balance at all. Broadcasting is blocked by method name, and anything not on an explicit read-only allow-list is refused outright, so a send variant we never anticipated can't slip through either. Reads go through and sends never do; a key that has ever been used aborts the run.

Then we pointed it at a perpetuals exchange. Its frontend, to be precise: the app booted locally from a feature branch in a throwaway git worktree, the real backend never ran, and the on-chain reads went to the live chain. The app gates access by wallet whitelist, so Sentinel stubs the gate endpoints at the network layer, and the whitelist stub is the fun one: it encrypts the burner's address with the app's own key, and the app decrypts it and finds the burner already whitelisted.

The boring problems arrived on schedule:

• The landing page fetched a backend during server rendering, so with no backend it returned a 500 and the health check never went green. We started the agent on the trading route instead.
• The public RPC handed back a malformed CORS header ('*,*', a doubled wildcard browsers reject), and the app's own on-chain reads all failed. A seven-step run racked up 71,597 console errors and 35,797 failed requests, cost $1.16, and found zero functional bugs. We routed the app's RPC calls through Node, where CORS doesn't apply and we could retry the flaky ones.
• A wallet SDK with a placeholder project id threw an error that tripped the framework's full-screen dev overlay, and the overlay silently swallowed every click, so the agent declared the whole page broken. We tore the overlay down and capped the error stream so a noisy app can't bury a run again.

What came out was a real report. On an unfunded burner the agent connected, opened the isolated-margin trade screen, and worked the form like a tester:

• The Open Position button hung outright on click (an eight-second timeout, no confirmation modal ever appearing).
• No order preview after entering collateral, no slippage control anywhere in the UI.
• No balance check: it typed 999,999 and the form shrugged.
• A wallet connection that silently dropped when you switched margin modes.
• A leverage slider showing its internal id (slider-ex-2) as the label.
• A negative amount the validation only half-caught.

Nine functional bugs and thirteen design findings in a 61-step session, for $0.28, with not one transaction ever reaching the chain.

Every run leaves its evidence on disk: a screenshot per step, the structured report, and the vision findings.

The stack

Sentinel is deliberately boring to operate — about 2,500 lines of bash, Node, and TypeScript, a launchd scheduler, and the CLIs it shells out to. There's no service to host and no database of its own, and you can read the whole thing in an afternoon. That's on purpose. We weren't going to leave an agent running unattended without being able to read everything it can do.

The same goes for what leaves the machine. Sending a repo's code, DOM, or diff to a model is opt-in per target (ai_allowed), so a work repo can't ride along by accident.

What it can't do yet

• Exploration still varies run to run; that's why every flow runs more than once.
• Booting a complex stack takes the right config (ports, auth, a test database), and you should never point write-capable QA at production data.
• Depth trades off against time and cost, all of it knobs: flows per run, attempts per flow, steps per attempt.
• For wallet apps it drives an unfunded burner, so it tests everything up to the moment of settlement, not a trade actually filling. A local anvil fork (Foundry's local Ethereum node) gives it real chain state to quote against, but the broadcast block stays on even there. Actually filling a trade on a fork is a thing we haven't built.

Try it

Sentinel is MIT-licensed and on GitHub: github.com/Simbastack-hq/sentinel.

git clone https://github.com/Simbastack-hq/sentinel.git && cd sentinel
npm install && ( cd pi-ext/qa-browser && npm install )
cp config/sentinel.env.example config/sentinel.env
cp config/targets.json.example config/targets.json   # placeholder targets inside; add your own repo
bin/sentinel doctor && bin/sentinel run <your-app> qa

The one hard prerequisite is the pi CLI with a provider configured; Mimo, authed with a Xiaomi key, is the default. Swapping it is config: QA_PROVIDER/QA_MODEL and the VISION_* variables point the decisions and the vision pass at any OpenAI-compatible endpoint, OpenRouter and local models included. doctor will tell you what's missing.

examples/ has ready-to-copy target registries, and the placeholder targets.json includes a complete web3 wallet-dApp sample like the exchange run above.

Point it at something real and see what it finds. PRs and issues welcome.

— Hemanshu, building Sentinel and KaribuKit at SimbaStack

The first half is a constant flood of footage from the iPhone, the DJI Pocket, the drone, the Nikon Z8, and lately the Ray-Ban Metas too. There's always something being recorded. Every photographer or videographer I know is sitting on the same problem: an archive that grows faster than they can edit it. The second half is why mine never gets touched.

Airport security somewhere between Nairobi and Spain. Two trays of cameras, headphones, drone bits, batteries, SSDs, more cables than anyone needs. Most of it records something. Almost none of what they record gets touched again any time soon.

Three months ago the lodge's social channels went dark. Not for lack of content; the lodge has years of raw footage across multiple SSDs. The bottleneck was editing time, and my time disappeared. Claude Code with Opus 4.5 (and then 4.6) hit the point in February where you could leave agents running for hours and come back to merged PRs. KaribuKit was going live with its first paying property in the same window. I stopped sleeping properly, started running three or four agents in parallel in the background, and the months when I would have cut reels turned into months when I shipped software instead.

So one weekend I sat down to fix it. The first thing I tried was wrong.

The wrong layer

The initial pitch (to myself, after about an hour of research) was a SaaS stack: Eddie AI for iterative editing, Higgsfield MCP for generative B-roll, Submagic for captions, Buffer for cross-posting. About $140 a month, slick on paper.

Two problems showed up before I ran any of it.

First, generative AI video has no place on a real travel brand. Guests pay $300 a night and up to see the actual place, and mislabeled AI shots equals TripAdvisor crucifixion. Higgsfield out.

Second, 3-5 posts a week was aggressive for me, and the realistic floor was more like 2-3. The pitch was optimistic in a way that would have me failing by week two.

Then I remembered I already own DaVinci Resolve Studio, and Resolve 21 ships IntelliSearch (semantic clip search), Smart Bins (auto-organizing folders), and Voice to Subtitle that produces 90-95% accurate captions on the timeline. That's roughly 70% of what Eddie sells, so Eddie was out too.

What I was left with was Claude Code driving Resolve via the open-source DaVinci Resolve MCP, with ElevenLabs handling voiceover on informational clips where it earned its place, and the cost had dropped from $140 a month to $22.

But the deeper thing only landed once I tried to actually use any of this. Every AI video editor on the market assumes your footage is already labeled. Mine is IMG_*.mov and DJI_*.mp4 across folders with names like Mara june 2024 backup final FINAL. Eddie can search by transcript, but none of these tools can find "the elephant on the hill at golden hour" against an unlabeled archive.

The AI editor is solving the wrong problem. Or more precisely, it's solving the second problem; the first problem is the index.

The question

The question I kept coming back to was: how does the agent know what's in each clip?

There's no answer for an unlabeled archive. You can throw transcripts at it, GPS coordinates, filenames, parent folders. None of that gives you "the wide shot at sunrise with the giraffe in the frame" unless something has actually looked at the pixels.

The leverage is upstream. Build the index first, make the archive queryable in English, and the editor on top becomes a thin layer doing what it was designed to do.

So I built the index, locally.

The build

This is the kind of AI-native build I do for clients at SimbaStack, except I was both the client and the engineer this time, which made the decision tree a lot shorter.

I had four constraints going in:

• It had to be local-first. The Mara Hilltop archive lives on physical SSDs and most of the personal stuff is on my laptop, and uploading thousands of multi-gigabyte clips to the cloud made no sense on cost, never mind as a way to hand the entire visual record of my life to a third party.
• I wanted sidecars rather than a central database: a .description.md per clip, living right next to it, plain text and grep-able. It survives if my indexer breaks tomorrow, and it travels with the data when files move between drives.
• One vision call had to capture everything, because the vision pass over the extracted frames is the expensive operation. Anything I might want to know about a clip later has to come out of that one call, so the schema is exhaustive on day one: rating, technical quality, lighting, time of day, color palette, audio quality, people count, keywords, faces, location, transcript, prose description. All in one shot.
• I wanted three vision backends to choose from: Claude via my Max subscription's CLI as the default (zero marginal cost), the Anthropic API for speed when I need it, and a local backend pointed at LM Studio for the bulk pass. The local one is the one that matters.

The per-clip pipeline:

1. ffprobe for metadata.
2. exiftool for GPS lat/lon/altitude. Works on iPhone, DJI Pocket, drone footage, all the same.
3. Reverse-geocode via Nominatim. Free, rate-limited, no API key.
4. ffmpeg extracts five evenly-spaced frames at 1920px.
5. WhisperX transcribes with word-level alignment and pyannote speaker diarization. Hindi, English, Swahili, 97 languages.
6. insightface detects faces and stores 512-dim ArcFace embeddings in a centralized SQLite face DB for cross-archive person queries later.
7. Vision model reads the frames, transcript snippet, and folder context, and returns YAML frontmatter plus a prose description.
8. Sidecar written to disk.

Here's what that looks like on a real clip from the Mara Hilltop archive.

One frame from IMG_1103.MOV. Ellie on the deck of one of the luxury tents at the lodge, midday. None of that context lives in the filename.

The sidecar Gemma wrote for the same clip. YAML on top (lighting enum, time-of-day enum, color palette, face embeddings, GPS), prose ## Description below. It picked up the safari-tent setting, the camera pan from interior to savanna, the shot type, and suggested two use cases (marketing reels and travel-vlog B-roll). The filename had IMG_1103.MOV; the sidecar has the rest of what I needed to find it again.

A real Mara Hilltop archive folder after the indexer has run through it. Every clip has a .description.md sidecar next to it; the _INDEX.json and _INDEX.md at the top are folder-level rollups for fast grep and LLM-friendly handoff.

The whole thing is a Claude Code skill, about 1,400 lines of Python. Claude Code wrote almost all of it. My work was the architecture, the prompts, the schema design, and the bug triage when things went wrong.

The absurdity

This is the part that actually surprised me.

I bought a 16-inch MacBook Pro M1 Max with 64GB of RAM in 2021, and the reason had nothing to do with LLMs. I'd been hitting 32GB limits on my previous machine for a while. A messy hacker brain running hundreds of Chrome tabs alongside DaVinci Resolve, Slack, Discord, and Drive was too much for pre-unified-memory hardware to handle without paging constantly. I maxed out the RAM on the new M1 Max because the old one wouldn't stop killing my workflow and I had the money to fix it.

Five years later, that same laptop is running Gemma 4 31B Q4 in LM Studio against a year of video footage.

LM Studio with Gemma 4 31B Q4 loaded. 28.40 GB of model in memory, REST API at 127.0.0.1:1234. The bottom panel is the server log during a real bulk run, encoding frames one clip at a time.

The bulk run pushed the laptop past where 64GB of RAM alone would carry it. Activity Monitor reported 50.89 GB of swap at the peak.

64 GB of physical RAM, 50.89 GB of swap used. Memory pressure in the yellow band, the kind of state you absolutely should not run on a normal Tuesday. Apple's swap is designed for it, and the fans were loud.

I Googled whether that would damage the SSD, and apparently for a day or two it's fine. Don't make it your normal operating state, but a weekend of pushing the machine hard is well within tolerance. My laptop ran hot, the fans spun up, and it kept producing sidecars while I worked on other things.

The M1 Max 16-inch is, honestly, legendary. People in the Mac community talk about it that way for good reason: five years on, it's running 31B-parameter models at usable speed with the kind of headroom that should not exist on hardware this old. I expect another three to five years out of this thing, comfortably, because local LLMs only get more efficient and the hardware is the floor, not the ceiling.

What broke

The build was mostly Claude Code holding the pen. The interesting work was the four times it almost shipped something wrong.

WhisperX 3.8 broke its diarization API between when I last touched it and now. Two breaking changes had landed: whisperx.DiarizationPipeline moved to the whisperx.diarize submodule, and the constructor kwarg use_auth_token was renamed to token (inherited from pyannote 3.x). The fix was signature introspection: the script tries token= first and falls back to use_auth_token= if the constructor raises a TypeError, so it survives the next API shuffle automatically. When you're shelling out to AI libraries that move this fast, defensive constructor calls are cheap insurance.

The Claude CLI returns permission errors as successful responses. On the first test of the CLI backend, all four sidecars came back identical with the text "I need permission to read the image frames...", and the script's success check passed because exit code was 0 and the output wasn't empty. The cause was that in non-interactive mode without --permission-mode bypassPermissions, the CLI returns the permission-denial text as the response body instead of prompting, which means the failure mode looks exactly like success unless you string-match for it. The fix was adding the flag plus a defensive check that flags any short response containing "I need permission" as an error rather than a description. When you script AI tools, the non-interactive permission flow is where the silent failures hide.

Gemma returned people_count: "many" instead of an integer. My vision prompt literally said integer or the string "many" if >10. Gemma followed instructions correctly; the bug was schema design. The fix was a stricter prompt (integer 0-99 with explicit guidance to estimate) plus a coercion in the parser for the legacy "many" responses. Don't union-type schema fields. Pick always-int or always-string, never "int or this one specific string," because every downstream consumer pays for the choice.

Then there was the motorcycle clip that shouldn't have been culled. My initial cull prompt was photographer-portfolio-shaped: heavy motion blur, soft focus, and jittery stability got rated cull. Technically correct. Then I tested it on a handheld nighttime motorcycle clip from a Spain trip and it culled it. I caught it: that's a fun memory, the blur is the vibe. I reframed the cull criteria to "not a real recording" only (lens cap, pocket footage, two-second test clips, fully clipped exposure), not "imperfect capture." Photo archives cull aggressively; video memories cull permissively. Same schema, different criteria, and you have to be explicit about which mode you're in.

What the build taught me

Enum constraints beat instructions for confabulation prevention. I tested Gemma 4 E4B on a coworking-space photo I'd taken at night, and it described the scene as "brightly lit, abundant natural light, floor-to-ceiling windows," except the windows were pitch black outside, because it was night. Then I tested 31B with a structured schema prompt that forces the model to pick from golden_hour | bright_daylight | overcast | dim_interior | nighttime | mixed | unclear, and both thinking-off and thinking-on recovered nighttime correctly. A model can lie about open-ended prose, but it can only mis-pick from an enum, never invent a new value. Use schemas, not instructions.

Local 31B with structured prompts closes most of the gap to cloud. Gemma 4 31B Q4 thinking-off against a structured schema produces output that's hard to distinguish from Sonnet 4.6 on most of my test clips. The cloud premium earns its keep on the hard 10-20%. Bulk indexing at scale (thousands of clips overnight) should run local; cloud is the re-rate pass on clips local flagged as review. That two-tier setup is the one that scales.

AI video editors are pitched one layer too high. The valuable layer is the index. Once your archive is queryable in plain English ("show me handheld interior clips from Mara, golden hour, with people, longer than 8 seconds"), the editor on top is straightforward. Most of the AI-editor space is competing for the surface above an index that doesn't exist, and the index is the prerequisite they're all skipping past.

What's next

Looking back, time wasn't really what kept this from getting fixed sooner. I had every AI superpower currently available pointed at the work side of my life: Claude Code refactoring codebases overnight, Codex writing most of my pull requests, the agentic stack I'd just spent three months using to ship KaribuKit. On the editing side, I was using none of it. The not-getting-to-it had become its own small, low-grade frustration that lived in the back of my head all year, the kind of thing you notice every time you open a folder on the SSD and close it again without doing anything. What clicked one Saturday was that the editing backlog was a tooling problem, and tooling is the one kind of problem I happen to be well-equipped to fix right now.

This weekend I'm building the editor: Claude Code as the orchestrator, DaVinci Resolve MCP for the cuts, ElevenLabs for voiceover on informational clips. There's one hard rule baked into the tooling: the voice clone is for utility content only. Directions, room descriptions, multilingual versions, factual stuff I'd say in person anyway. Never for testimonials or founder messages. Disclosure laws are real in 2026, and trust in a hospitality brand is too easy to lose.

The index makes all of that tractable. Without it, I would still be scrubbing through 47GB of DJI Pocket footage looking for the sunrise wide.

For now: a year of Mara Hilltop footage is queryable in English on a five-year-old laptop. Cost was a weekend of my time and 50GB of swap. The remaining years across older SSDs are next.

A fair check on all of this: Mara Hilltop's social channels are still dead today. The indexer solves only half the problem (finding the right clip); the editor that turns those clips into finished reels is the other half, and that's the part I'm building this weekend. If it works, the channels light back up and I write part two. If it doesn't, I write about why.

In all honesty, the right answer here might be to hire someone. Finding an editor with the right sensibility for Mara Hilltop (warm, observational, no over-cut MTV-energy reels) is harder than writing another skill. If you know someone who works in that register, send them my way.

Edit: code at github.com/Simbastack-hq/framedex. PRs and issues welcome. Thanks to the HN commenter who flagged that the original local-path reference wasn't useful.

— NJ

Building KaribuKit (AI-native PMS for hospitality), running Mara Hilltop (eco-lodge in the Maasai Mara), and consulting through SimbaStack.

You are a JSON API. Respond ONLY with valid JSON.
Do not include explanations. Do not wrap in markdown.
Do not say "Here is the JSON".
The output is parsed by a strict parser.
If the JSON is invalid, downstream systems break.
Return only the JSON object.

I am embarrassed by every line of it. I am also, in early 2026, still using a near-identical version in three production systems, because it works and because nothing else available at the time worked better.

That prompt is an artifact of an era, and the era has a story worth telling because it's already repeating itself.

A field guide to the era

The structured-output era runs roughly from late 2022 to August 2024. Here is what shipped, with verified dates:

If you were paying attention in 2023, every row in that table felt like progress. We were inventing the abstractions at the same time we were using them in production. The Instructor library was created on June 14, 2023, the day after OpenAI launched function calling on June 13. That is the pace.

Looking back from 2026, the table reads as a single arc with a beginning, a middle, and an apparent end. The beginning is "ask the model nicely, then more nicely, then with threats." The middle is a Cambrian explosion of libraries trying to put structural rails on prompt engineering. The apparent end is OpenAI's August 2024 launch of strict-schema Structured Outputs, which claimed 100% schema conformance on gpt-4o-2024-08-06.

Phase one closed with confetti. We had won the war.

Then phase two started, and we noticed it was the same war.

The libraries that mattered

The libraries weren't the headline at the time. The model launches were. But the libraries were doing the actual work of teaching us what structured output is, and they're worth naming.

Microsoft Guidance was the earliest of them, with its repo created in November 2022, before ChatGPT had finished its launch news cycle. Guidance pioneered the idea of treating LLM output as something you compose with control flow, regex constraints, and grammars. Most of the patterns we now take for granted (JSON-from-grammar, structured generation as opposed to structured prompting) trace back here.

Outlines, launched in March 2023 by .txt, was the cleanest expression of grammar-constrained decoding. The thesis was straightforward: if you can write a grammar for the output, the sampler should refuse to emit anything that violates it. This is a profoundly correct idea that, three years later, is finally becoming the default in open-source inference engines like vLLM and XGrammar.

Instructor, shipped by jxnl on June 14, 2023, took a different path. It wrapped OpenAI's brand-new function calling and pretended the result was structured output. Define a Pydantic model, get a Pydantic model back. Retries handled. Validation handled. It was the right abstraction at the right moment, and it's part of why the term "structured outputs" stuck as the framing for the whole field.

llama.cpp added grammar-based sampling in July 2023, and that's when the conversation about constrained decoding got serious in the open-source world. The fact that you could ship a four-line grammar file and force a 7B model to emit perfect JSON every time was, at the time, witchcraft.

LangChain's output parsers, especially OutputFixingParser, standardized the retry-and-repair loop: parse, fail, send the validation error back to the model, ask it to fix the JSON, parse again. That pattern is everywhere now. It's also a lot of what people complain about when they complain about LangChain.

These libraries split into three approaches. Constrained decoding (Outlines, Guidance, llama.cpp grammars) intervenes at sampling time. Schema-driven wrappers (Instructor, later Pydantic AI) lean on the underlying API and add validation. Retry-and-repair loops (LangChain, Guardrails) treat the LLM as an unreliable black box and validate after the fact. All three still ship.

The fork in the road

By mid-2024 the field had bifurcated into camps that didn't always realize they were having the same argument.

The constrained-decoding camp said: structure should be enforced at sampling time, by the inference engine, before the model can emit anything wrong. Outlines and llama.cpp grammars are this. So is OpenAI's Structured Outputs feature, which almost certainly uses grammar-constrained decoding under the hood, building on research from llguidance and XGrammar.

The instruction-tuned camp said: train the model to emit schema-conformant output. OpenAI's function calling was the first major implementation. Anthropic's tool use, when it went GA in May 2024, was the second. The argument was that for closed-source APIs you don't control sampling, so you have to bake schema awareness into the model itself.

The wrapper camp said: it doesn't matter how the model emits the JSON, you still need validation, retries, and provider-agnosticism. Instructor and Pydantic AI (publicly launched December 2, 2024) are the cleanest expressions of this.

What actually won? All three did, layered on top of each other. For most app developers using closed APIs, OpenAI Structured Outputs and Anthropic tool use are the default. Underneath them, constrained decoding is the implementation. On top of them, validation wrappers like Instructor handle retries and content checks the schema can't enforce.

Hamel Husain wrote the canonical skeptical essay about wrappers in February 2024 (Fuck You, Show Me The Prompt), and he was right that many of these libraries are mostly prompt manipulation. He was also wrong that this means they're not useful. Prompt manipulation, well-engineered and validated, was the bridge that got us from "respond ONLY in valid JSON" to schema-strict APIs. Bridges are useful, and they're also temporary.

Three lessons we learned

The era taught us things, and the things are worth naming clearly because they're falling out of working memory already.

The model wasn't broken; the interface was

In early 2023 the consensus was that LLMs "couldn't follow instructions reliably." By late 2024 the consensus was that LLMs "follow JSON schemas with 100% reliability." The models hadn't fundamentally changed in that window. The interface had. When a model seems unreliable at a task, the most productive question to ask is "what does the API surface look like?" before "is the model good enough?"

Every workaround eventually becomes infrastructure

The retry-and-repair loop was a hack. It became LangChain's OutputFixingParser, then Instructor's tenacity-backed retries, then a built-in part of Pydantic AI. The "respond ONLY in valid JSON" prompt was a hack. It became, near-verbatim, the default system-prompt example in OpenAI's own documentation for years. The lesson is to take your hacks seriously, because the half-life of a "temporary" workaround in this field is approximately five years.

The constraint doesn't disappear, it moves up the stack

The 2022 problem was: how do I get this model to emit a parseable JSON object. By 2024 we had solved that. The 2025 problem became: how do I get this model to pick the right tool from a list of 250 of them, with each tool's schema preloaded into the system prompt, and not blow my context window before the model has emitted a single token. The shape of the problem didn't change. The layer changed. We're in the middle of the same pattern again.

Why we're forgetting the lessons

If the lessons of the structured-output era were propagating, the tool-calling and MCP era would look different than it does. It doesn't.

The clearest public example: in late 2025 / early 2026, Anthropic published an engineering post titled Code execution with MCP: building more efficient AI agents. The argument, in their own framing, is that direct tool calls don't scale because each tool definition consumes context, and a five-server MCP setup with 58 tools can burn ~50,000 tokens before the user has typed anything. Their proposed fix is to let the model write code that calls tools, instead of expecting the model to pick from a flat list of pre-loaded tool schemas. They report that lazy-loaded tool discovery improves Claude Opus 4.5 task accuracy from 79.5% to 88.1% while cutting tokens by roughly 85%.

That paragraph deserves to be re-read. Anthropic, the company that designed MCP, is publicly arguing that the way MCP currently works has a fundamental scaling problem, and the fix is to abandon flat tool lists in favor of letting the model write code. That maps cleanly onto the lessons above. The model isn't bad at tool calling; the interface is. The "load all your tools upfront" pattern is becoming infrastructure even though it was always a hack. The constraint moved up: from "make the model emit valid JSON" to "make the model pick the right tool from a flat list of 250."

A second example, with a useful counterpoint: Waleed K's piece The Evolution of AI Tool Use: MCP Went Sideways makes a related observation about MCP's context-bloat problem with concrete numbers and a concrete war story. He argues, drawing on Cloudflare's framing, that LLMs are "bad at tool calling" because tool-call traces are "out-of-distribution" for the base models. I think he's half right. The base distributions are absolutely thin on canonical tool-call traces, which is exactly why model providers fine-tune for tool use and why Anthropic's lazy-loaded tool search lifts Opus 4.5 from 79.5% to 88.1%. But the framing "models can't do tool calling" misses the same point we missed in 2022 about JSON. The model isn't bad at the task. We've handed it a clumsy interface to the task. Code execution feels like a fix because it routes around the clumsy interface and lets the model do something it has trillions of training examples for: write code. That's the same insight as "use a JSON Schema grammar." It's "the interface was wrong, again." The lesson generalizes, and it's the lesson worth carrying forward.

A third example, drawn from less rigorous evidence: production systems I've worked on in 2026 still ship system prompts that say things like "you are a backend tool router. Output ONLY a single tool call. Do not explain. Do not apologize." Strip the word "tool" and replace with "JSON object" and you have my 2023 prompt. Same shape, same hack, just at a different layer.

What the next "JSON repair library" looks like

Three predictions, with low hedging.

First, tool-search-as-routing becomes the default agent design pattern within twelve months. Flat tool lists in the system prompt will look as embarrassing in 2027 as "respond ONLY in valid JSON" looks now. The tool-search tool is the new JSON-Schema strict mode.

Second, agent-as-code-author beats agent-as-tool-picker for any non-trivial workflow. Anthropic's MCP-code-execution post is the most prominent signal, but the same pattern shows up in the way Claude Code agents get work done internally and in OpenAI's evolving Responses API semantics. We'll look back at "load 250 tools into the system prompt" the same way we look back at the kitten prompt.

Third, whatever standard replaces or absorbs MCP will be a protocol-of-protocols: a thin layer that brokers between code-executing agents and the underlying tool servers, rather than a flat schema dump. MCP solved the "describe your tools" problem. The next standard has to solve "let the model discover tools as needed without paying upfront context."

I have been writing system prompts that look like 2023 prompts again, this time around tool calling, and so has everyone else. The fix is the same one we figured out the first time: stop coercing, change the interface.

If you've shipped systems that hit this same wall, I'd be glad to compare notes.

— NJ

Building KaribuKit (AI-native PMS for hospitality), running Mara Hilltop (eco-lodge in the Maasai Mara), and consulting through SimbaStack.