Studio notes — robustness before trust

Every week, the same fleet of agents is at work across several clients at once. This week, no new commits landed at any client — ten days without a code change. Rather than manufacture a story, we look back at two builds already running in production across the studio, picked for what they teach a business owner who never opens a terminal: the reliability of automatic document reading at an insurance broker, and the access framework built around an agent at a courier company.

An insurance broker: never trust a single pass at reading a document

At this broker, the agent has to cross-reference several policies and several contracts at once to spot a coverage gap — a missing clause, an insufficient cap. The first build wasn’t the analysis itself but what precedes it: the original PDF-reading library broke on certain files in the serverless environment, and was replaced with a more robust one, tested specifically for rejecting non-conforming files, before the tool was even allowed to run a combined analysis across several documents. A study by Dan Hendrycks and coauthors, published at NeurIPS Datasets and Benchmarks 2021 under the name CUAD, shows that even the best models still lag well behind a human expert at spotting the clauses that matter in a contract — a reminder that the document-ingestion step deserves as much engineering care as the model consuming it afterward. Lesson for a business owner: before trusting an automated document analysis, check that the reading step was tested on your real documents — inconsistent layouts, scans, corrupted files — not just the clean case shown in a demo.

A courier company: who gets to act, and how to show it off without risk

At this courier company, the studio added roles and teams around the operations dashboard, so only the right people can approve or edit an action the agent proposes — and built, alongside it, a demo version of that same dashboard, fed by automatically generated synthetic data, with no login and no access to a single real record. A business owner can watch the agent run, and test new access profiles, before handing either one anything in production. A study by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, run across more than 5,000 customer-support agents and published by the NBER in 2023, measured productivity gains concentrated among the least experienced — up to 34% — precisely when the AI’s suggestions stayed visible and editable rather than acting alone. Lesson for a CIO: an access framework (who can approve what) and a demo mode with no real data aren’t side details — they’re what lets a team ramp up on an AI agent in production without exposing customer data along the way.

What connects both builds: before discussing a model’s performance, the studio always checks two things first — that what it reads is reliable, and that what it can do is bounded by clear roles you can test without touching a single real record. That discipline, more than the model in use, decides whether an AI agent in production holds up over time.