Three lessons about getting more predictable results from a multi-agent AI system

Same prompt, different results. And the fix had nothing to do with my prompts.

I've spent the last month building a complete video course with AI doing most of the production work. Descript's AI agent handles the timeline edits: adding text, shapes, images, and video clips. Claude Code orchestrates everything around it. And for a while, it was a mess... the kind of mess that makes you question whether any of this AI technology is ready for real work.

I've found that it is ready. But I had to learn three important lessons. And based on what I've learned so far, and the systems I've built, these lessons transfer to anything you build with AI agents. Here's what happened.

The problem: no pattern

After a couple of hours working on the first few videos, I was getting unusable results from Descript's AI. Same prompt, same stated intent, completely different videos.

I blamed my prompts and immediately started iterating. I tightened the formatting, stripped things out, tested variation after variation. Every test seemed to prove me right, but every hypothesis proved to be incorrect the next time I tried to add another element to the timeline.

Then we learned that Descript defaults to "Auto" for the model, for every request. Meaning... the model that Descript employs on its end varies from prompt to prompt. Turns out I wasn't actually debugging MY inputs at all.

The bake off

To start taking control of the situation, I began isolating variables. I started by specifying a model to use rather than relying on "Auto". I pinned each model and sent the SAME job four different ways, creating a fresh composition in an isolated project each time. I graded each one by reviewing manually against what I expected.

claude-sonnet-5: 4 usable builds out of 4, at 7 credits per build
claude-opus-4.8: 4 out of 4, at 14 credits
gemini-flash: 3 out of 4, at 7 credits
haiku: 1 out of 4
gpt-5.4: 0 out of 4
gpt-5.5: 1 out of 1, at 49 credits per build (I stopped testing it there)
"Auto": anywhere from 0 out of 4 to 4 out of 4

The only variable was the model. The newly released Claude Sonnet 5 was perfect on renderings and only half the cost of Opus 4.8. GPT-5.5 worked, but used WAY more credits than needed.

Descript's model lineup has already changed since I ran this test: Claude Fable 5 joined the roster within a couple of weeks. It will keep changing, and that churn is part of the point.

Lesson 1: Predictability and repeatability starts with choosing a model.

I pinned ALL future AI calls to Sonnet 5. It gave me the best results at the most cost-effective credit use. Now, when any system I use starts behaving unpredictably, I dig in to look for sources of variability that I didn't account for.

What pinning the model fixed, and what it didn't

Pinning the model worked REALLY well for treatments I had specified in the design system. I created 4 locked components with variants, specifically for reuse across 14 different videos, and they came out consistent, build after build.

It did NOT, however, improve the model's ability to execute on adding and editing clips and syncing them to the spoken audio tracks. Descript was still struggling. Why?

Lesson 2: Eliminate the room for interpretation.

Claude defaulted to prose wherever it could. Despite design system components fully encoded in deterministic JSON (more on the design system I used coming soon), it was using prose-based prompts for anything to do with adding clips or stills to the timeline. This was exactly the opposite of what I wanted. I specified to use JSON for all parts of the Descript agent interaction. Claude just... ignored me.

I kept seeing results that baffled me... and each time, I asked "Are you sending those prompts in JSON or prose?" Guess which format it had used? You already know the answer.

EVERY. TIME.

Prose was failing. Prose descriptions left room for interpretation, and interpretation was exactly what I needed to eliminate for positioning, timing, and edits. So we updated the system to ensure that details for all clips and stills were specified in JSON as well.

Rules don't hold. Mechanisms do.

Here's the part that surprised me most, and I think it matters well beyond video production. I created a carefully written rule: every instruction to the editing agent goes out as JSON. Claude had even added it to its memory. And yet, the rule kept getting broken anyway.

Lesson 3: You need enforcement.

I had Claude Code build the scripts that prepare each payload for Descript's agent. What I hadn't done is force those scripts to ONLY allow JSON. Now? No JSON, no send. The script just errors out and lets me know.

After that, I reviewed everything else I was doing and built enforcement in BEFORE allowing Claude to send off any prompts to Descript's AI. Every rule I cared about became either a program with an exit code, or a path that simply doesn't exist anymore.

A few examples of what these guardrails look like in practice:

The payload script refuses to send anything that isn't JSON. Not a warning. An error.
No request goes out without naming its model. "Auto" isn't a choice the code can make.
Image and video generation stops at a spending cap, instead of trusting anyone to remember one.
A build can't start until its plan has been approved. If the plan hasn't been signed off, the build refuses to run.
The agent can claim an edit is done, but "done" only happens when a human looks at the rendered frame.
The finished module cannot be pushed to Cloudflare unless it has its full set of receipts: approvals, design decisions, and a pass on the full suite of tests and checks.

A rule you write down (or tell AI to follow) is really just a wish. If your team has a checklist that everyone has read but nobody consistently runs, you already know this feeling, whether your workers are agents or people.

Where this leaves me

Is this system perfect? Nope.

Will I still get some unpredictable results? Very likely.

But building these three things into the system took me from an unpredictable and frustrating mess of results to a predictable system that gets me 90 to 95% of the way to done, every time. Pin the variables. Eliminate interpretation where you need precision. Enforce with mechanisms, not intentions.

These three lessons now sit at the foundation of how I design and architect AI systems.