How the Orchestrator Learned to Fix Itself

TL;DR

Ten days after the original orchestrator post, all three documented limitations are fixed: keyword-only persona selection upgraded to LLM-based with keyword fallback, "no retry logic" replaced by a 2-attempt system that injects failure context into the retry prompt, and "simple existence check" graduated to three-tier quality gates (existence, format, test). The meta-learning system identified these gaps as meta_gap entries before I prioritized the fixes.

Three Limitations, Ten Days

On February 3, I published a post about how multi-agent orchestration works. The last section was titled "The Honest Limitations," and it listed three:

1. Keyword-based persona selection misses nuance
2. No retry logic for flaky failures
3. Context handoff between phases is file-based

Ten days later, two of the three are fixed. The third turned out to be a feature, not a bug. But the interesting part is not that I fixed them — it's that the orchestrator identified them first.

The meta-learning system I built to capture agent feedback had been tagging PERSONA_MISMATCH and META entries during routine missions. When I reviewed the meta-learnings database, the same pain points I documented in the blog post were already sitting there, captured by agents encountering them in the field. The system was telling me what to fix, in order of severity, before I decided the priority myself.

This is the story of how those fixes work.

The Selector Got an LLM

The original persona selection was keyword matching. The word "research" in a phase description matched the researcher persona. "Security" matched security-auditor. It worked for obvious cases but missed anything nuanced — "investigate the performance of the auth system" would match researcher when performance-engineer was the better fit.

The fix: an LLM call as the primary selection path, with keyword matching as the fallback.

func Select(ctx context.Context, taskDescription string, personas *PersonaCatalog,
  skills *SkillPool, workspacePath string, metaContext string) (*MissionProfile, error) {

  prompt := buildSelectorPrompt(taskDescription, personas, skills, metaContext)

  opts := &sdk.AgentOptions{
      Cwd:            workspacePath,
      MaxTurns:       1,
      PermissionMode: sdk.PermissionAcceptEdits,
      SystemPrompt:   "You are a task analyzer. Select the best persona(s) and skill(s)...",
  }

  output, err := sdk.QueryText(ctx, prompt, opts)
  if err != nil {
      return selectByKeywords(taskDescription, personas, skills), nil
  }

  profile, err := parseSelectorOutput(output, personas, skills)
  if err != nil {
      return selectByKeywords(taskDescription, personas, skills), nil
  }

  profile.Skills = mergeDefaultSkills(profile.Personas, profile.Skills, personas)
  return profile, nil
}

The Select function returns a MissionProfile — not just a persona, but a bundle of recommended personas and skills for the entire mission. The decomposer then uses this profile to assign personas to individual phases.

type MissionProfile struct {
  Personas  []string // recommended persona names (1-4)
  Skills    []string // recommended skill names (1-4)
  Reasoning string   // one-line explanation
}

The design decision that matters most: double fallback. If the LLM call fails — timeout, rate limit, malformed output — the system falls back to keyword matching silently. No crash, no degraded mission. The selector just picks the best keyword match and moves on. Every mission gets a persona, even when the LLM is unavailable.

But the selector does something the keyword matcher never could. It reads historical feedback from the meta-learning database and adjusts its recommendations:

// Inject historical feedback between Mission and Rules when available
if metaContext != "" {
  sb.WriteString("
")
  sb.WriteString(metaContext)
}

That metaContext comes from FetchSelectorContext, which pulls unresolved meta_gap and meta_mismatch entries from the database. If agents have been flagging "wrong persona for this task" in previous missions, the selector sees those complaints in its prompt. The feedback loop closes: agents report problems, the selector reads the reports, future missions get better assignments.

## Historical Feedback

Past persona issues in this domain:
- persona gap: No available persona for Kubernetes networking expertise (3x)
- persona mismatch: blog-writer selected but task required code review (2x)

Retry That Learns from Failure

The original orchestrator had no retry logic. If a phase hit a rate limit or produced garbage output, it failed. The mission stopped. I had to manually re-run the whole thing.

The retry system now wraps every phase execution. The defaults are conservative: 2 retries, 5-second delay.

type RetryPolicy struct {
  MaxRetries           int           `json:"max_retries"`
  RetryDelay           time.Duration `json:"retry_delay"`
  EnableEnrichedContext bool         `json:"enable_enriched_context"`
}

func DefaultRetryPolicy() *RetryPolicy {
  return &RetryPolicy{
      MaxRetries:            2,
      RetryDelay:            5 * time.Second,
      EnableEnrichedContext:  true,
  }
}

But the interesting part is not the retry itself — it's what goes into the retry prompt. When a phase fails, the retry doesn't just re-run the same prompt. It injects three kinds of failure context:

func EnrichPromptForRetry(originalPrompt string, attempt int, failureOutput string,
  gateErrors []string, domain string) string {

  sb.WriteString(fmt.Sprintf("## Retry Attempt %d

", attempt))

  // Add failure output (truncated to 2000 chars)
  if failureOutput != "" {
      sb.WriteString("### Previous Failure Output

")
  }

  // Add gate errors
  if len(gateErrors) > 0 {
      sb.WriteString("### Gate Failures

")
  }

  // Inject gotcha learnings from the learning store
  gotchas := fetchGotchaLearnings(domain)
  if gotchas != "" {
      sb.WriteString("### Known Gotchas

")
  }
}

The retry prompt tells the agent: here's what you produced last time, here's why the quality gate rejected it, and here are gotcha-type learnings from the database that might be relevant. The agent sees its own failure and gets advice from past agents who hit similar problems.

This is the throughline connecting the retry system to the learnings system. The 360 error-type learnings in the database are not just historical records. They're ammunition for retry prompts. An agent that failed because go build choked on a missing embed directive gets a retry prompt that includes GOTCHA: go build fails silently when embedding directive references missing file — because a previous agent already discovered that and tagged it.

Quality Gates Graduated

The original post described quality gates as "a simple existence check." Did the agent produce output? Is the file non-empty? That's it. If the file existed, the gate passed.

The graduated system has three levels:

const (
  GateExistence GateLevel = "existence" // Files exist with >100 bytes
  GateFormat    GateLevel = "format"    // Files are valid format (JSON/YAML/Go)
  GateTest      GateLevel = "test"      // Mission-defined test commands pass
)

Each level subsumes the previous. A format gate checks existence first, then validates syntax. A test gate checks existence, then format, then runs shell commands.

func RunGate(config *GateConfig, phaseArtifactDir string, workDir string) *GateResult {
  // Level 1: Existence gate
  if err := runExistenceGate(config, phaseArtifactDir, minSize); err != nil {
      result.Errors = append(result.Errors, err...)
      return result
  }
  if config.Level == GateExistence {
      result.Passed = true
      return result
  }

  // Level 2: Format gate
  if err := runFormatGate(phaseArtifactDir); err != nil {
      // ...
  }
  if config.Level == GateFormat {
      result.Passed = true
      return result
  }

  // Level 3: Test gate
  if err := runTestGate(config.TestCommands, workDir); err != nil {
      // ...
  }

  result.Passed = true
  result.Details = "all gates passed"
  return result
}

The format gate validates Go files with gofmt, JSON files with json.Valid, and YAML files with a syntax check. This catches a specific class of agent failure I kept seeing: an agent would produce a Go file that had the right structure but contained syntax errors — a missing closing brace, an unclosed string literal. The existence gate would pass because the file had content. The format gate catches it.

The test gate is the most powerful. Missions can define shell commands that must pass: go build ./..., go test ./..., npm run lint. If the mission specifies test commands, the gate runs them against the agent's output. Failure feeds back into the retry system — the agent sees the exact test failure and tries again.

The graduated approach matters because not every phase needs the same level of validation. A research phase that produces markdown gets an existence gate. An implementation phase that produces Go code gets a format gate. A phase with specific acceptance criteria gets a test gate. The orchestrator matches the gate level to the task.

The Self-Improving Loop

The meta-learning system deserves its own section here, because it's the mechanism that connected these fixes to each other. I wrote about the data it produces and the capture mechanics separately. But the self-improvement loop — where the system identifies its own shortcomings and surfaces them for fixes — is new.

Every agent in every mission gets a compact meta-marker prompt appended to its instructions:

If you notice issues with your role, instructions, or capabilities, flag them:
- AGENT_ISSUE: problem with your assigned role or constraints
- PERSONA_GAP: task needs expertise not covered by available personas
- PERSONA_MISMATCH: you were the wrong choice for this task
- CAPABILITY_REQUEST: you need a tool or skill that's not available
- META: observation about agent coordination or orchestration

Agents use these markers naturally. A researcher assigned to an implementation task writes PERSONA_MISMATCH: I'm a researcher but this phase requires writing Go code — a writer persona would be more effective. A security auditor who needs Kubernetes expertise writes PERSONA_GAP: No persona covers Kubernetes networking analysis.

These markers flow through the same capture pipeline as regular learnings, but they land in a separate meta_learnings table. The meta-learning store deduplicates them, tracks occurrence counts, and surfaces the highest-frequency complaints to the selector. The loop looks like this:

1. Agents run missions and flag problems (PERSONA_MISMATCH, PERSONA_GAP)
2. Meta-learnings accumulate in the database with occurrence counts
3. FetchSelectorContext pulls the top unresolved complaints
4. The LLM selector reads those complaints before picking personas for the next mission
5. Future missions get better persona assignments
6. When a fix ships (like the LLM selector replacing keyword matching), the meta-learning gets resolved

The validation is built in too. When I mark a meta-learning as resolved, the system can re-validate the resolution against current state — checking whether the claimed fix actually holds. If a persona was added but later removed, the resolution becomes stale and the gap resurfaces.

Skill Materialization at Spawn Time

One more capability that shipped since the original post: skills are now discovered, indexed, and injected into agent workspaces at spawn time.

func (sp *SkillPool) Materialize(names []string, targetDir string) error {
  skillsDir := filepath.Join(targetDir, ".claude", "skills")
  for _, name := range names {
      meta := sp.Get(name)
      if meta.Source == "system" {
          // Symlink system skills
          os.Symlink(meta.BasePath, destDir)
      } else {
          // Copy pool skills (agents are self-contained)
          copyDir(meta.BasePath, destDir)
      }
  }
}

The skill pool has two tiers: pool skills that live in ~/.via/skill-pool/ and system skills in ~/.claude/skills/. When the LLM selector recommends skills for a mission, SpawnForPhaseWithPersona materializes them into the agent's workspace directory. System skills get symlinked (they're shared). Pool skills get copied (each agent is self-contained).

The result: an agent spawned for a Go implementation task gets the go-programming skill materialized into its workspace, along with any mission-specific skills the selector recommended. The skill content appears in the agent's CLAUDE.md via an auto-generated index. The agent has domain knowledge without needing it in its base prompt.

func SpawnForPhaseWithPersona(ws *workspace.Workspace, personaName, provider, phaseID string,
  persona *PersonaDefinition, agentCatalog *AgentCatalog,
  skillPool *SkillPool, skillNames []string) (*SpawnedAgent, error) {

  // Compose persona into AgentDefinition by merging with archetype
  agentDef := persona.ComposeAgentDefinition(personaName, agentCatalog)

  // Generate CLAUDE.md from composed definition
  templateContent := agentDef.RenderClaudeMD(personaName)

  // Materialize skills into agent .claude/skills/
  if skillPool != nil && len(skillNames) > 0 {
      skillPool.Materialize(skillNames, agentPath)
      if idx, err := GenerateSkillsIndex(skillsDir); err == nil && idx != "" {
          templateContent += "
" + idx + "
"
      }
  }

  // Register persona markers with learning system
  learning.RegisterCatalogMarkers(personaName, catalogMarkersToLearning(persona.Markers))
}

The Honest Limitations

The orchestrator is meaningfully better than it was ten days ago, but the gaps are still real:

Context handoff is still file-based. I listed this as a limitation in the original post, and it remains. Phase 1 writes markdown, Phase 2 reads it. There's no back-channel for clarification. But after 130 workspaces, this has caused fewer problems than I expected. The BFS dependency walk means each phase only sees output from its direct upstream dependencies, which keeps the context focused. File-based handoff is a feature for isolation, even if it's a limitation for dialogue.

No learning quality feedback. The learnings system tracks how often a learning is seen (via dedup counts) and how often it's injected (via used_count). But there's no signal for whether the injected learning actually helped the agent. An irrelevant learning wastes tokens in the prompt. I need a mechanism — possibly gate pass/fail correlation — to score learning utility.

Gate-retry coupling is simple. When a gate fails, the retry injects the gate errors verbatim. But the retry doesn't adapt its strategy. A format gate failure and a test gate failure get the same treatment: paste the error, add gotcha learnings, try again. Smarter retry strategies — like automatically reducing scope on format failures or requesting a different approach on test failures — would improve the success rate.

Meta-learning resolution is manual. I resolve meta-learnings by hand with meta-resolve. The system validates that resolutions still hold, but it can't automatically resolve them when a fix ships. Connecting code changes to meta-learning resolution would close the loop fully.

Next in series: Fault Tolerance for AI Agents

Related Reading

• How Multi-Agent Orchestration Works — The original post documenting the three limitations that this post addresses
• Teaching AI to Learn From Its Mistakes — The learnings system that captures and injects knowledge across agent runs
• What 1,600+ AI Learnings Reveal — Data analysis of the meta-learnings that drove these improvements