Lesson 4: Prompt contracts, JSON schema & regression testing for LLMs

Introduction

Many teams fail not because the model is weak, but because prompts lack clear contracts. When prompts change, output drifts and the app silently breaks.

1. What Is a Prompt Contract?

A prompt contract consists of 4 parts:

• Role and task scope
• Input format
• Output format
• Failure behavior when data is insufficient

Example failure behavior: return an exact phrase like "insufficient data" instead of making things up.

2. JSON Schema for Output

Example schema for extracting action items:

{
  "type": "object",
  "required": ["summary", "items"],
  "properties": {
    "summary": {"type": "string"},
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["task", "owner"],
        "properties": {
          "task": {"type": "string"},
          "owner": {"type": "string"},
          "deadline": {"type": "string"}
        }
      }
    }
  }
}

Validate the schema immediately after receiving model output.

3. Designing Golden Tests

Create a test file containing:

• Input prompt
• Must include
• Must not include
• Schema pass/fail

Every time you change a prompt or model, re-run the entire test suite.

4. Error Classification

• Format error: doesn't match schema
• Grounding error: doesn't stick to context
• Safety error: returns sensitive data
• Instruction error: ignores system constraints

Clear error codes help track trends over time.

5. Prompt Versioning

Apply these rules:

• Every prompt has a version code
• Brief changelog for each change
• Quick rollback if quality drops

Never edit prompts directly in production without saving the version.

Demo Code

Prompt contract test results — 6/6 passed:

Source code: 03-prompt-contracts

Summary