LLM-as-a-Judge Evaluation: Automated Structured Evals using Pydantic

1. Deep Architecture

Free-form LLM outputs are hard to parse. Using structured outputs, we configure LLM APIs to return JSON matching a specific schema. The API limits output choices to match the schema, which we parse into Pydantic models for validation.

2. The Feynman Gatekeeper

[KNOWLEDGE CHECK] Explain how logit constraints at the LLM decoder layer guarantee that outputs match a specific JSON schema.

3. The Code

from pydantic import BaseModel, Field

class ScriptReport(BaseModel):
    hook_score: int = Field(..., description="Hook quality from 1 to 10")
    retention_critique: str = Field(..., description="Feedback on pacing")
    suggestions: list[str] = Field(default_factory=list)

4. The Funnel

Stat Level-Up: Calibration Lead (Lvl 1).
Sanjaya Integration: Standardize video script scores before saving results to the database.