Evaluating LLM Applications

1. WHAT is LLM Evaluation?

In classical software development, testing is a resolved problem. You write functional unit tests with exact assertions: assert calculate_total(100, 0.05) == 105. But LLMs are probabilistic engines—they generate the next most likely token based on a probability distribution. This introduces the non-deterministic testing problem: the exact same input can yield slightly different outputs every time.

LLM Evaluation is the structured discipline of assessing the accuracy, robustness, toxicity, speed, cost, and alignment of natural language outputs. Rather than testing for binary string matches, we measure semantic semantic overlap, logical reasoning, and adherence to constraints using three primary methodologies:

A. Traditional Heuristic Metrics (N-Gram Overlaps)

Historically, NLP models were evaluated using statistical comparisons against reference texts:

• BLEU (Bilingual Evaluation Understudy): Primarily used in translation. It calculates precision by matching n-grams from the generated text against the reference text.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Primarily used in summarization. It measures recall by counting how many n-grams in the reference text appear in the generated output.

The Crucial Flaw: These metrics completely ignore context and synonyms. If the target answer is “The patient should take the medication at night,” and your LLM outputs “The individual must consume their pills before bed,” BLEU/ROUGE will award a score close to 0 despite the meaning being 100% correct. Therefore, they are rarely used for modern generative agents.

B. Semantic Similarity (Embeddings)

By transforming both the generated text and reference text into high-dimensional vector representations using models like text-embedding-3-small, we can calculate the Cosine Similarity between the vectors. If the angle between the vectors is near 0, the meaning is highly aligned, bypassing the synonym limitation of BLEU/ROUGE.

C. LLM-as-a-Judge (Hybrid Cognitive Evals)

Standardized by frameworks like **G-Eval**, this approach uses advanced, highly capable frontier models (like GPT-4o or Claude 3.5 Sonnet) as the evaluators. The judge model is fed a strict scoring rubric, a Chain-of-Thought instruction, and the output, returning a numerical grade (usually 1-5) and written rationale.

2. WHY Do We Need It?

Relying on manual “vibe checks” (trying 5 prompts and deploying if they look good) is the leading cause of LLM failures in production. Evals are critical for several reasons:

• The Regression Trap: Modifying a system prompt to fix a bug in Edge Case A might inadvertently introduce hallucinations or safety vulnerabilities in Edge Cases B through Z. Automated evaluation suites catch these regressions before they reach production.
• Model Drift and Upstream Updates: Model APIs (e.g., GPT-4 updates) undergo silent updates. You need a daily benchmark suite to ensure that your prompt performance hasn’t decayed.
• Quantifying Inflection Points: Should you deploy a cheaper 8B parameter model over a premium 70B model? Without automated evals, you cannot calculate the cost-to-performance inflection point to optimize your API spend.
• Calibration & Human Correlation: Evals allow you to compute correlation coefficients (such as Cohen’s Kappa) to ensure your automated LLM Judge aligns with real human reviews, giving you a statistically reliable test suite.

3. WHEN Do We Evaluate?

LLM evaluation is a continuous lifecycle integrated into every step of development:

Phase 1: Development Time (Offline Evals)

When: While writing initial prompts and selecting models.
Focus: Fast iteration. You test variations manually or on small datasets (10-30 samples) using local playground tools to establish baseline quality.

Phase 2: Pre-Production & CI/CD (Gated Evals)

When: Before merging code or deploying prompt templates.
Focus: The “Golden Dataset.” An automated script runs your LLM app against 100-500 curated inputs (edge cases, typical queries, adversarial attacks) and blocks deployment if scores decay.

Phase 3: Production (Online Observability)

When: Live monitoring of real user interactions.
Focus: High-volume, low-overhead analytics. You log input-output pairs, capture explicit feedback (likes/dislikes), and passively sample 1-5% of traffic to route to your Judge model to check for hallucinations in the wild.

4. APPLICATION: Scenario-Specific Evals

A single metric cannot evaluate all applications. Your eval metrics must match the system architecture:

5. HOW to Implement It: Pydantic & Structured Evals

Rather than using raw text prompts that can output unstructured summaries, we can use OpenAI Structured Outputs combined with Pydantic to build a robust LLM-as-a-Judge system that always returns clean JSON. Here is the complete implementation:

import os
from typing import List
from pydantic import BaseModel, Field
from openai import OpenAI

# 1. Initialize OpenAI client
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY", "your-api-key-here"))

# 2. Define the Pydantic Schema for the Evaluation Output
class EvaluationResult(BaseModel):
    reasoning: str = Field(
        description="Step-by-step reasoning explaining the score. Highlight any claims not supported by the context."
    )
    score: int = Field(
        description="The groundedness score from 1 to 5. 1 = completely hallucinated, 5 = 100% grounded in context."
    )
    unsupported_claims: List[str] = Field(
        description="List of specific sentences or assertions in the answer that were not found in the context."
    )

# 3. Our Golden Dataset
golden_dataset = [
    {
        "query": "What is the return policy?",
        "context": "Items can be returned within 30 days of receipt. Returns must be in their original packaging.",
        "answer": "You can return your purchase up to 30 days after receiving it, provided it is in its original box."
    },
    {
        "query": "Is shipping free?",
        "context": "Free standard shipping applies to orders over $50. Otherwise, shipping costs a flat rate of $5.99.",
        "answer": "Yes! Standard shipping is free on all orders with no minimum purchase requirement, and we deliver worldwide." # Hallucination!
    }
]

# 4. Evaluator Function using Structured Outputs
def evaluate_groundedness(context: str, answer: str) -> EvaluationResult:
    prompt = f"""
    Evaluate the following GENERATED ANSWER against the provided CONTEXT. 
    Verify if the generated answer is fully supported by the context without hallucinating facts.
    
    CONTEXT:
    {context}
    
    GENERATED ANSWER:
    {answer}
    """
    
    # We call beta.chat.completions.parse to enforce the Pydantic schema
    completion = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system", 
                "content": "You are a precise quality control judge enforcing factuality and alignment."
            },
            {"role": "user", "content": prompt}
        ],
        response_format=EvaluationResult,
        temperature=0.0
    )
    
    return completion.choices[0].message.parsed

# 5. Executing the Evaluation Loop
def main():
    print("=== STARTING SEMANTIC EVALUATION SUITE ===")
    
    for i, test in enumerate(golden_dataset):
        print(f"\n--- Running Test #{i+1} ---")
        print(f"Query: {test['query']}")
        
        # Run the evaluator
        result = evaluate_groundedness(test["context"], test["answer"])
        
        print(f"Groundedness Score: {result.score}/5")
        print(f"Reasoning: {result.reasoning}")
        if result.unsupported_claims:
            print(f"Detected Hallucinations: {result.unsupported_claims}")
            
    print("\n=== EVALUATION COMPLETE ===")

if __name__ == "__main__":
    main()

6. Advanced Evaluation Concepts

To take your eval engineering to the senior developer level, implement these strategies:

• Synthetic Data Generation (Bootstrap Evals): You don’t have to wait for users to create a dataset. You can feed your documentation files to an LLM and prompt it to generate 100 realistic questions, contexts, and reference answers to seed your test suite.
• Evaluation Alignment (Kappa Scoring): Have 3 humans manually score a sample of 50 LLM outputs. Then run your LLM-as-a-Judge script on those same 50 outputs. Calculate Cohen’s Kappa to see how well the LLM matches human judgment. If correlation is low, refine the judge’s scoring rubric.