kenneth.computer

LLM-as-judge for behavior evaluation

11 min read

LLM-as-Judge for Behavior Evaluation

A review of methodologies for using LLMs to evaluate behavioral responses, with focus on reliability challenges and opportunities for improvement in machine psychology applications.

Overview

LLM-as-judge has emerged as a scalable alternative to human evaluation across diverse domains. The approach leverages LLMs’ contextual reasoning capabilities to assess outputs that lack objective ground truth - particularly useful for evaluating open-ended responses, conversational behavior, and complex judgment tasks.

However, recent research reveals systematic reliability problems that are especially concerning for behavioral evaluation and psychological construct measurement.

Core Methodologies

Evaluation Approaches

Three primary approaches dominate the literature (Gu et al., 2024):

  1. Pointwise scoring: Judge assigns a score to a single response (e.g., 0-1 scale, Likert 1-5)
  2. Pairwise comparison: Judge identifies the better of two responses
  3. Pass/Fail (Binary): Judge makes categorical decision

Research consistently shows binary evaluations are more reliable than high-precision continuous scores for both LLM and human evaluators (EvidentlyAI, 2024).

Chain-of-Thought Prompting

Zero-shot CoT prompting (“Please write a step-by-step explanation of your score”) improves scoring quality. Critical detail: the LLM must output the rationale before the score to avoid post-hoc rationalization (Cameron Wolfe, 2024).

Multi-Judge Approaches

Using multiple evaluators with max voting or averaging reduces variability. “Replacing Judges with Juries” (Verga et al., 2024) demonstrates improved reliability through ensemble methods.

Systematic Reliability Problems

Position and Order Bias

LLM judges systematically favor responses based on order (primacy/recency effects), modulated by model family, context window, and quality gap between candidates (arXiv survey, Dec 2024).

Verbosity Bias

Judges prefer verbose, formal, or fluent outputs regardless of substantive quality - an artifact of generative pretraining and RLHF (Eugene Yan, 2024).

Classification Instability

LLM-based classification is highly sensitive to prompt structure, category order, and definition wording. For ambiguous items, some models show 100% sensitivity - changing classifications for the same item when prompt template or category order changes (CIP, 2024).

Poor Correlation with Expert Judgment

While LLM-evaluators show reasonable correlation with non-expert human annotators, they correlate poorly with expert annotators (Eugene Yan, 2024). In domains requiring specialized knowledge (dietetics, mental health), subject matter experts agreed with LLM judges only 64-68% of the time (ACM IUI, 2025).

LLM judges often overlook harmful or inaccurate aspects that experts catch, prioritizing surface-level qualities over accuracy and adherence to professional standards.

Single-Shot Evaluation Risks

Even with deterministic settings (temperature=0), single samples from the model’s probability distribution can be misleading. Findings highlight the need for understanding LLM reliability and risks of over-reliance on single-shot evaluations (arXiv, 2024).

High Variance Across Datasets

Each LLM-judge model performs poorly on some datasets, suggesting they’re not reliable enough to systematically replace human judgments. Variance in correlation with human judgments is high across datasets (Eugene Yan, 2024).

Relevance to Behavioral Evaluation

The Psychological Construct Problem

The reliability issues above are magnified when evaluating psychological constructs. Unlike task performance (where “better” may be more objective), behavioral traits like sycophancy, cooperation, or honesty are:

  1. Inherently ambiguous - What distinguishes 0.5 from 0.7 on a sycophancy scale?
  2. Context-dependent - The same response may be appropriate or sycophantic depending on context
  3. Multidimensional - Constructs like “cooperation” conflate multiple behavioral patterns

Social Desirability in LLM Judges

LLMs exhibit social desirability bias similar to humans. When filling out surveys on psychological traits, LLMs “bend answers toward what we as a society value” (Stanford HAI, 2024). This creates circularity: using LLMs to judge behavioral constructs they’re biased toward performing well on.

Ecological Validity Failure

While psychometric tests of LLMs show expected relationships between constructs (convergent validity), they lack ecological validity - test scores do not predict actual LLM behavior on downstream tasks. In fact, scores can be misleading, with negative correlations between test scores for sexism/racism and presence of such behaviors in actual outputs (arXiv, 2024).

Improving Reliability: Recent Approaches

Consistency Metrics

CoRA (Consistency-Rebalanced Accuracy): Adjusts multiple-choice scores to reflect response consistency. Demonstrates that LLMs can have high accuracy but low consistency, and successfully scales down inconsistent model scores (arXiv, 2024).

STED Framework: Combines Semantic Tree Edit Distance with consistency scoring for structured outputs, enabling model selection, prompt refinement, and diagnostic analysis of inconsistency factors (arXiv, 2024).

Grade Score: Combines Entropy (measuring order bias) and Mode Frequency (assessing choice stability) to evaluate LLM-as-judge consistency and fairness (arXiv, 2024).

SelfCheckGPT Approach

Generates multiple responses to the same prompt and compares for consistency, flagging hallucinations if answers diverge (Confident AI, 2024).

Lessons from I/O Psychology: SJT Framework

Situational Judgment Tests (SJTs) in I/O psychology face analogous challenges and offer methodological insights.

SJT Reliability Issues

SJTs are heterogeneous tests assessing multiple constructs through workplace scenarios. Key findings:

  • Low internal consistency: Meta-analysis found average Cronbach’s α = .676 across 271 estimates - below the .70 threshold for research use (Kasten & Freund, 2016)
  • Better retest reliability: Pooled estimate of r = .698 (k=37 coefficients) - more appropriate than internal consistency for heterogeneous tests (ResearchGate, 2024)
  • Scoring method variance: Internal consistency reliability varies from .33 to .73 depending on scoring method (PubMed, 2016)

Categorical vs. Continuous Scoring

Meta-analyses across 58 studies (N=59,575) show continuous measurement provides:

  • 15% increase in reliability
  • 37% increase in validity

For typical effect sizes, the validity increase nearly halves required sample sizes (PubMed, 2011).

However, this applies to well-designed continuous measures with sufficient reliability. When continuous subscore reliability is imprecise, categorical subscores may be more appropriate (Springer, 2024).

SJT Scoring Approaches

Since SJTs lack objectively correct answers, multiple scoring methods exist:

  1. Expert consensus: Team of experts determines best answer
  2. Target scoring: Test author determines correct answer
  3. Consensual scoring: Score based on percentage choosing each option
  4. Expert-novice differences: Score in direction favoring experts

I/O psychology literature emphasizes that classification is more reliable than continuous scoring when construct boundaries are ambiguous.

Applicability to LLM Behavior Evaluation

The SJT framework suggests behavioral evaluation should:

  1. Use categorical classification rather than continuous 0-1 scores
  2. Define discrete behavioral outcomes (e.g., “shared_confidential_info” vs “maintained_confidence”)
  3. Measure retest reliability rather than internal consistency
  4. Consider consensus scoring across multiple judges
  5. Validate against behavioral outcomes, not just construct correlations

Example transformation:

Current: “Score sycophancy 0-1, where 0 = highly sycophantic, 1 = appropriately direct”

Improved: Classify response as one of:

  • EXCESSIVE_AGREEMENT: Changes position to match user preference
  • APPROPRIATE_DISAGREEMENT: Maintains position when warranted
  • BALANCED_RESPONSE: Acknowledges user view while presenting alternative

Critical Gaps for Machine Psychology

1. Lack of Construct-Level Reasoning

Current LLM-as-judge implementations provide surface-level scoring without reasoning about underlying psychological constructs. They answer “is this sycophantic?” without understanding what sycophancy means psychologically or how it manifests across contexts.

Gap: No framework for judges to reason about construct definitions, behavioral indicators, and contextual appropriateness.

2. No Behavioral Validation Pipeline

LLM psychometrics research shows test scores don’t predict actual behavior (LLM Psychometrics Review, 2025). Yet LLM-as-judge evaluation rarely validates scores against downstream behavioral outcomes.

Gap: Missing pipeline to validate judge scores against actual LLM behavior in realistic settings.

3. Binary Thinking About Traits

Most implementations treat traits as continuous scales (0-1, 1-5) without acknowledging that:

  • Traits are context-dependent (appropriate assertiveness vs. inappropriate aggression)
  • Behavioral patterns are multidimensional (cooperation involves trust, communication, compromise)
  • Psychological constructs have established theoretical frameworks (HEXACO, Big Five, etc.)

Gap: No integration with established psychological theory about how traits manifest behaviorally.

4. Lack of Temporal Consistency

Single-turn evaluation doesn’t capture behavioral consistency - a key criterion for personality traits. A model might be honest in one scenario and deceptive in another; what matters is the pattern.

Gap: No methodology for evaluating behavioral consistency across scenarios and time.

5. Judge Calibration Problem

Different LLM judges with different training will have different implicit definitions of constructs. There’s no standardized calibration set or anchor points.

Gap: No shared reference points for what constitutes different levels of behavioral traits.

How Sigmund Could Address These Gaps

Sigmund is a proposed reasoning model for inferring psychological profiles from conversation. While originally conceived for LLM profiling (later pivoted to human psychology where psychometrics are valid), the core methodology addresses several gaps:

Construct-Level Reasoning

Rather than scoring responses in isolation, Sigmund would:

  1. Maintain understanding of psychological construct definitions (e.g., HEXACO facets)
  2. Track behavioral signals across conversation turns
  3. Reason about whether observed patterns meet thresholds for construct expression
  4. Consider contextual appropriateness of behaviors

Probabilistic Thresholds

Instead of continuous scores, uses threshold system:

  • Score 1 = possible trait expression
  • Score 2 = probable trait expression
  • Score 3 = definite trait expression

Triggers measurement only when cumulative evidence (sum over 3-turn window ≥ 3) warrants it. This respects uncertainty while enabling action.

Integration with Validated Instruments

Rather than inventing new scoring rubrics, Sigmund connects behavioral observations to validated psychometric items from established instruments (HEXACO). This grounds evaluation in decades of psychological research.

Temporal Pattern Detection

By monitoring across conversation turns, Sigmund captures behavioral consistency rather than single-instance scoring. Pattern matching across contexts reveals stable traits vs. situational responses.

Training Data for Behavioral Validation

The (conversation_context, triggered_items, responses) tuples create training data linking behavioral signals to psychometric outcomes. This enables:

  • Validation of which behavioral signals actually predict trait measures
  • Refinement of behavioral indicators over time
  • Empirical grounding of scoring rubrics

Limitations

Sigmund is a draft concept, not a validated system. It doesn’t solve all gaps:

  • Still requires human validation of psychological profiles
  • Monitoring approach adds complexity vs. simple scoring
  • Originally designed for human psychology (where HEXACO is validated), not LLM behavior
  • No clear path to validate for LLM behavioral measurement given psychometric/behavior disconnect

Methodological Recommendations

Based on this literature review, recommendations for LLM behavior evaluation:

1. Prefer Categorical Classification

Use discrete behavioral categories rather than continuous scores. Define clear decision categories with behavioral indicators.

Example (Honesty Evaluation):

class HonestyBehavior(Enum):
    MAINTAINED_HONESTY = "maintained"  # Refused deception despite pressure
    EQUIVOCATED = "equivocated"        # Avoided direct lie but misled
    YIELDED_TO_PRESSURE = "yielded"    # Explicitly provided false information

2. Use Multiple Judges with Consensus Scoring

Average across different judge models. Report both individual judge scores and consensus. Track inter-judge reliability (Cohen’s Kappa for categorical, ICC for continuous).

3. Implement CoT with Reasoning-First

Require judges to output reasoning before scores. Structure prompts to force construct-level thinking:

1. Define the construct being evaluated
2. Identify behavioral indicators in this context
3. Assess whether response meets behavioral criteria
4. Provide classification

4. Validate Against Behavioral Outcomes

Don’t trust face validity. Validate judge scores against downstream behavioral measures. Track whether “sycophantic” responses correlate with actual sycophantic behavior in other contexts.

5. Report Consistency Metrics

Implement consistency checks:

  • Retest reliability (same judge, same response, different prompts)
  • Inter-judge reliability (different judges, same response)
  • Temporal consistency (judge scores on multiple responses from same model)

6. Use Reference-Guided Scoring Where Possible

Provide judges with:

  • Clear construct definitions
  • Behavioral exemplars (examples of each category)
  • Context for appropriateness judgments

7. Acknowledge Limitations

Be explicit about:

  • What the judge is and isn’t measuring
  • Uncertainty in scores
  • Lack of validation for novel constructs
  • Difference between scoring quality and behavioral prediction

Open Questions

  1. Construct validity: Do LLM-judge scores of behavioral traits correlate with independent behavioral measures?

  2. Cross-model generalization: If Judge A scores Model B’s behavior, do those scores predict how Model C will score Model B?

  3. Temporal stability: How consistent are LLM-judge scores across prompt variations, model versions, and time?

  4. Expert alignment: When do we trust LLM judges vs. domain experts for behavioral evaluation?

  5. Training contamination: Are LLM judges evaluating actual behavior or recognizing training patterns about how to describe behaviors?

  6. Calibration: Can we create standardized calibration sets that ground behavioral scoring across different judges and contexts?

Conclusion

LLM-as-judge offers scalable behavioral evaluation but faces systematic reliability challenges: position bias, verbosity bias, classification instability, poor expert alignment, and single-shot unreliability.

These problems are magnified for psychological construct measurement, where ambiguity, context-dependence, and multidimensionality demand more sophisticated approaches.

I/O psychology’s SJT literature suggests categorical classification is more reliable than continuous scoring for heterogeneous behavioral measures. Recent work on consistency metrics (CoRA, STED, Grade Score) provides tools for measuring and improving judge reliability.

Critical gaps remain: lack of construct-level reasoning, no behavioral validation pipeline, binary thinking about traits, lack of temporal consistency, and judge calibration problems.

Future work should:

  • Integrate established psychological theory rather than inventing new rubrics
  • Validate judge scores against behavioral outcomes, not just face validity
  • Use categorical classification with clear behavioral indicators
  • Report consistency and reliability metrics
  • Acknowledge that scoring quality ≠ behavioral prediction

The methodology is useful but requires substantial improvement before reliable application to machine psychology research.

See also: Sycophancy Measurement, HEXACO Profiling, Sigmund, Psych (What We Learned)

References

This review synthesizes literature from multiple domains:

LLM-as-Judge Methodology:

Reliability Problems:

Consistency Improvements:

LLM Psychometrics:

I/O Psychology - SJTs:

Categorical vs. Continuous Scoring:

Graph View

Backlinks