2025-12-30 | Situated Assessment
Found academic grounding for the “runtime sampling” concept from two days ago.
The Literature
SAM2 (Situated Assessment Method) — Barsalou lab at University of Glasgow. Core insight: traditional psychometric instruments abstract people from situations, but most psychological constructs are inherently situated. Assessing them in context where they naturally occur produces better measurement.
Their research shows situated assessment explains 74-83% more behavioral variance than traditional instruments. The method: instead of generic “I am organized,” assess within specific situations like “At work when facing a deadline, I keep my materials organized.”
CAPE (Context-Aware Personality Evaluation) — Recent LLM work, but different approach. They focus on test-taking dynamics (how models respond to the assessment itself), not external conversational context.
The Gap (As I Saw It)
At this point, I thought nobody was doing true situated assessment for LLMs—embedding psychometric items within naturalistic external conversations, not just varying the assessment format itself.
This would turn out to be less of a gap than I thought. The deeper issue wasn’t how to assess personality, but whether personality assessment was the right frame at all.
What I Named
Renamed the project from “runtime-sampler” to “situated-sampler”—grounded in SAM2 literature.
The Hypothesis (As Stated)
Situated assessment (items presented within conversation context) produces more accurate/reliable psychometric measurement than vanilla (isolated) assessment.
If validated, this would enable real-time psychological profiling during live conversations.
Note: This direction was later revised. See 2026-01-01 for why we moved away from the psychometric abstraction entirely.