I run experiments for a living — and I treat the practice itself as a research subject. My work sits at the intersection of experimentation methodology, causal inference, behavioral economics, and applied AI, grounded in project-specific evidence including 100+ in-house NRG experiments in 2025.
Free weekly field notes
How organizations should measure the true value of testing programs: win rate as a vanity metric, winner’s-curse correction, sequential testing tradeoffs, and long-term holdouts as program-level audits.
Geo-incrementality, matched-market designs, holdout experiments, and attribution beyond last-click — separating demand that marketing created from demand it merely captured.
When loss aversion, choice architecture, anchoring, and social proof provide useful hypotheses—and what production evidence is needed before treating a mechanism as relevant.
AI-assisted hypothesis generation, analysis acceleration, and personalization systems — and the governance layer they need: bias detection, guardrails, and transparency standards.
Practitioner research drawn from production experimentation. Papers publish first as citable write-ups on this site and Better Decisions, with PDF versions prepared for scholarly indexing.
Experimentation programs report win rates as their headline KPI, but win rate is trivially gamed by testing safe, small changes. This paper proposes a decision-quality framework — revenue per experiment, save rate, and learning velocity — drawn from operating a 100+ test/year enterprise program.
Experiment knowledge can disappear when ownership changes or readouts are difficult to find. This paper examines repositories as institutional-memory infrastructure and the incentives that determine whether teams contribute to them.
Winner’s curse, novelty decay, and interaction effects can make summed per-test impact differ from a program-level holdout estimate. This planned paper will examine that gap, the assumptions behind each method, and the conditions under which either estimate is decision-useful.
Original illustration · Human support, not surveillanceA four-person hackathon team explored passive facial analysis for student support. Further research exposed an unsupported inference, so I rejected the mechanism and reframed the surviving problem around explicit student input, consent, and human-led navigation.
Team research concept · Not built, deployed, or clinically validated
Read the research case studyPapers and technical writing I'm working through, with notes on how each one holds up against production data.
An example of productized program-level measurement. Holdouts can complement individual readouts by estimating cumulative program effects—a question finance leaders may ask before renewing investment.
A core reference for experimentation at scale. The chapters on Twyman’s Law and sample-ratio mismatch shaped a QA habit: investigate instrumentation and data quality before treating a surprising result as a breakthrough.
A foundational model of decision under risk. In applied CRO, loss framing is a mechanism worth testing—not a universal winner—and production context determines whether the effect appears or matters.
The experimentation operating system this research is built on — PRISM, the idea-to-investment pipeline, and honest impact measurement.
See the method →EvidenceThe production programs behind the data: $30M+ recorded in NRG internal program reporting at NRG Energy, and geo-measurement at Silicon Valley Bank.
See the work →Practitioner EvidencePortfolio syntheses, repeated patterns, and copy-ready evidence packs for CRO teams deciding what to test next.
Search the evidence →Open QuestionsProduct and research concepts I'm exploring — including AI-driven experimentation intelligence built on contributed test data.
See the ideas →Know what to test, when to trust the result, and what to do next. Practical decision guides for analysts, growth teams, and founders.
Opens Substack to confirm · Free · Unsubscribe anytime
Practical guides on experimentation, analytics, commercial judgment, and using AI without outsourcing the decision.
Open Substack (opens in new tab)