Measurement & Survey Methods Brief — 4 September 2026
2026-09-07. Published at https://varna.substack.com/p/measurement-and-survey-methods-brief. This file is a generated copy; the Substack post is the original.
1. Meyer et al. — The Anatomy and Evolution of Survey Error
NBER Working Paper 35680, August 2026. This is my top read this week. Meyer and colleagues develop an empirical Total Survey Error framework that puts coverage error, unit nonresponse, item nonresponse and measurement error onto a common scale of bias and absolute error, using linked administrative records for 18 CPS income and programme-receipt variables over more than two decades. The contribution is important because survey-error literatures usually examine these components separately, making it difficult to know where data-quality investments have the highest returns. (National Bureau of Economic Research)
Why it matters: The framework is directly transferable conceptually to large development surveys. In particular, it argues against assuming that improving response rates necessarily tackles the dominant source of error.
Worth reading? Essential. Especially useful for thinking about validation studies and survey-quality dashboards.
2. Sen & Lahiri — Improving Measurement Error and Representativeness in Nonprobability Surveys
Survey Methodology, released 29 June 2026. Most adjustment methods for nonprobability samples concentrate on selection bias while implicitly assuming that responses themselves are accurately measured. Sen and Lahiri address the two problems jointly, developing methods intended to correct both sampling/selection bias and measurement error. (Statistics Canada)
Why it matters: This is highly relevant to the growing use of opt-in online panels, social-media recruitment and hybrid data collection in development research. Weighting a badly measured outcome does not make it a good outcome.
Worth reading? Yes—high priority, particularly if using convenience, digital or programme-based samples.
3. Kosfeld et al. — Measuring Economic Preferences with Behavioral Experiments and Surveys Across the Globe
CESifo Working Paper 11631, 2026. The authors replicate the behavioural validation underlying Global Preference Survey-type measures with nearly 2,000 participants across China, Colombia, Iran, Kenya and the US. Survey measures often correlate with incentivised behaviour, but relationships vary substantially. More importantly, qualitative self-assessments and behavioural measures appear to capture different dimensions of the same latent preference, rather than being interchangeable measures. (ifo Institut)
Why it matters: A useful warning about the language of “validated measures.” Correlation with a behavioural benchmark does not establish that two instruments measure precisely the same construct.
Worth reading? Essential for measurement-validity work, especially studies using psychological, behavioural or empowerment constructs.
4. DiGiuseppe & Flynn — Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
Public Opinion Quarterly 90(3), 2026, pp. 630–656. DOI: 10.1093/poq/nfag013. Instead of asking an LLM to assign absolute 0–10 scores to open-text answers, the method asks models to compare pairs of responses and then estimates latent scores using a Bayesian Bradley–Terry model. Pairwise judgements were more consistent than zero-shot numeric ratings and corresponded reasonably well with human judgements. (PubMed)
What is genuinely useful here: Pairwise comparison reduces the anchoring and scale-calibration problem that makes direct LLM scoring particularly suspect.
Why it matters for development research: Potential applications include coding open-ended answers on empowerment, political efficacy, aspirations, perceived wellbeing and programme experiences.
Worth reading? Yes. Promising method, though I would still require human-coded validation samples before treating resulting scores as measurements.
5. Austin et al. — An Experimental Comparison of AI-Enabled Semi-Structured Interviews and Fixed Surveys
Public Opinion Quarterly, 2026. In a randomized experiment, AI-mediated interviews elicited longer and more specific explanations without materially increasing dropout or dissatisfaction. But there is a serious measurement warning: answering the AI probes changed subsequent responses, producing greater polarization on later attitude items. (SciLove)
Why it matters: AI interviewing may generate richer qualitative material while simultaneously creating instrument-induced treatment effects. The interview itself can change the construct subsequently being measured.
Worth reading? Definitely. It is one of the more important cautions emerging from the rush toward conversational surveys.
6. Wildner, Bosch & Meinfelder — Calibrating Nonresponse Bias: A Cautionary Tale
Journal of Official Statistics 42(2), first published 13 May 2026. DOI: 10.1177/0282423X261425800. The paper challenges the common intuition that calibrating survey weights to known population margins will necessarily reduce bias from nonresponse. Calibration depends critically on whether auxiliary variables actually capture the missingness mechanism and its relationship with outcomes. (Sage Journals)
Why it matters: In applied work, post-stratification or raking is too often described simply as having “corrected” nonresponse. That claim is generally stronger than the method warrants.
Worth reading? High priority for anyone designing weighting protocols.
7. Allorant & Smith — Algorithm-Assisted Inference and the Future of Official Statistics
Journal of Official Statistics, first published 14 May 2026. DOI: 10.1177/0282423X261443590. The paper places machine-learning-assisted estimation within the longer tradition of model-assisted survey inference. Its most useful contribution is a concrete quality framework for AI-assisted statistics: document training data, ensure algorithmic transparency, validate against established procedures, characterize uncertainty and preserve reproducibility. The authors also advocate continuous monitoring of calibration and model drift. (Sage Journals)
Why it matters: This is a better conceptual position than treating AI predictions as another clean administrative covariate. Prediction introduces its own error-generating process.
Worth reading? Yes, particularly for survey–administrative–remote-sensing integration and small-area estimation.
8. Heckman et al. — Measuring the Growth of Skills
NBER Working Paper 34737 / AEA Papers and Proceedings 116 (2026): 278–283. Heckman and co-authors question a foundational psychometric assumption: that skill at different developmental stages can always be represented on a common continuous scale. Their empirical results reject common-scale assumptions for language and cognitive skills and propose a measurement approach that permits qualitatively new skills to emerge over time. (National Bureau of Economic Research)
Why it matters: Development studies routinely construct longitudinal indices of cognition, learning, agency and child development as though the latent construct has unchanged meaning across ages or survey rounds. That assumption deserves much more testing.
Worth reading? Yes—conceptually important, particularly for longitudinal measurement.
9. Ambekar et al. — National Survey on Magnitude of Substance Use in India: Survey Methodology
Indian Journal of Psychological Medicine, online 18 August 2026. DOI: 10.1177/02537176261471784. This is the strongest India-specific methods publication I found this week. It documents the design and implementation of India’s national survey, covering 5,808 PSUs and nearly 471,000 analysed individual records. Selection probabilities were explicitly reconstructed across district, PSU, segmentation, census-block and household stages; weights were subsequently adjusted for response rates. The national response rate was 89.2%, with considerable state variation. (Sage Journals)
Why it matters: The methodological interest is not the substantive topic but the practical documentation of a very large, sensitive-outcome household survey in India, including rare-population estimation alongside conventional household sampling.
Worth reading? Yes, selectively. Particularly useful as an Indian field-design case study rather than as a methodological innovation.
10. UN Statistics Division — new Handbook of Surveys on Households and Individuals
The UN Statistical Commission adopted the new household-survey handbook in March 2026, replacing guidance that had not received a comparable major update for more than four decades. The chapter on Questionnaire Design, Translation and Instrumentation explicitly treats translation, mode, digital instrumentation and questionnaire testing as parts of measurement quality rather than downstream implementation issues. (UNSD)
Why it matters for South Asia: Multilingual survey translation remains one of the least adequately documented sources of measurement non-equivalence in Indian research. Treating translation as instrument design—not clerical translation—is overdue.
Worth reading? Yes as reference guidance, rather than as a research paper.
UN chapter preview and materials
My shortlist
If you read only four, I would choose Meyer et al. on Total Survey Error, Kosfeld et al. on behavioural validation, Austin et al. on AI interviews changing subsequent responses, and Sen & Lahiri on jointly addressing measurement and selection error.
For our kind of development research, the larger lesson is important: “validated,” “representative,” and “weighted” are properties that need much more qualified use than they generally receive. Several of these papers converge on precisely that point.