Methods corner
Short methods notes and research-history pieces: the difference between odds and likelihood, reach and impact, participation as a design property, the Hawthorne effect.
Generated by scripts/build_companions.py from archive/; edit the source, not this file.
Participation is a design property, not a label
| *From [The Research Rundown | June 2025](/SignalStack/archive/2025-06-26-the-research-rundown-june-2025-edition.html), 2025-06-26.* |
“Participatory research” covers a very wide range of arrangements.
In one project, participants may help refine a questionnaire. In another, community organisations may choose the research question. Elsewhere, residents may collect data while researchers retain control over analysis, publication and funding. All three can be described as participatory.
That makes the label surprisingly uninformative unless we ask where authority actually sits.
For any participatory project, I would want to know:
- Who chose the research question?
- Who decided what counted as evidence?
- Who controlled the budget?
- Who interpreted the findings?
- Who could disagree with the researchers?
- Who owns the data?
- Who decides where and how the work is published?
- What happens after the project ends?
Participation can improve research. It can reveal categories and causal stories an external team would miss. It can also impose substantial unpaid work on participants or reproduce local hierarchies.
The method is therefore not virtuous by definition.
A useful discussion of the distinctions between action research, participatory action research and community-based participatory action research:
And the Tamil Nadu paper above provides a current Indian case:
https://discovery.ucl.ac.uk/id/eprint/10212113/
The Hawthorne Effect
| *From [The Research Rundown | June 2024](/SignalStack/archive/2024-06-05-the-research-rundown-june-edition.html), 2024-06-05.* |
The Hawthorne Effect is one of those research concepts everyone learns and few people revisit.
The label comes from studies conducted at Western Electric’s Hawthorne Works near Chicago in the 1920s and early 1930s. The popular account says worker productivity rose whenever researchers changed working conditions because workers knew they were being observed.
The history is messier.
Steven Levitt and John List recovered and reanalysed data from the original illumination experiments. They found that the familiar claim that productivity rose after virtually every lighting change was not supported by the original data. They did find weaker evidence consistent with observer effects.
The useful lesson survives. Measurement can alter behaviour. The effect should be treated as a hypothesis about a study setting rather than a universal explanation applied whenever participants know researchers are present.
Levitt, Steven D. and John A. List. 2009. “Was there Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments.” NBER Working Paper 15016. Published subsequently in American Economic Journal: Applied Economics 3(1), 2011.
https://www.nber.org/papers/w15016
Odds and likelihood are different things
| *From [The Research Rundown | June 2024](/SignalStack/archive/2024-06-05-the-research-rundown-june-edition.html), 2024-06-05.* |
Odds compare the probability of an event occurring with the probability of it not occurring.
If the probability of an event is pp, its odds are:
p/(1−p)p/(1-p)
Likelihood is a function used in statistical estimation. It asks how compatible observed data are with different parameter values under a specified model.
They sound similar. They do different jobs.
Reach and impact are also different things
| *From [The Research Rundown | June 2024](/SignalStack/archive/2024-06-05-the-research-rundown-june-edition.html), 2024-06-05.* |
Reach describes exposure.
Impact concerns change attributable to an intervention.
A campaign may reach 100,000 people. That number tells us nothing by itself about whether behaviour, knowledge, income or health changed because of the campaign.
Reach is an output measure. Impact requires a counterfactual claim.
This distinction would improve a surprising number of monitoring dashboards.