Understanding researcher and participant bias and how to overcome it in user research

Photo by tito pixel on Unsplash

This article explores how The Hawthorne Effect and Observer Bias impact user research and how to mitigate it.

The Hawthorne Effect

The Hawthorne Effect refers to the phenomenon when people change their behavior when they know they’re being observed.

This effect was discovered after a series of studies were conducted to determine how lighting impacted workers’ productivity at the Hawthorne Works factory. Researchers found productivity improvements when any change in lighting occurred, including reverting to the original conditions. They concluded the act of being observed made the workers more productive, rather than the changes in lighting conditions.

User research implications

In the context of user research, the Hawthorne Effect can be particularly problematic. For example, during usability testing, participants might try harder to perform tasks, be less likely to give up when facing difficulties, or not behave as they normally would if they were not observed.

Observer Bias

In addition to The Hawthorne Effect, Observer Bias can further muddy research results. It occurs when researchers unconsciously project their expectations or judgments onto the participants or the data.

Observer bias usually manifests in several ways:

1. Selective Observation: Focusing only on specific behaviors or outcomes based on the observer’s expectations or hypotheses.
2. Interpretation Bias: Interpreting ambiguous actions or statements that align with the observer’s expectations or theories.
3. Confirmation Bias: Giving undue weight to data points that confirm the observer’s pre-existing beliefs while disregarding data that contradicts them.
4. Influence on Participants: The presence or behavior of the observer can subtly influence the participants’ behavior, skewing results.

If researchers only notice behaviors that confirm their hypotheses or interpret data in ways that align with their expectations, the insights derived from the research may be skewed.

Techniques to Mitigate The Hawthorne Effect and Observer Bias:

1. Blinding: In some research settings, you can blind the observers to the testing conditions so their expectations can’t influence their observations.
2. Multiple Observers: Having multiple people independently observe and record data can help to offset individual biases.
3. Objective Metrics: Review quantitative metrics alongside qualitative observations to cross-verify results.
4. Think-Aloud Protocols: Asking participants to think aloud can offset the Hawthorne Effect by providing insights into their thought processes, which remain consistent even if their behavior changes due to observation.
5. Naturalistic Observation: Observing people in more natural settings or over more extended periods can sometimes reduce the Hawthorne Effect.
6. Transparency and Debriefing: Being transparent with participants about the study’s aims and debriefing them afterward can help control observer bias and the Hawthorne Effect.