Session 7 – How to Critically Read the Research on Change Efforts [S7-24-25]

This presentation will focus on ten questions that should be considered in assessing the methodological soundness of contemporary research on change efforts. These questions are critical in determining how accurately study results are being interpreted and applied to legal and professional resolutions and policy. I will accompany these questions and the concerns they raise by examples taken directly from published studies. The ten questions to be examined are:

1. Do the researchers attribute causation to correlational statistics?
2. How prejudicial is the researcher’s appeal for recruiting participants?
3. Is there bias evident in the nature of the sample?
4. Do the researcher’s conclusions and policy recommendations generalize beyond what their sample allows?
5. Are there indications participants could be motivated to respond in a manner desired by the researchers?
6. What are the researcher’s likely beliefs about change efforts?
7. Does the source of funding or data create obligatory research outcomes?
8. Do the researchers account for potentially confounding factors?
9. Does the research use reliable and valid measures?
10. Do the researchers provide not just indicators of statistical significance but also provide effect sizes?

In the course of examining these questions, I will discuss and explain several relevant concepts to assist the non-researcher in understanding this body of research. These concepts include

1. Association versus causation.
2. Correlational, longitudinal, quasi-experimental and randomized controlled trial (RCT) research designs.
3. Generalizability and external validity.
4. Snowball and network sampling techniques.
5. “Me-search”.
6. Researcher neutrality versus researcher bias.
7. “Performative” research versus accuracy seeking research.
8. Reliability and validity.
9. Statistical significance versus statistical importance.
10. Statistical significance versus effect size.
11. Statistical significance versus practical significance. 
12. High-low fallacy. 

In many respects, my IFTCC 2023 presentation (“The science of SOCE is NOT settled”) is complementary to this year’s presentation and the two can be beneficially viewed sequentially. In 2023 I provided evidence that the change effort literature is (1) hampered by serious methodological weaknesses, leaving it (2) at best incomplete and at worst seriously inaccurate, but (3) guarded by an ideological monoculture and (4) an interdependent relationship between SOCE researchers and the organizations that fund them. This year I will expand on some of these themes with a particular emphasis on providing multiple examples taken directly from published studies on change efforts. Whether you are a researcher, therapist, pastor, or lay person, my intent is for you to leave this presentation with a means to more deeply and skeptically reflect on the change effort social science literature. This year I will again argue that no topic of science should be off limits for perspectives that challenge dominant narratives, including and perhaps especially the topic of change efforts.