Comparison of outcomes and effects (analysis)

The comparison of outcomes and effects is a family of analysis methods that answer research questions as to whether an educational product or its outcomes are comparatively better than alternatives that are considered in the analysis. This type of analysis helps people make decisions about products by providing evidence that they work better than others at achieving specified goals.

Definition

The family of analysis methods that compare outcomes and effects take data from one product's outcomes and compare it to another product's outcomes, usually with data that are measured in the same way.

Evaluators interpret results from comparison analysis methods to answer research questions about whether a product, its outcomes, or its effects are better than or worse than an alternative that it is being compared to.

Additional Information

The goal of comparison of outcomes and effects is to identify differences between the outcomes of two or more educational products. This method can provide evidence for whether products performed either similarly or differently (i.e., if one was better than the other), or it can answer how and why each product produced either the same or different outcomes. Both qualitative and quantitative methods can be used to investigate this family of questions, depending on the desired output.

As with most analysis methods, the quality of the results from the method will depend upon (1) the quality of the data that are collected from the instruments that are used and (2) the skill and consistency of the researcher in applying the method and procedure.

Common comparison of outcomes and effects methods

Quantitative

• Experiments and quasi-experiments. A structured approach toward comparison of two or more items in which as many factors and environmental contexts are held constant and that participants are randomly selected and assigned to a condition, except for the differences between compared items themselves. In the case of educational evaluation, items are typically educational products, as measured by one or more outcome or learning objective variables.
  • By controlling for and holding constant outside factors, this method allows a researcher to interpret that the observed differences in one educational product compared to another are the result of the product itself, and not caused by other psychological or environmental factors. A quasi-experiment is one in which all factors cannot be controlled, or complete random assignment of participants into conditions is possible, so a best effort is made to ensure as many conditions as possible are considered.
  • Experiments in education typically investigate differences between a treatment condition, which is the educational product under evaluation, and one or more control conditions, which are educational approaches or products that are alternatives to the treatment condition. Experiments typically use ANOVA, Multiple Regression, and HLM methods to identify and interpret differences between items (see below).
• Longitudinal studies investigate how outcomes change over time. Participants' outcomes are measured at multiple times over the course of the study and differences between each time point are tracked. With this approach, researchers identify how evidence of learning outcome achievement increases or decreases over time
• Comparison of means tests. T-tests, ANOVA, and MANOVA statistical methods are commonly used in education to compare the averages (means) of two or more measurement variables in different groups (such as a treatment and control condition) to see if they are statistically different.
• Multiple regression, which predicts outcomes (i.e., dependent variable) based on one or more independent variables. Each independent variable receives a coefficient value, which when standardized, can be compared to see which has a stronger impact at predicting the outcome. A more advanced method of regression used in educational research is called Hierarchical Linear Modeling (HLM) or multi-level modeling, which additionally accounts for nested or layered levels of data (like students within classrooms within schools)

Qualitative

• Constant comparison. An approach for analyzing qualitative data to continually be contrasting and comparing differences between data points or analysis units. These comparisons, similarities, and differences between qualitative data can be used to identify evidence of learning outcome achievement in data that can be later measured quantitatively, or they can be linked to processes and activities that learners do via a process analysis.
• Process analysis. This approach identifies and examines the processes that people do in an activity. Process analyses identify the specific verbs and processes that people actually do, and can create hypotheses as to how and why these processes affect learning outcomes. Thus, comparison of two different sets of verbs that are identified in a process analysis might yield two different hypotheses about how these processes influenced learners' achievement of the outcome (i.e., the learning objective)
• Qualitative comparative analysis (QCA). A method for investigating qualitative data systematically to find causal links between what people do in a learning experience, factors in the environment, and achievement of learning objectives. The goal of QCA is to use data from implementation to identify the actions that lead to achieving a learning objective and to theorize how and why these actions lead to achievement.
  • Summary of Qualitative Comparative Analysis: Better Evaluation (n.d.). Qualitative comparative analysis. Retrieved from https://www.betterevaluation.org/en/evaluation-options/qualitative_comparative_analysis. 3 Jul. 2022.

Note: It is beyond the scope of this knowledgebase to expand on each of these methods. It is recommended that researchers and evaluators seek additional training, web resources, or courses on individual methods they would like to use.

Tips and Tricks

• Remember, every method is only good for answering specific research questions. You may have the need to compare items within your product, such as how well two features compared to each other in how they contributed to learning objective achievement. Or, you may wish to compare the effects of your whole product to other products (such as through an experiment). Use comparison methods only if they answer the questions you seek!
• An evaluation of a learning product will likely need to ask whether the product worked. Although a comparison does not necessarily need to be made with another alternative product or approach, it could be useful to make claims about a product working better or that it is a good investment. To make these claims about your product, you will need to ask research questions about how the product compares to others in how learning objectives are achieved.
• An experiment or quasi-experiment is the most common approach for comparing the outcomes of two or more products. Experiments measure outcomes in a systematic way and use statistics to identify whether there were likely differences observed between each of the products being measured. Consider including an experimental approach for comparing your product to other products, especially if you want to convince other people that your approach does better at achieving the goals.

Related Concepts

• Evaluation and assessment
• Analysis method
• Research question
• Learning objectives and outcomes
• Instrument

Examples

None yet - check back soon!

External Resources

• Summary of Qualitative Comparative Analysis: Better Evaluation (n.d.). Qualitative comparative analysis. Retrieved from https://www.betterevaluation.org/en/evaluation-options/qualitative_comparative_analysis. 3 Jul. 2022.