View source for Comparison of outcomes and effects (analysis)

'''The comparison of outcomes and effects''' is a family of [[Analysis method|analysis methods]] that answer [[Research question|research questions]] as to whether an [[educational product]] or its [[Learning objectives and outcomes|outcomes]] are comparatively ''better than'' alternatives that are considered in the analysis. This type of analysis helps people make decisions about products by providing [[evidence]] that they work better than others at [[Learning objectives and outcomes|achieving specified goals]].

== Definition==
The family of [[Analysis method|analysis methods]] that '''compare outcomes and effects''' take data from one product's [[Learning objectives and outcomes|outcomes]] and compare it to another product's outcomes, usually with [[data]] that are [[Instrument|measured]] in the same way. 

Evaluators [[Interpretation of data and evaluation results|interpret results]] from comparison [[Analysis method|analysis methods]] to answer [[Research question|research questions]] about whether a product, its outcomes, or its effects are '''better than or worse than''' an alternative that it is being compared to.

==Additional Information==
The goal of '''comparison of outcomes and effects''' is to identify differences between the ''outcomes'' of two or more educational products. This method can [[Evidence|provide evidence]] for whether products performed either ''similarly or differently'' (i.e., if one was ''better'' than the other), or it can answer ''how and why'' each product produced either the same or different outcomes. Both [[Qualitative analysis|qualitative]] and [[Quantitative analysis|quantitative]] methods can be used to investigate this family of questions, depending on the desired output.

As with most [[Analysis method|analysis methods]], the quality of the results from the method will depend upon (1) the quality of the data that are collected from the instruments that are used and (2) the skill and consistency of the researcher in applying the method and procedure.

'''Common comparison of outcomes and effects methods'''

'''[[Quantitative analysis|Quantitative]]'''

* '''Experiments and quasi-experiments.''' A structured approach toward comparison of two or more items in which as many factors and environmental contexts are ''held constant'' and that participants are ''randomly selected and assigned to a condition,'' except for the differences between compared items themselves''.'' In the case of educational evaluation, items are typically educational products, as measured by one or more [[Learning objectives and outcomes|outcome or learning objective variables]]. 
** By controlling for and holding constant outside factors, this method allows a researcher to interpret that the observed differences in one educational product compared to another are the result of the product itself, and not caused by other psychological or environmental factors. A '''quasi-experiment''' is one in which all factors cannot be controlled, or complete random assignment of participants into conditions is possible, so a best effort is made to ensure as many conditions as possible are considered. 
** Experiments in education typically investigate differences between a '''treatment condition''', which is the educational product under evaluation, and one or more '''control conditions''', which are educational approaches or products that are alternatives to the treatment condition. Experiments typically use ANOVA, Multiple Regression, and HLM methods to identify and interpret differences between items (see below).
* '''Longitudinal studies''' investigate how outcomes change ''over time''. Participants' outcomes are measured at multiple times over the course of the study and differences between each time point are tracked. With this approach, researchers identify how [[evidence]] of [[Learning objectives and outcomes|learning outcome achievement]] increases or decreases over time
* '''Comparison of means tests.''' T-tests, ANOVA, and MANOVA statistical methods are commonly used in education to compare the averages (means) of two or more measurement variables in different groups (such as a treatment and control condition) to see if they are statistically different.
* '''Multiple regression''', which predicts outcomes (i.e., dependent variable) based on one or more independent variables. Each independent variable receives a ''coefficient value'', which when standardized, can be compared to see which has a stronger impact at predicting the outcome. A more advanced method of regression used in educational research is called '''Hierarchical Linear Modeling (HLM) or multi-level modeling,''' which additionally accounts for nested or layered levels of data (like students within classrooms within schools) 

[[Qualitative analysis|'''Qualitative''']]

* '''Constant comparison.''' An approach for analyzing qualitative data to continually be contrasting and comparing differences between data points or analysis units. These comparisons, similarities, and differences between [[Qualitative analysis|qualitative]] data can be used to [[Evidence|identify evidence]] of [[Learning objectives and outcomes|learning outcome achievement]] in [[data]] that can be later measured [[Quantitative analysis|quantitatively]], or they can be linked to processes and activities that learners do via a process analysis.
* '''Process analysis.''' This approach identifies and examines the ''processes'' that people do in an activity. Process analyses identify the specific verbs and processes that people actually do, and can create hypotheses as to how and why these processes affect learning outcomes. Thus, comparison of two different sets of verbs that are identified in a process analysis might yield two different hypotheses about how these processes influenced learners' achievement of the outcome (i.e., the [[Learning objectives and outcomes|learning objective]]) 
* '''Qualitative comparative analysis (QCA).''' A method for investigating qualitative data systematically to find causal links between [[Interactions and activities|what people do]] in a learning experience, [[Learning environment|factors in the environment]], and [[Learning objectives and outcomes|achievement of learning objectives]]. The goal of QCA is to use data from [[implementation]] to identify the actions that lead to achieving a learning objective and to theorize how and why these actions lead to achievement. 
** Summary of Qualitative Comparative Analysis: Better Evaluation (n.d.). Qualitative comparative analysis. Retrieved from https://www.betterevaluation.org/en/evaluation-options/qualitative_comparative_analysis. 3 Jul. 2022. 

'''''Note''': It is beyond the scope of this knowledgebase to expand on each of these methods. It is recommended that researchers and evaluators seek additional training, web resources, or courses on individual methods they would like to use.''

==Tips and Tricks==

* Remember, every method is only good for answering specific [[Research question|research questions]]. You may have the need to compare items ''within'' your product, such as how well two features compared to each other in how they contributed to [[Learning objectives and outcomes|learning objective achievement]]. Or, you may wish to compare the effects of your whole product to other products (such as through an experiment). Use comparison methods only if they answer the questions you seek!
* An evaluation of a learning product will likely need to ask whether the product ''worked''. Although a comparison does not necessarily need to be made with another alternative product or approach, it could be useful to make claims about a product working ''better'' or that it is a good investment. To make these claims about your product, you will need to ask [[Research question|research questions]] about how the product compares to others in how learning objectives are achieved. 
* An experiment or quasi-experiment is the most common approach for comparing the outcomes of two or more products. Experiments measure outcomes in a systematic way and use statistics to identify whether there were likely differences observed between each of the products being measured. Consider including an experimental approach for comparing your product to other products, especially if you want to convince other people that your approach does better at achieving the goals. 

==Related Concepts ==

* [[Evaluation and assessment]]
* [[Analysis method]]
* [[Research question]]
* [[Learning objectives and outcomes]]
* [[Instrument]]

==Examples ==

None yet - check back soon!

==External Resources==

* Summary of Qualitative Comparative Analysis: Better Evaluation (n.d.). Qualitative comparative analysis. Retrieved from https://www.betterevaluation.org/en/evaluation-options/qualitative_comparative_analysis. 3 Jul. 2022.