Analysis method
From The Learning Engineer's Knowledgebase
An analysis method is a procedure used to examine and interpret data collected from instruments to make a claim or generate insights about the evidence that was identified during evaluation.
Definitions
An analysis method is the procedure used to interpret data and evidence to answer a research question. Evaluations of educational products seek to answer research questions about whether the product achieved its intended goals of educational achievement of its participants or whether the product was used as intended.
Additional Descriptions
Analysis methods are the tools that are used by the evaluator to look at evidence, interpret that evidence, and make claims about what, how, and why things occurred as a result of participants' use of the educational product. In other words, the evaluator will say whether or not learning happened, or whether or not the educational product was used as intended. These are important insights for understanding whether goals were met and how the designer may look at improving their product to better achieve their goals.
Each analysis method is used to answer a specific type of research question that is generated during the evaluation planning phase. Research questions are typically tied to the stated learning objectives of an educational product. Research questions also investigate how and why the educational product was used. In an evaluation plan, the research questions should be clearly defined, and for each research question, the corresponding analysis methods and instruments for data collection should also be defined.
Specifically, analysis methods are specific step-by-step procedures that are well documented and are typically agreed upon by the IDT field. An analysis method is therefore a systematic process of looking at and interpreting evidence so that the same results may be achieved by another evaluator at a later time. This ability to reproduce the same insights using the same data and analysis method is important to research, as it can improve the reliability of the results and counter biases and subjective interpretations of the evaluator.
In other words, we can rely on a systematic and documented analysis method to provide better results that we can trust in comparison to one that is not documented or based on the individual interpretations or interests of a single evaluator, who may be pressured to show positive or promising results. This reduces the biases inherent in research and improves the reliability of the insights generated from the analysis.
An analysis cannot be performed without data - there is otherwise nothing to analyze! Data that provide evidence about learning achievement and how the product is used are collected with evaluation instruments, which are embedded in the educational product design. As a result, analysis methods are always paired with instruments to collect data on how things happened within an educational product.
Analysis Methods by Research Question Category
Below are the 9 types of research question categories that are described on this Wiki. Within each are lists of common analysis methods that are used to answer questions of this type and analyze the data.
- Evaluating learning and competency outcomes
- Identifying factors and relationships that influence outcomes
- Identifying participant perceptions and satisfaction
- Identifying how people participate, interact, and learn
- Evaluating the quality and standards of educational products
- Comparing educational products and alternatives
- Identifying product and design feature usage
- Reporting on design processes and design specifications
- Analyzing cost and efficiency
Tips and Tricks
- When developing an evaluation plan, each research question should be well defined as to what will be evaluated and why. For each research question, the corresponding analysis methods that will be used to answer the question, as well as the method's corresponding instruments for collecting data should be clearly defined.
- Each analysis method serves to help evaluators interpret data and evidence in a way that can be used to answer specific types of research questions. Make sure that the type of analysis method that you choose matches the type of research questions that you are asking!
- You should also be attentive to what kinds of data that each analysis method uses. You will need to ensure that you design and implement appropriate instruments to collect this data while the product is being used.
Related To:
Examples
None yet - check again later!
Additional Resources
None yet - check again later!