Evaluation and assessment

Evaluation and assessment is the final step of the ADDIE model of design. In this phase, educational products are evaluated for whether they met the intended goals and whether learners benefited from the experience.

Definition

Evaluation is the process of analyzing and making judgement on whether an educational product achieved its goals. An evaluation, as a noun, is the activity of evaluating, or the final results and report from an evaluation.

Assessment is typically the process of evaluating people for competence, learning, or change. Assessment, as a noun, can also mean the instruments used to assess a person.

Although there are slight differences between the terms of evaluation and assessment, they are often used synonymously. The word evaluation tends to be used more to describe, compare, and judge the quality of educational products, where the word assessment is more frequently used when concerning an individual and measuring their learning or other important changes (such as participation or affect).

Additional Information

Evaluation is the final step of the ADDIE model and seeks to answer questions about whether the product worked as intended and if people who participated achieved the learning objectives of the product. Using an evaluation model, evaluators will develop an evaluation plan and conduct the evaluation plan when the product is implemented. After an evaluation is completed, the evaluator will write reports and share the information with anyone who is interested.

Scientifically, the best evaluations are pre-planned and are subsequently completed according to the plan. Any deviations from the plan are documented and shared as a part of the results. This is to increase the reliability and validity of the findings, to improve the ability for others to reproduce and follow the steps that the evaluator took to come up with their findings, and ultimately improve the trust people have when they read the results and generalizability of the findings to other contexts.

Purposes of evaluation and assessment

Two primary types of research questions are answered during the process of evaluation and assessment: (1) answering research questions about learning objective achievement by learners (who are the primary focus of an educational product) and (2) whether the product was used as intended, by both the participants and the people who implement the product. Evaluations also sometimes ask questions about the quality and value of an educational product, such as whether it is a "good product" and if it was useful or better compared to alternative products.

Evaluators take a systemic approach to investigating what, how, and why things happened as a result of using an educational product. They use an evaluation model to guide them as the do the step-by-step process of developing their evaluation plan and later following through on the plan. Evaluation models are step-by-step processes that are commonly used and are based on best practices in the field.

The product of the evaluation process is a collection of insights and findings about how and why people used the educational product and how and why they demonstrated competency or learning from the experience. Evidence about the use of the product and its effects is used to describe the value of the product, as well as to improve the product in its future versions. Additionally, evaluation and assessment is often used to providing a grade, certification, or other credential to a learner to signify their competence after participating with the product.

Common process and steps of developing an evaluation plan

Evaluation typically occurs over a series of general steps. Of course, each evaluation plan is different based on the research questions being asked and the context in which the product was implemented.

Define the research questions. The beginning of any evaluation involves defining the questions that will be asked and why they matter. Evaluators must define specific questions whose answers will provide useful information toward improving the product and demonstrating that the product works.
Identify the audience. Each audience is different and may have varied needs. To interpret the evaluation validly, reliably, and fairly, the evaluators should consider the unique composition of any audiences that will use the product, as well as the contexts in which the product will be implemented.
Determine analysis methods and instruments. To answer the research questions, the evaluators must define both the analysis methods and the instruments that will to be used to collect and analyze data.
Conduct the evaluation. After the planning is complete, the evaluation is conducted while the educational product is being implemented. Learners are assessed using both summative and formative approaches and data are collected on how the product and its individual design features are used.
Conduct the analysis. Using the data that were collected during implementation, analyses are completed based on the evaluation plan and results are generated after implementation. However, with formative assessment, the analyses and interpretation are completed while the product is being implemented so that the experience can be improved for learners as they participate.
Interpret and review the results. From the analyses and the results they generate, the evaluators will generate insights about what happened, how people learned, and how people used the product as intended. The evaluators will interpret the findings using standard industry practices and make claims about what the evidence means.
Report and share findings. The results and findings from evaluations are typically shared with parties interested in knowing how a product worked, including funders, potential implementers, educators, and purchasers. Results from evaluations are most often shared in the form of reports, descriptive studies, and research papers that describe the product, the research questions, the data, the analysis approaches, and the results.

When to evaluate

Conducting an evaluation occurs when an educational product is being implemented. From the first moment of its use, data are being collected on the participants' learning and how they are changing as a result of their participation. Additionally, data are often collected on how people use the product throughout implementation. However, evaluations and analyses are conducted throughout the implementation of an educational product.

Formative assessment is conducted actively while the product is being implemented. This real-time evaluation approach is used to identify how and why participants are learning with the system and what factors are influencing their learning, including their level of participation with the product's various design features. With formative assessment, the product, learning environment, and overall experience can be improved while it is being used, based on the learner's performance and perceptions.

Summative assessment is conducted at the end of the product's use. Data that are collected at the end of the product's use provides an "end result" of a participant's competence and learning. Analyses that are conducted at the end of the product's use can investigate, as a whole, how the product was used and what effects it had on learning. At the end of a product's use, evaluators make claims about how well the product worked, based on the analyses and evidence shown in the results. Evaluations at the conclusion are typically written as a report and shared with interested parties.

Types of evaluations and common research questions

There are common families of research questions that appear frequently in evaluation studies. Each responds to a particular research need for evaluators to answer about the quality of the product and how or why the participants learned from it.

Tips and Tricks

Every educational product should be evaluated for whether it was used as intended and whether people learned from it. This is completed by designing an evaluation plan that asks specific research questions about how and why the product worked. Consider following the step-by-step guidance on evaluation plans on how to begin designing an evaluation for a product.
Evaluation can occur both while a product is being used (via formative assessment) and at the end of the product's use (via summative assessment). Think about how you may use formative methods to evaluate how things are going for participants while they are active in the learning experience, as well as how teachers or facilitators may be informed via formative methods on how to improve the experience for participants.
With summative approaches, consider what kinds of claims you would like to make about your product if you were to share it with other people. Such claims include:
- This is a good product
- People used the product as expected
- This product worked as intended
- People learned from this product
- This product is better than alternatives
While using an evaluation model to design the evaluation plan, determine what kinds of analyses you need to conduct and what kinds of data you need to collect using instruments to make the kinds of claims that you determined above. The evaluation model can help you think about and identify the analyses and data that you need to make strong claims.
Evaluation is a challenging task. Many evaluations are conducted by a team. Freely share your plans and ideas with others to get feedback on how to improve your plan so that you can make the strongest claims. Consult with research methodologists to consider what kinds of analyses and data are appropriate for the kinds of research questions that you are asking.

Evaluation and assessment

From The Learning Engineer's Knowledgebase

(Redirected from Evaluation)

Contents