Feasibility of evaluation
From The Learning Engineer's Knowledgebase
The feasibility of an evaluation is the degree to which an evaluation team can realistically complete all of the planned evaluation tasks.
Definition
When an evaluation plan is feasible, the evaluation tasks are realistically able to be completed given the skills, time, and money available.
Some evaluation tasks require more human resources or money than others, and plans should be made accordingly that keep the evaluation plan as feasible as possible for the people having to conduct evaluations and deliver assessments.
Additional Information
When developing an evaluation plan, designers should consider whether they can successfully implement the evaluation tasks that they have planned. This includes ensuring that all of the planned assessments and tests are adequately administered and that any teachers or people implementing any evaluation instruments are properly trained.
To ensure feasibility of an evaluation plan, an important suggestion is to ensure that the evaluation plan does not overpromise. It can be easy to plan a bunch of assessments and evaluation actions only to realize later that it is a substantial amount of work and that the evaluation team is far underequipped to handle the tasks. Each evaluation task requires a certain amount of time and other resources to complete. Designers should be sure that the evaluation plan is feasible and can be carried out based on the amount of available staff time and financial resources. Otherwise, the evaluation team risks losing data or having incomplete or incoherent data on learners, and thus making it difficult to make any conclusions about whether the educational product worked as expected.
Key aspects of ensuring that an evaluation remains feasible:
There are some things that can be considered by evaluation designers who wish to review whether their evaluation plan is feasible:
- Determine staff capacity and budget. Evaluations all cost something - either staff time or skill in collecting, analyzing, interpreting, or disseminating the data, or financial costs associated with collecting data, maintaining data and assessment systems, or interpreting results.
- Align with participant availability. Participants need to have enough time and capacity to complete any assessments and tasks that will be evaluated (i.e., not requiring an unreasonable amount of testing or assessment during the learning activity)
- Formative assessment upkeep. Ensure that enough staff time and resources to conduct any formative assessments and monitoring, which can take a substantial amount of effort to conduct alongside the actual implementation and instruction tasks of the product.
- Data collection system upkeep. Consider the maintenance and upkeep costs for digital data collection systems, especially as systems sometimes fail or stop collecting data.
- Privacy and security. Consider resource requirements that data of participants remain securely stored and privacy is maintained
- Adequate skills. Ensure that the evaluation team maintains members with adequate skills for conducting analyses and interpretation of results, particularly for any specialized qualitative or quantitative methods. There also needs to be an adequate level of skill for grading, scoring, and interpreting the assessments and work products of the participants.
- Dissemination costs. Ensure that adequate budget and staff time are dedicated to generating reports about evaluations and disseminating information.
Tips and Tricks
- Make sure that you do not overpromise in your evaluation plan. Keep the evaluation tasks feasible by identifying how much time and financial resources that you and your team have available and planning accordingly.
- Consider how much time each data collection and analysis task will take, especially with methods that require in-depth coding or analysis from a human scorer. An evaluation plan that promises more work than you have the capacity to complete can be really harmful for the future of the product.
- Keep evaluation plans simple to begin with, and then build additional tasks if you have the capacity to do so. A good evaluation is one that you can complete realistically. If you need to complete more intensive evaluation tasks, you will need to consider hiring additional staff or finding technologies to support these tasks.
Related Concepts
Examples
None yet - check back soon!
External Resources
None yet - check back soon!