Documentation of evaluation plan
From The Learning Engineer's Knowledgebase
The documentation of an evaluation plan is the process of recording each of the elements and decisions of a planned evaluation of an educational product. Plan documentation is necessary so that the plan may be referred to by both the design team, newcomers to a project, and to people external to a project (such as funding agencies, purchasers, and people who may implement the product being evaluated). Evaluation plan documentation is part of an overall project's documentation of design.
Definition
Documentation of evaluation plans consist of making records of the decisions made to evaluate an educational product, the rationale behind these decisions, and the specific procedures that will be followed in carrying out the plan.
Additional Information
An evaluation plan should exist in a documented, recorded form so that the people who conduct the evaluation know what exactly to do and so that the evaluation team can rely on the documentation to describe the results. Evaluations that show effectiveness of a product in particular need well-designed and documented plans to demonstrate higher degrees of reliability, validity, and reproducibility by demonstrating that the plan was conducted using a systematic process that can largely be repeated, if necessary.
Common elements of an evaluation plan document
Similar to general design process documentation, an evaluation team should regularly record the work that they perform and the decisions that they make. Additionally, evaluators should record the up-front ideas and goals of the evaluation, the methods that are chosen, the specific procedures that will be followed. Accompanied with each of these items should be rationale for why each decision was made. Rationale can be tied to testing educational theory, or toward established evaluation principles and best practices.
Common elements of an evaluation plan that should be documented include:
- Information on the tasks done from the planning and idea generation processes, and what was generated during these. A record of work and tasks done for evaluation
- The learning and design objectives of the educational product
- The research questions that will be asked in the evaluation, usually tied to the learning and design objectives
- Theories of learning that are used with the product to predict how people will use it
- Models of evaluation
- The specific methods and instruments to be used for collecting and analyzing data.
- Specifying the procedures that need to be followed to carry out the plan, including how results should be interpreted
Methods for documenting the decisions, rationale, and procedures of an evaluation plan
Similar to general design documentation, a team should document their work in both (1) actual product documents (e.g., the written evaluation plan that will be shared) and (2) reflective logs or journaling on the process by the design team members. The basic premise behind documentation is for designers to record and archive their process of work, and to have a moment to reflect on what, how, and why they are doing what they are doing. Designers who document well find that the reflection process is very valuable in helping them think through the reasons why they do things. This becomes particularly valuable later on in the design process when needing to describe the rationale for designs to people who are making decisions about adopting, downloading, or buying an educational product.
Methods for documenting evaluation plan work:
- Documentation through project journals or notes. Much like a lab scientist will keep a lab notebook or lab journal. Doesn't have to be written, but written documents might be easier to share and analyze later. Project journals can be written on physical paper, on an electronic word document, or can also be an online web form with multiple fields or questions that can be saved into a spreadsheet. Many people also document by recording design sessions and team meetings, or by doing voice dictation where designers speak about their decisions instead of writing (a vocal log).
- Documentation through generating specific documents on design features. Each of the design features may have a series of notes and decisions as they were developed. These should be created and preserved for review later on.
- Documentation through daily reflective journaling and notetaking. Daily or semi-regularly journaling is a good habit and skill for a designer to develop. Journals can be prompted, in that the journal asks the writer similar questions every day, or the journaler can write openly (or record themselves speaking) about the things that they encountered each day and how they worked to complete each task. Tasks of course do not need to be completed each day, but a daily journal can record some of the progress made on each task and what factors influenced the decisions of the designer. See the documentation of design page for sample journal prompts for completing a daily journal.
- Documentation through project documents and deliverables. The actual documents and elements of the product, including all of the documents and deliverables, all serve as evidence of decisions and design choices that were made. Although the end results do not explain how and why, the product's end results are another part of documentation of the product's work.
Tips and Tricks
- When generating an evaluation plan, it is important to document your decisions, rationale, and general reflections on how and why you made the decisions you did. Evaluation plans should be well documented so that they can be passed onto people outside of the design team, and evidence of your plan can be shown to others to improve the reliability and reproducibility of the methods used.
- Additionally, it could be helpful to keep a separate journal or log on your daily work when designing the evaluation plan. The record of your decisions, tasks that you do each day, and how you solved problems and challenges that emerged can help you better describe the evaluation plan later on when you are asked by third parties or potential funders and purchasers of the product.
- Consult the page on documentation of design for more ideas on how to document your design work in general, and for information on why it is useful as a designer to keep regular documentation.
Related Concepts
Examples
None yet - check back soon!
External Resources
None yet - check back soon!