Evaluating the quality and standards of educational products

The evaluation of the quality and standards of educational products is a category of research questions that identify whether an educational product or experience meets a pre-established threshold for if it is considered high quality, or a "good" product. This typically requires the product to be compared to some list of standards, criteria, or design principles so that the product can be evaluated.

Definition

Quality and standards are criteria that are used to determine whether an educational product achieved a pre-established level of standards that indicate that the product was "good".

Additional Information

Standards and quality measures typically emerge in specific fields based on best practices and research that has repeatedly demonstrated what makes a product successful or approaches that help people learn best.

Standards and quality measures are based on the goals that they are seeking to measure. For example, different standards can measure high quality online learning (e.g., Quality Matters Standards), media production value or standards, high levels of accessibility (e.g., W3C web accessibility standards), and the usability of technology (e.g., ISTE standards).

Quality measures and standards are typically established by professional organizations, governing bodies, and administrative decision-makers in various fields.

Design teams, funders, investors, or other decision makers may also provide a custom list of criteria by which the product should be evaluated (e.g., a custom set of goals or performance criteria).

Criteria and standards-based measures are often evaluated with yes or no questions like "did it meet the mark?", rather than more exploratory questions like "how did people learn?" or "in what ways did people use the product?" This allows evaluators to claim that a product did or did not achieve an acceptable level of quality or standards. What is deemed "acceptable" or "passing level", however, is decided by the writers of the standards. Additionally, the design team of the product can choose a benchmark or expected level for criteria that they want to meet, such as the product's features being used with a high enough frequency, or participants scoring high enough of

Common standards in the education industry include:

• The ISTE Standards for technology integration into educational products, classrooms, and curricula
• The Quality Matters standards for online courses and programs
• The W3C standards for web accessibility
• Educational standards for curriculum, content, and skills that should be within and aligned with K-12 educational products, including:
  • The Common Core State Standards
  • The Next Generation Science Standards
  • The National Curriculum Standards for Social Studies
• The Program Evaluation Standards, which are standards for the design of evaluation plans
• The Universal Design for Learning (UDL) Standards, which are standards for designing educational products with a focus on inclusivity and encouraging everyone to learn
• The AECT Instructional Design Standards, which are for ensuring that educational products use industry best practices in their design
• The Kilpatrick Model of educational evaluation, which reviews an educational product's performance on four levels and compares it to "benchmarks" or levels that are expected to be achieved.

Common Research Questions

In this category of research questions, the actual questions typically take one or more of the following general forms:

• Did the product meet the expected mark on each of the criteria?
• Did the product achieve a "passing grade" on this list of standards?
• Did the participants on average score adequately on the exams that measured learning outcomes?
• Did the product include each of the required elements in the list of quality measures?
• Does the product show evidence of addressing each of the items on the checklist for quality?
• Did the product show positive evidence at achieving each of the established outcomes?
• Did each of the product's features get used by people at the expected level?

Common Instruments and Data

There are a variety of instruments and data sources that are commonly used for this category of research questions:

• A list(s) of standards, criteria, or design principles, which tells the design team what exactly is expected and at what level they should achieve these things to have an adequate score to "meet the standards"
• Checklists of present/absent items that are expected to be present in the design.
• Rubrics that judge the varying levels of quality or demonstrated evidence for each item.
• Documented design specifications of the educational product being evaluated.
• The developed educational product itself will be evaluated for whether it meets the established criteria or not.

Common Variables and Concepts of Interest

To answer research questions in this category, evaluators are typically interested in measuring and examining one or more of these concepts or variables:

• Achievement measures, such as test scores, competence on tasks, or successful completion
• Ratios of people who finish tasks successfully to those who do not
• Frequency of product use, particularly with specific product design features
• Presence/absence of specific design features, media, elements, or stylistic choices

Common Analysis Methods to Answer the Research Questions

Both quantitative and qualitative analysis methods are commonly used to measure and describe the degree to which educational products achieve the intended goals of competency and learning. Method choices should align with the intents and desires of the people who will be using the evaluation data. Quantitative data on effectiveness is typically preferred in most contexts, but both standalone qualitative methods and mixed methods that combine both are becoming increasingly practiced.

Note: It is beyond the scope of this knowledgebase to expand on each of these methods. It is recommended that researchers and evaluators seek additional training, web resources, or courses on individual methods they would like to use.

Quantitative Methods

• Presence / absence of key elements. A simple presence/absence checklist is often used to identify whether key elements within a product are present or absent from the product's design. A criteria checklist can also be used to numerically score and analyze components as a total score, for which a "passing grade" can be established.
• Measurements of product use. The criteria for standards and "good design" can often include the use of measures of the frequency of how often certain design features and elements of the product were used. Low-scoring features would likely not meet pre-established criteria.
• Rubric comparisons. Rubrics are frequently used for measuring standards and quality criteria. The rubric provides a numeric score based on the level of performance or evidence of a criterion measure, which can be combined into a summed-up scale or total score.

Qualitative Methods

• Case Study. Case studies are frequently conducted to evaluate whether a product achieved a set level of standards or pre-established quality measures. Each standard or criterion is explored and discussed in the case study, and the study makes a claim of whether or not the product met the standard.
• Basic descriptive analysis. Design reports and descriptive accounts can evaluate an educational product based on whether it met expected standards or quality criteria. Descriptive analyses are always stronger if they include data sources and evidence from multiple areas to show that it is not just the subjective opinion

Mixed Methods

• Both qualitative and quantitative methods may be used in evaluating what was learned and how effective educational products were at achieving their goals or met quality standards or critieria. Both methods may be used simultaneously to confirm and support arguments.
• Mixed methodology is becoming an increasingly common approach toward showing multiple sides of the same type of analysis to answer research questions. Such approaches add cross-validation and triangulation so that multiple perspectives are considered and support each other when making claims about a product.

Related Concepts

• Evaluation and assessment
• Research question
• Data
• Instrument
• Analysis method

Examples

None yet - check back soon!

External Resources

None yet - check back soon!