Generalizability

Generalizability is the ability for an evaluation or research findings to be applied to other contexts broadly. It is the ability for other people to trust that the findings of a study are reflective of other contexts, and not just of those from which the study was conducted.

Definition

Generalizability is the ability to trust that the claims made from a study are reflective of all other similar contexts, despite differences in the data, composition of the subjects, design variations of products, and contexts in which an educational product was implemented.

Additional Information

Generalizability influences the breadth of the claim that an evaluator can make on how the findings from the product can be used in different contexts. In other words, generalizability is how much people can trust the findings from one specific product implementation relates to all similar products that are implemented in different contexts. Audiences may differ, activities might vary, and products may be implemented in different industries or subject domains which have different requirements and ways of working - each of which may make the claims from one study not apply to another study. As a result, generalizability is proportional to the specific conditions and situations that applied to the evaluation in the first place.

If an instructional designer wants to increase the generalizability of their findings from evaluations (such as whether a certain type of learning activity works for learning in all cases, every time), they will need to include and account for in their evaluations the multiple contexts, audiences, and data types that are possible when the product is implemented. For a large-scale, generalizable study, this typically requires a large sample size that is representative across audiences and implementation contexts.

Factors that influence generalizability

The generalizability of a claim will be influenced by a number of factors from an evaluation study. If many of these factors are considered, represented, or controlled for in an evaluation, then the generalizability of the claim can be improved and better trusted to apply to a broader range of contexts.

• Audience and sample selection, including representativeness of the sample chosen for evaluation in comparison to full populations.
• Design features and design variation, including whether the educational product being evaluated can be compared with products with different features or design contexts (i.e., online vs. face-to-face, long vs. short duration).
• Environmental factors and implementation contexts, including when, how, and where an educational product is implemented, in what industry, and what factors related to space/place, environment, and organizational mandates are in place.
• Validity of data, in whether the data of the study are accurately capturing the variables of interest and accurately capture evidence of learning.
• Reliability of data, in whether the data collected would be collected in the same way if the study was conducted again.

Replicability of results from analyses in that the analyses would produce the same results if it were conducted again at a later time and a different evaluation team. This includes whether the analysis methods, data collection, and measurement approaches were documented, and the quality of how the methods were skillfully conducted according to plan.

• Data quality, in that the data are of high enough quality to be trusted.
• Instrument choice is appropriate to capture the types of data that are desired to answer the research questions
• Fidelity of implementation, in that the evaluators measure how closely the educational product is implemented based on the intentions of the designer.
• Participation with and use of design features, in that people's level of participation with the features of an educational product may vary and should be measured to account for different levels of exposure or dosage.

Tips and Tricks

• In most evaluations of a single educational product, large-scale generalizability is not possible, nor desired. Products are typically being evaluated for their performance in specific contexts. If large-scale generalizability is desired, an evaluation of multiple similar products across varied contexts needs to be conducted.
• Generalizability needs to be planned in the earliest stages of the evaluation process. This allows for the evaluators to ensure than a representative sample of participants is recruited to participate with the product that reflects the audience composition and implementation contexts that would allow for broader claims to be made about the effectiveness or usage of a product.
• Limit the breath of the claims that you make about your product to reflect the scope of your project. It is very difficult to claim with certainty or trust that your product works in every situation and for every person, unless you tested that specifically in an evaluation. Instead, limit your claims to what you actually analyzed and found in the evaluation. It is instead more common for an evaluation to claim that a product worked or didn't work in only very specific circumstances. As an example, "this K-12 educational product demonstrated learning achievement for the specific grade level and subject that it was designed" is a more appropriate claim when generalizability was not the goal of the evaluation.

Related Concepts

• Evaluation and assessment
• Audience considerations of design
• Validity
• Reliability
• Data
• Fidelity of implementation (FOI)
• Instrument
• Analysis method

Examples

None yet - check back soon!

External Resources

None yet - check back soon!