Research question

A research question, when answered, provides evidence and interpretative results in an evaluation of an educational product or experience. All evaluations ask research questions that, when answered, give evaluators insights about the product or experience being evaluated on whether the product worked as intended or the degree to which participants learned from their use of the product or experience.

Definition

A research question is a guiding question that is asked in an evaluation to produce data and insights about how a product was used, and what effects or outcomes the product had on learning.

Data that are collected for evaluations must describe the things that the research question is trying to ask for the analysis to have strong validity.

Research questions are typically answered in a systematic and documented way so that the insights can be trusted to be able to be replicated and are not just the purely subjective opinions of the evaluator, but instead are generated empirically using evidence from actual use of the product.

A checklist of potential research questions for educational evaluations. Not all of these research questions have to be asked in an evaluation. Research questions can be categorized as participant-level, or those that focus on the evaluation of individual participation and learning, or the product-level, which focuses on the evaluation of both individual design features and the whole product or experience.

Additional Information

Answering research questions is the primary functional goal of educational evaluation and assessment. Without a research question to answer, an evaluator has no clear path or direction on how to collect and analyze evidence to see how and why an educational product worked.

Research questions are typically the starting point for any evaluation of an educational product. Once written, research questions are subsequently used to determine what kinds of data should be collected (via instruments) and how the data should be analyzed (via analysis methods). As the research question determines what kinds of insights can be generated, the combination of instruments for data collection and analysis methods for an evaluation is customized to the research question being asked.

The answer to what research questions to ask in an evaluation is that it depends based on the goals of the evaluator. In short, this means that there is no standard evaluation plan or model that is conducted for every educational product. With this in mind, all evaluations should be custom built in a way that balances what the evaluators and design team wish to know about the participants' experiences and product use, as well as what the audience(s) of the product wish to know about the product's quality and value.

Additionally, evaluations may be limited in scope and the number or types of questions that can be asked based on the available budget for evaluation, limitations in the types of data that can be realistically collected for analysis from the participants, or the level of expertise of the evaluation team.

A balanced evaluation or assessment of an educational product should ask enough research questions to reveal valuable insights about participant learning, participant perceptions, and whether the product was used as expected - but does not overburden the evaluation or design team with evaluation tasks or data collection.

Types of research questions

There are multiple types of research questions that are commonly asked in educational evaluation and assessment plans. Each research question category elicits a specific type of data requirements and subsequent analysis methods to validly and reliably answer the research question in a trusted, believable way.

An evaluation plan does not need to contain each of these question types, but instead it is more important to have a balanced set of questions that provide enough information to reveal what the evaluator wants to find out, but does not overburden the evaluator, design team, or participants with evaluation activities at the expense of learning.

Participant-Level Research Questions

Participant-level research questions are those that investigate individual participants' experiences and interactions with educational products, interactions with other participants, and whether their participation had any influence on competence or learning. Participant-level research questions typically collect data on individuals and their knowledge, behaviors, interactions, and psychological states. Participant-level research questions generate insights for making claims about individuals and their experiences.

Frequently, participant-level research questions are directly related to the learning objectives of a product, with questions asking whether people achieved or demonstrated the established learning objectives.

Categories of participant-level research questions:

• Evaluating learning and competency outcomes
• Identifying factors and relationships that influence outcomes
• Identifying participant perceptions and satisfaction
• Identifying how people participate, interact, and learn

Product-Level Research Questions

Product-level research questions investigate the product itself and its features. Product-level research questions allow evaluators to generate insights and evidence for making claims about how features of a product was used, whether products work, or how well products compare to alternative approaches.

Categories of product-level research questions:

• Evaluating the quality and standards of educational products
• Comparing educational products and alternatives
• Identifying product and design feature usage
• Reporting on design processes and design specifications
• Analyzing cost and efficiency

Tips and Tricks

• When writing research questions for an evaluation, start by considering what kinds of things you want to know. Think about the kinds of claims that you want to make about a product, such as "the product helped people learn" or "the features were used as expected." You need good evidence to make these claims, and research questions that are appropriately aligned to the things that you want to know can provide you with a roadmap on how to find out information and insights about a product.
• Most research questions for evaluations typically come from one of the categories listed in the "types of research questions" section discussed above. If you are writing a new evaluation plan, these basic questions are good additions to any plan.
• Your research questions will determine what kinds of data that you need to collect and what kinds of methods you will use in your analysis. Don't try to force a method or data collection instrument that you wan't to use into a research question for which they are ill-equipped to answer! The instruments and analysis methods respond to the research question, not the other way around.

Related Concepts

• Evaluation and assessment
• Instrument
• Analysis method
• Evaluation plan
• Evaluation model
• Learning objectives and outcomes

Examples

None yet - check back soon!

External Resources

None yet - check back soon!