Design experiments
From The Learning Engineer's Knowledgebase
A design experiment is a method for the continual development and testing of an educational product to hypothesize, test, and improve the product while it is being used. Design focuses on iterative cycles to continually test and improve (1) the product itself, (2) participation with the product to match expectations, and (3) achievement and outcomes from the product.
Definition
A design experiment is a research methodology that uses an actual educational product to test theory, hypothesize how things work, and investigate how and why people use the product to achieve the desired goals.
In a design experiment, the product or learning environment is continually modified or iteratively changed to see how minor changes to the product subsequently alter the behaviors, attitudes, and structure of the experience. Designers and researchers ask research questions related to how and why certain design features lead to different interactions by participants.
Additional Information
In a design experiment, the design team makes hypotheses as to why they think that elements of design may influencing how people interact and learn with the product. These hypotheses typically examine (1) the specified procedures of the learning experience, (2) the design specifications of media and technology, or (3) the look and feel of the learning environment. By making small changes to any of these elements while the product is being used, the design team can see if any changes in participation occur, as well as learning if the team uses formative evaluation methods.
Participation changes are a primary indicator of how learning might be influenced by changes to the product or environment because a person's involvement and engagement with the product is the first level of the product having any effect. If a person does not interact with the product, a designer can safely assume that the product had no effect. Furthermore, participation with or use of a product is more readily measurable with today's digital analytics tools, especially within digital environments that can capture participation data unobtrusively without bothering the participant to take an assessment or survey.
As such, investigating participation levels and patterns is a good immediate step and source of evidence for design experiments that are actively being implemented in real world settings. This is especially the case when there is little time for conducting summative assessments after every small change or iteration is made to the product.
How to conduct design experiments are beyond the scope of this page. However, the initial concepts for a design experiment can be planned if the research team embraces the idea of making multiple small-level changes to the product while it is being used to improve the user experience, as well as test hypotheses as to how and why the product is being used.
In the education field, projects that are based on design experiments are often called design-based research (DBR). When using DBR, the entire project is organized around the principles of rapid iterative design and testing hypotheses to generate understandings about how and why people interact the way they do. As such, DBR and design experiments are a useful tool for better understanding how learning theories work in practice, as well as for refining existing theories and even generating new theories of learning.
Tips and Tricks
- Consider how you might have the resources to make small improvements and changes to your educational product or experience while it is actively being used. How might you handle this without too much disruption or downtime of the product?
- Think about how you or the design team might generate hypotheses about how and why people are or are not using the product to your level of expectation. Consider the learning theories that your product uses to predict why people will benefit from its use. Also consider other environmental, psychological, or interpersonal factors that might be arising during the use of your product.
- If you would like to consider using design experiments or DBR, set aside specific times for your team to review the logs and participation levels of the learners so that you may regularly how people are using it at expected levels. During this time, generate questions and hypotheses as to why people are or are not using features as you would expect them to. Then, consider changes to the environment that might influence a change in participation.
- After learners are done with your product, consider summative assessments that survey or interview them specifically on the changes that were done to the environment while they were using it (as a part of the iterative improvements in the design experiment). It would be particularly useful to ask participants to reveal their perceptions about the change and what they remember from their experience as to how these changes to the product influenced their behavior.
Related concepts
Examples
None yet - check back soon!
External Resources
None yet - check back soon!