Competency evaluation

Evaluating the competency of a participant investigates whether the person has specific knowledge, or can do specific tasks. It does not investigate whether a person learned how to do that task from an educational experience, as they could have already known it before they began the activity (which is instead investigated in a learning effectiveness evaluation).

Definition

Competency is the degree to which a person can perform a specific skill, or recite and talk about their knowledge.

Competency evaluation are assessments and evaluation tasks that measure and question whether a person is competent at the learning objectives, specifically whether a person can perform desired tasks or recite/talk about knowledge.

Competency evaluation does not evaluate whether learning occurred. Learning, in comparison to competency, is whether a person actually learned from their experience and had a change in their skills or knowledge, as a person could have already had the knowledge and skills before they engaged with the experience.

Evidence of competency is required for such an evaluation to happen. Instruments, also known as assessments, are used to collect evidence that demonstrate a person's skill level with reciting knowledge or performing specific tasks. This evidence can be reviewed using analysis methods to allow the evaluator to interpret how, why, and to what degree a person demonstrated competency.

Additional Information

Competency evaluations is one of the most common forms of evaluation that are done on educational products and experiences.

In most learning situations, an individual participant is assessed at the end of a learning experience to see whether they can satisfactorily perform the skills or recite the knowledge that the experience aimed to teach (i.e., the learning objectives). This readily seen in K-12 and higher education settings where final exams, projects, and papers all ask students to perform specific tasks and demonstrate their competency with the material that was covered in the class. A student performs well in this type of evaluation and demonstrates mastery if they can adequately mimic the types of behavior that are desired, such as answering multiple choice questions correctly, or including specific knowledge on a term paper.

Measuring competency can be a challenging task for evaluators. Finding evidence of a learner's skills and knowledge is needed before any judgement of competency can be made. The instruments that are used to collect data must require students to produce any desired skills and actions, as well as recite any knowledge that are in the learning objectives of the educational experience.

Common instruments and ways to measure competency

Tests, quizzes, and surveys that require students to identify, recite, and describe their knowledge, as well as use skills to perform tasks or procedures on a test. The test, quiz, or survey is measured by how well a student performs on test items in comparison to the expectation (such as, finding the correct answer, or including all the required information). A total score of correctly responded items can be summed to indicate overall competency with the test.
Projects and documents that are made by learners. Completion of projects requires the learner to use specific skills. Elements of the project that are present or absent can be used to show evidence that a learner knows how to do something.
Evidence of participation in social settings can be used to see the quality of interaction that a learner has in social settings, depending on the learning outcomes. The participation of an individual may be evidence for achieving some types of learning objectives, such as whether the learner can participate in various roles in a social setting, make constructive arguments, or share information productively.
Analytics / logs of how a person uses a digital technology, which can capture records of a person's use of a technology. If the learner demonstrates that they can perform certain tasks within the system as evidenced by a record of them doing it in the log, then competency might be evidenced in this way.
Rubrics can be used to clearly state how competency is measured. Each item of competency usually ranges from 0 to a higher number (3 or 4, for instance), with higher scores indicating more demonstrated competence with each item.

Common approaches to evaluating competency

As with any evaluation approach, any given specific approach is beyond the scope of this wiki. Each method could be an entire college course of its own. Readers are encouraged to learn about and review methods in more in depth if they are interested in the types of research questions that each method can answer.

Tips and Tricks

Consider what exactly you would like your learners to be able to do after they finish using your educational product. These are the types of competencies that you would be wanting to find evidence for when evaluating whether your product worked. Think about how you might use tests, projects, or other instruments to find out if learners can do the skills or recite knowledge from the learning experience.
When looking for evidence of competency using an instrument like a test or a project, think about what successful use of a skill or talking about knowledge looks like.
- For knowledge, it may be as simple as a person being able to recite specific information or repeat from memory definitions. It could also be people filling out a correct answer on a multiple choice exam about a concept's definition.
- For skills, you may consider having a person do a skill or procedure that leads to a specific answer. This is most commonly seen in math problems, where you must do the problem to get a specific correct answer (which can be answered on a test). For other skills without a single correct answer, you may think about whether expected pieces of a project are present or absent, or the how well each piece of a project is completed based on an established set of criteria or rubrics.

Competency evaluation

From The Learning Engineer's Knowledgebase

Contents

Definition

Additional Information

Common instruments and ways to measure competency

Common approaches to evaluating competency

Tips and Tricks

Related Concepts

Examples

External Resources