Actions

Analyzing cost and efficiency

From The Learning Engineer's Knowledgebase

Analyzing cost and efficiency is a category of research questions for evaluation that investigates how well an educational product or experience is effective at achieving its goals in comparison to its monetary or staffing resource costs.

Definition

The analysis of cost and efficiency is a category of research questions in the evaluation of educational products that identifies the actual, intangible, and other costs of an educational product. Often, the research questions in this category will also compare the costs of a product to alternative educational approaches to provide information on the cost effectiveness or efficiency of products. Cost analysis and efficiency is used for making decisions about investments, adoption, and implementation of educational products and technologies.

Additional Information

Research questions related to cost are focused on identifying the actual, potential, and perceived costs of an educational product. Sometimes, these costs are compared to other products or alternative approaches. Nonmonetary costs are also considered, which are often converted to actual money costs, including staff time, unpaid labor, facilities, and administration, which are often called indirect costs.

Knowing the costs of a product helps the designer stay within budget, provide a detailed description of the design of the product, and remain within the scope of design and development. Additionally, many purchasers or decision makers of educational technologies now desire information on cost effectiveness, efficiency, and justification. Such information most frequently takes the form of evaluation studies and reports so that people can make informed decisions about the cost of a product in relation to how well they work.

Costs of products can be measured in actual money via total amounts of a product's development, cost per participant, cost per classroom, cost per download, etc. If cost studies are done correctly, costs can additionally be measured as a ratio of cost per unit of change in learners, on average (such as how much the product costs per student to generate one half of an additional standard deviation of improvement, or how much it costs per student to have that student score an A versus a B).

Decision makers who make educational investments in any industry need to typically justify the costs that are incurred with that investment. This typically requires some kind of analysis that answers questions to whether the costs were high or low in comparison to alternative approaches, whether the product was efficient at achieving the goals in terms of costs, and whether the costs can justify continued spending based on the results that have been observed.

Common Research Questions

In this category of research questions, the actual questions typically take one or more of the following general forms:

  • What kinds of costs are individually required for designing, developing, and implementing the product?
  • How much does the product (or did it) cost in total up front, and over time when it is implemented?
  • What kinds of cost savings can be realized over time?
  • How much does the product cost compared to an alternative approach?
  • How much more cost effective is the product compared to an alternative? In other words, how much better does the product work compared to an alternative approach in terms of cost? (this is usually measured as units of something per dollar, such as points on a test increase per dollar)

Common Instruments and Data

There are a variety of instruments and data sources that are commonly used for this category of research questions:

  • Any data source related to costs:
    • Budgets
    • Invoices, bills, charges
    • Work estimates, time sheets, and staffing requirements
    • Any actual expenses incurred for design, development, implementation, and maintenance
    • Cost of tools, technologies, and subscriptions (e.g., web hosting, servers, computing devices)
  • Any instrument that can be used to evaluate learning outcomes, particularly administered in a pre-post way
    • Tests, quizzes, surveys
    • Documents, work products, artifacts
    • Observations and logs of participation or engagement
  • Instruments used to identify and measure participation and engagement with a product, to see the degree to which people use the product in the first place
    • Observations
    • Digital logfiles and learning analytics
    • Self-reported participation surveys
    • Journals, writings, and work products (i.e., artifacts) of participants

Common Variables and Concepts of Interest

To answer research questions in this category, evaluators are typically interested in measuring and examining some or all of these concepts or variables:

  • Direct monetary costs (e.g., budgeted staffing work, assets, tools, contracted work - all in currency measures such as dollars)
  • Indirect costs (e.g., the ongoing overhead of organization/business operations)
  • In-kind staff hours (e.g., staffing work that is not specifically paid in the project budget, often measured in money per hour)
  • Intangible costs (e.g., things that are invisible to budgets and not easily measured in dollar terms, such as emotion, personal effort, morale, reputation)
  • Variables related to learning and achievement, to identify how much different levels of achievement cost

Common Analysis Methods to Answer the Research Questions

Research questions related to cost are usually quantitative in nature due to their relationship with monetary measurements (i.e., dollars). Some qualitative work is usually involved in interpreting the costs, but costs are typically always represented and analyzed quantitatively.

Note: It is beyond the scope of this knowledgebase to expand on each of these methods. It is recommended that researchers and evaluators seek additional training, web resources, or courses on individual methods they would like to use.

  • Cost analysis. In a cost analysis, evaluators will calculate all potential and actual costs of developing, using, maintaining, and evaluating the product. This includes the up-front costs of development, the costs of implementing the product and training any facilitators that are needed, and the costs of ongoing maintenance that are needed. Evaluation costs are also included, especially when evaluation is ongoing or formative in nature. Non-financial costs (i.e., resources, facilities and administration, in-kind goods and services, technology, staff time, unpaid development time) are also included and are converted into monetary amounts.
  • Cost/benefit analysis. A cost/benefit analysis investigates all of the financial and social costs of implementing a new technology or educational product in comparison to the benefits that would be gained from implementing it. It is often a descriptive type evaluation that discusses how the benefits or the costs outweigh each other for a pending decision in an organization or school, such as the implementation of a new learning product or expensive educational technology.
  • Cost effectiveness analysis. These types of more complex analyses compare the overall cost of a product (as determined in a cost analysis) and convert it to a figure of cost per unit of change in the learner. These studies interpret the cost as a ratio of the cost per participant per unit of change - with the unit of change being defined by the evaluator. This requires the learning objectives to be clearly defined and measured in an evaluation so that the total cost can be divided by the learning outcomes measurement. For instance, a college-level course would measure the cost effectiveness of an educational technology tool to help with collaboration by stating that for every grade letter change (say, from a B to an A), the cost per student is $200. This could help teachers and administrators compare products to see which products bring about a positive learning outcome for which price.
  • Price or resource comparison. In this type of analysis, the calculated financial or resource costs are directly compared to multiple products, often without much regard to what different features they provide or a comparison of their learning objectives (like in a cost effectiveness analysis). Product costs are simply compared across each other.

Related Concepts

Examples

None yet - check back soon!

External Resources

Retrieved from "https://lekb.org/index.php?title=Analyzing_cost_and_efficiency&oldid=29"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.