EVALUATION TOOLS

Evaluation Matrix

• Although by all appearances, the "Evaluation Matrix" is a very simple tool, it has a powerful purpose. It helps you to consider a wider range of data collection methods than you might otherwise consider in relation to each of the questions addressed by your evaluation. Evaluators sometimes get into the habit of using one or other data collection method, e.g., an end-of-training questionnaire, without considering the advantages of alternative methods. This tool prompts you to consider each evaluation question and to decide which of the many data collection options have the greatest potential for providing the desired information.

Anecdotal Record Form

• Evaluation data does not have to be reported as "cold hard statistics." Often you will want to tell the "human story" involved in your development or implementation project. One way of capturing those important stories and critical incidents that provide the human story is the "Anecdotal Record Form." Participants in an interactive multimedia design project can use this instrument to describe a noteworthy event and to offer their own interpretation of its relevance. It is very important to try to complete an Anecdotal Record Form as soon as possible after a critical event has occurred so as not to forget critical information. It is equally important to separate your description of the incident from your interpretation of it!

Expert Review Checklist

• Expert review is one of the primary evaluation strategies used in both formative (How can this multimedia program be improved?) and summative (What is the effectiveness and worth of this multimedia program?) evaluation. It is often a good idea to provide experts with some sort of instrument or guide to insure that they critique all of the important aspects of the IMM program that you want reviewed. This "Expert Review Checklist" has been designed for use by an instructional design expert. You would employ different sorts of Expert Review Checklists with different types of experts such as a content expert or a human computer interface expert.

Focus Group Protocol

• Focus groups are a powerful means of collecting data about learner or instructor reactions to a new interactive multimedia program. However, focus groups need to be carefully planned so that you get the kind and quality of information you are seeking. This "Focus Group Protocol" is a brief example of a list of questions that might be addressed during a focus group regarding an interactive multimedia program.

Formative Review Log

• The "Formative Review Log" is a simple instrument that can be used by anyone you have asked to review your program in its formative stages. The instrument has three columns, the first for recording the screen or format sheet number that the person is reviewing, the second for writing down observations (e.g., errors, confusing points, or ideas), and the third for recording what actions have been taken in reaction to the feedback provided by members of the project team. Using an instrument like this with many different types of users will probably have the greatest pay-off for formative evaluation throughout the life of the project.

Implementation Log

• It is one thing to plan and develop a good interactive multimedia program. It is entirely another thing to implement it as planned. Many training innovations have failed because implementation factors (such as instructor motivation) were not considered. It is essential to make every effort to collect information regarding the actual use of an interactive multimedia program as compared to the planned use. The "Implementation Log" tool has been designed to make that comparison a little more systematic.

Interview Protocol

• Interviews are a powerful means of collecting data about learner or instructor reactions to a new interactive multimedia program. However, interviews need to be carefully planned so that you get the kind and quality of information you are seeking. This "Interview Protocol" is a brief example of a list of questions that might be addressed during an interview regarding an interactive multimedia program.

Questionnaire

• Questionnaires are undoubtedly the single most frequently used type of evaluation instrument. Poorly designed questionnaires are often administered at the close of a course or training session as a "smilometer" or "happiness indicator." They are also often distributed to users of interactive multimedia programs. If the only thing you find out about your interactive multimedia program with a questionnaire is whether the trainees liked it, you are not making good use of this strategy. As shown in the "Questionnaire," a wealth of information can be provided by a well-designed instrument.

User Interface Rating Form

• The "User Interface" of an interactive instructional product, e.g, a multimedia program, is a critical element of the product that must be carefully evaluated. If the user interface is not well-designed, learners will have little opportunity to learn from the program. This rating form includes ten major criteria for assessing the user interface for an interactive program, such as "ease of use" and "screen design." Not all of the criteria may be relevant to the particular program you are evaluating, but most of them will. You may need to add additional criteria to the list. Novice users of interactive instructional products are generally not good candidates for using this form. The people rating the user interface should be experienced users of the type of program you are asking them to rate. Even better, they could be experienced designers of interactive programs.

Evaluation Report Sample

• The "Evaluation Report Sample" presents one way of structuring an evaluation report. Evaluation reports are notorious for being weighty volumes that few people read. Not surprisingly, lengthy reports have little effect on decision-makers. This tool illustrates a strategy for dividing an evaluation report into two-page sections that each include four parts:

  1) an attention-getting headline,
  2) a description of the major issues related to the headline,
  3) a presentation of data related to the issues, and
  4) a bottom-line recommendation or summary of the findings.

  People who receive a report in this format can take two or three sections at a time and make them agenda items for their team meetings. In this way, the evaluation findings are much more likely to have an impact on practical decisions.