Evaluation of instructional technology resource

Guidelines for the Evaluation of Instructional Technology Resources.

Appendix C

Screening Criteria for Interactive Technology Resources

(Criteria is in black. My responses are in blue).

CURRICULUM CONTENT

The program appears to cover the content recommended in appropriate local, state, or national curriculum frameworks and content and performance standards. The 8 musical animations were created as a supplementary resource and not as an entire curriculum. There are practical components for Performing Arts that rightly require students to make their own music and not simply learn about musical theory. (see Curriculum)

The program is without cultural, gender, or racial bias in content and format. Yes.

When appropriate, there is representation of diversity in population groups. N/A

The content is presented impartially and without bias or distortion. Yes.

Any inherent bias is clearly identified and is necessary to the program design. N/A

INSTRUCTIONAL DESIGN: OBJECTIVES

The main objectives are readily identifiable and are stated clearly. Yes. See Introduction to research issue.

The scope / scale of the program as defined is appropriate to the stated objectives, e.g., whether this is a single concept program or a multi-year courseware system. I hadn't specified this as my thesis was on the animation medium itself more than the content of the animations. I would however, say that this approach lends itself to a courseware system as the exact content can be expanded for any range of age groups or purpose.

Appropriate use of the program appears likely to assist in the attainment of the stated objectives. The improvement demonstrated by all six viewers in the Data section supports this.

INSTRUCTIONAL DESIGN: TEACHING AND LEARNING

The program content clearly is of value in an educational setting and provides a sufficient quantity of useful information. Yes.

The instructional design of the program appears to show some attention to pedagogy and to current research on technology-based instruction. Yes. My literature survey on teaching with animation explores this area in detail.

Ideas, concepts, and theories are presented in a manner that supplements / enhances more traditional instructional materials. Yes. Supplementing and enhancing traditional methods and resources was my prime objective.

The instructional design, e.g., tutorial, demonstration, simulation, skill building, etc., is appropriate for the stated goals and objectives. Yes.

The program content follows a logical development and organization. Yes. The six animations are sequential and the thesis, although presented in HTML, follows conventional layout and categories.

Reading and vocabulary levels are commensurate with the skill level of intended learners. The vocabulary of the thesis was for adults but the vocabulary of the animations was for upper primary and above. This was my intention and the consensus of most of the viewers when asked whether the animations were suitable for children. (see Interviews)

Word lists, problem sets, sound, etc., can be customized by the student or teacher when appropriate; any such teacher settings cannot be altered by the learner. No.

Any game format utilized for instruction, reinforcement, or motivation is appropriate and enhances the overall instructional design. No games were used although the educational merit of games was discussed in the literature survey on teaching with animation. The key signature animation was a learning environment.

The design of the program avoids excessive competition and violence, especially instructional games where the goal is to shoot, kill, or destroy. N/A

INSTRUCTIONAL DESIGN: STUDENT RESPONSE

The learner is actively engaged in the learning process with a focus on the goal of attaining program objectives. Yes. Animation is a very engaging medium. This was discussed in various sections especially interactivity.

There is consistency between the content of a lesson and the way it is presented that is appropriate for the interests, skill level, and age level of the learner. Yes.

Responses to learners are supportive, appropriate, non-judgmental, neither demeaning nor ego threatening, and are not highly repetitive. There were no specific responses required from the user. The nature of the animations being user controlled could be considered “non ego threatening” as the user can watch them many times without fear of standing out in the classroom.

The reward for success is designed appropriately to motivate continued good performance while avoiding over-praising. N/A

INSTRUCTIONAL DESIGN: TECHNOLOGY

The technology used is an effective medium for presenting program content; it is not the case that technology is being used only for technology’s sake. My whole rationale for teaching with animation is that it is effective for presenting program content.

When two ore more technologies are combined, the result adds to the learning experience over that attained from the use of one technology alone. The animations use various multimedia elements but they could be considered to only use one technology. The multiple elements, such as simultaneous video and audio, is very much a part of the teaching objective.

All audio and visual components are appropriate and do not detract from the program content. Yes. There were no unnecessary sounds, images or movement.

Graphics, icons, and instructional support materials for the computer and / or video portions of the program clearly support and reinforce the basic instructional design of the program. This was my conscious intention. The details of this are discussed in the Research methodology section.

PROGRAM DESIGN (USER INTERFACE): STUDENT RESPONSE

Interactive strategies are intuitive, straightforward, and clear, allowing learners to focus on the instructional concepts rather than on the mechanics of the program. Yes.

Students can begin using the program effectively and independently within a short period of time with only limited teacher instruction. Yes.

Learners at all levels can easily and independently control program operation. Yes.

Learners can alter responses before they are processed, unless instructional objectives dictate otherwise. N/A

The student is prompted to make a response after a reasonable amount of time has elapsed. N/A. If this work was expanded into a complete courseware type package there would be student input with timed prompting.

When appropriate, the program limits the number of incorrect responses and, except in timed drills or tests, provides help, hints, or the correct answer after a reasonable number of attempts. N/A

The learner is instructed to enter the correct response before continuing with the program. The learner was not instructed to enter any responses. Viewer noted that maybe the user should have to answers questions correctly before moving onto the next section.

The amount of typing is appropriate to the grade or ability level of the learner. N/A

PROGRAM DESIGN (USER INTERFACE): INSTRUCTIONS/HELP

Instructions for program operation are clear, concise, complete, and avoid the use of “computer jargon.” Yes.

Instructions are available on screen as needed. Yes.

Learners can bypass instructions and / or return to them at will. N/A . Although instructions weren't a big part of this thesis, the concept of not boring the user with information they might already know was utilised throughout the various sections within the Theory menu. e.g. When learning about Straight / swing, the word triplets is linked to a section all of its own. A user who already understands the triplet sub-divisions of swing would not need to see this information unnecessarily.

Help screens can be accessed quickly and easily, provide clear explanations for all aspects of the program, and offer context-specific help as appropriate. N/A. The various other programs that are used to display the information have their own help menus, i.e. web browser, media player, etc.

Any error messages are simple, explicit, and helpful. Yes although these messages are generated by the user's own software such as their web browser.

The program offers help when a response appears in an unexpected form. N/A

On-screen tutorials designed to explain operation of the program are organized logically, are clear and understandable, and can be accessed and exited with ease. N/A

Tutorials are logically organized, understandable, and helpful. N/A

PROGRAM DESIGN (USER INTERFACE): NAVIGATION

Menu choices are logical and are displayed clearly. Yes.

Any icons used are logical, easily understood, and represent metaphors from the real world whenever possible. Yes. The only icons used are for video , audioand the Theory menu . I made a slight variation on the video icon for the animation video in the index page.

The user interface is consistent throughout the program. Yes.

Function keys are used effectively and consistently. N/A

One designated key is used consistently throughout to return learners to a previous menu. Yes. The "back button" on the user's web browser always takes them back to the previous page and every page has a link to the Main menu at the bottom.

There is a cursor or other indicator to show where the response is to go. N/A

The teacher or the student, as appropriate, determines the pace at which the learner moves through the program. Yes. Being self-paced in an important aspect of this thesis.

Introductory screens, including company logo and copyright statements, are kept to a minimum and / or can be can be bypassed by repeat learners. Yes.

Program entry points are varied and are appropriate for the program. No. The Main menu is the entry point because it is the master menu. Someone could go directly to a different page if they typed the extended address into their web browser but it would be quicker to navigate their way from the index.

Menus or icons allow direct access to specific parts of the program. Yes.

The learner can exit from any point in the program through a consistent routine. Yes.

Keyboard templates, guides, labels, and / or displays are provided to assist the learner as appropriate. N/A

PROGRAM DESIGN (USER INTERFACE): SCREEN DISPLAY

Any reading of on-screen text is kept to a minimum. In the animations, text is kept to a minimum. In the HTML pages of the thesis there is a lot of text but diagrams are used wherever they are needed to enhance the text.

In programs requiring lengthy text, the amount displayed on screen and the style of the display is appropriate for the intended learner. Yes.

Any lengthy textual sequences can be printed to be read off-line. Yes.

Graphics, icons, and instructional support materials for both the computer and video portions of the program have been carefully designed to have the same “look and feel.” Yes.

PROGRAM DESIGN (USER INTERFACE): ACCURATE AND CURRENT

All information is current, thorough, relevant, and is updated as often as is appropriate. Yes.

The content is not likely to be dated by slang phrases or poorly selected examples. I always avoid using slang.

All content is factually accurate, including style, usage, formulae, calculations, etc. Yes.

All grammar and spelling are correct unless program design requires inaccuracy. Yes.

Where simulations are used the models are valid and all data are accurate, with the models neither oversimplified nor too complex for the developmental level of the intended learner. Yes.

PROGRAM DESIGN (USER INTERFACE): PRESENTATION

Sequencing of content is presented in a variety of ways to interest repeat users. The sequence is user determined.

Maps, graphs, and other illustrations are easy to read, simple to interpret, and technically accurate. Yes.

Any commentary complements and supports the visual material and the visuals enhance the commentary. Yes.

For commentary in any language, the grammar and syntax are correct, the pronunciation is clear and easily understood, and all pronunciation and grammar are standard for that language. Yes.

Any use of accent or dialect is designed specifically to support the instructional objectives, and content of the program. There were different accents between the two voice-over people involved but in each case, they used their natural accent.

Any use of on-camera narration (talking heads), voice-over narration, dialog, and / or dramatization is designed to maintain student interest and involvement. Yes.

Pace of the narration is appropriate for the intended grade level of the learner. Yes although most people noted that the pace of the Reading charts animation was too fast.

Pace of the narration is designed to maintain student interest and is suited to the content. Yes.

As appropriate, the program uses a variety of image types, e.g., full-motion video, live action, still images, light microscopy, x-rays, computer graphics, animation, etc. Yes.

Motion sequences are paced appropriately for the program content. Yes.

Any non-narrated visual sequences are designed to enhance the production and set the mood. Yes.

PROGRAM DESIGN (USER INTERFACE): INSTRUCTIONAL SUPPORT MATERIALS

A description of program content is provided. Yes. See Introduction to research issue .

A user’s manual and / or an on-screen tutorial is available as needed. N/A

The user’s manual is indexed thoroughly and arranged in a logical manner. N/A

Teacher support materials and instructions for using the program are provided as needed. N/A

Instructional objectives or skills to be acquired or practiced are identified as appropriate. Yes.

There is a list of any books, equipment, additional resources, or materials required or recommended for use with the program. N/A

TECHNICAL QUALITY: PROGRAM OPERATION

The program and all components operate correctly and as advertised. Yes.

The program and all components operate on “industry standard” equipment. Yes.

All components operate effectively and efficiently with minimum delays and without error. Yes.

Unanticipated input does not disrupt program use. N/A

When appropriate, the program recognizes the absence of, or a problem with, peripherals and delivers appropriate messages to continue operation. The electronic presentation of this thesis is not actually a program so normal responses from the user's operating system can occur as usual.

Navigation and other commands / icons conform to any emerging industry standards and are displayed consistently throughout the program. Yes.

Program installation requires a minimal level of computer expertise. N/A

Staff time required for maintenance is reasonable. N/A

Access speed is fast enough to maintain student interest when the product is used on hardware components recommended by the publisher. Yes.

The program allows the learner to save and / or print work in progress, when appropriate. Printing is possible on each page but there is no student work to be saved.

Multiple levels of resolution are available depending on hardware capability. Yes.

The size of any on-screen button is appropriate for the developmental and / or skill level of the learner. Yes.

Software allows for upper and lower case character printing. Yes.

A mouse or other non-keyboard control is supported by the program as appropriate. Yes.

A text-only program operates effectively with both colour and monochrome monitors. Yes.

Graphic images can be saved to file and / or printed if hardware supports this function. Yes.

TECHNICAL QUALITY: AUDIO

The audio level is consistent throughout the program and can be set for individual or large group use. Yes.

The speech is clearly enunciated and can be understood by both native and non-native English speakers. Yes.

Sound can be controlled by the teacher or the learner except when it is an essential element of the instructional strategy. Yes.

Any available audio can be controlled by the learner, unless preset by the teacher, and / or a headset can be used. Audio volume can be controlled by the user, with or without headphones. Yes.

All narration, dialogue, and background sounds and / or music are clear. Yes.

All music is reproduced with high fidelity. All audio is CD quality, i.e. 16 bit stereo at 44,100 Hz.

Any digitized speech is clear and easy to understand. Yes.

Any digitized music is reproduced accurately and clearly. Yes.

The program has high-quality stereo reproduction and sound separation. Yes.

The volume of any background sounds or music enhances rather than detracts from the production. Yes.

There is complete synchronization of narrative with the visuals. The original animations made available to the viewers were in Flash format as Shockwave files. (.swf) Although I ensured that the audio and visual were synchronized within the Flash environment, I could not guarantee how they would play on the each viewer’s computer. Although Flash has an output to video function, this did not sync the audio and visual accurately. Using the original audio tracks as the master timeline, I used a video editing program (Adobe Premiere Pro) to speed up or slow down various passages to ensure complete synchronicity. These video files were then saved in Windows Media format (.wmv) to reduce the file sizes.

Programs with audio features requiring a sound card can run silently if the card is not available. Yes.

When there is provision for the learner to input audio responses, these can be played back with sound that is clear and easy to understand. N/A

TECHNICAL QUALITY: VISUAL

Screen displays are uncluttered, easy to read, and aesthetically pleasing. Yes.

Visual space is used appropriately for program goals, intended use, and delivery medium. Yes.

The amount of information displayed at one time is appropriate to the age and skill level of the learner and is not so much that it becomes confusing. Some of the pages are quite long but the process of scrolling allows the user to control how much they see at one time.

All visual presentations are clear and easy to interpret. Yes.

The quality of any visuals is as good as the resolution of the equipment used will permit. Yes.

All colors are vivid and provide an accurate representation within the limits of the display. Yes.

All motion sequences flow smoothly when the product is used on hardware components recommended by the publisher or producer. Yes.

Any zoom function displays the enlarged or reduced images in clear detail. Yes.

There is complete synchronization of narrative with the visuals. Yes.

Exposure and contrast are correct. Yes.

Motion and still visuals are sharp and in focus. Yes.

Still frames are frozen with no distracting jitter or vertical or horizontal vibration. Yes.

All motion sequences are filmed at 30 fps to eliminate inter-field flicker. In Australia we use the PAL system which is at 25 frames per second. There is some flickering on the video files but I worked hard to minimise this. Flickering is removed when viewing the original flash files at the expense of audio sync.

Any stock footage is clean and free of splices. N/A

Only edit masters have been used for film to videodisc transfers. N/A

Cropping of visual images is accurate. Yes.

Transfer of images to videodisc is exact, i.e., images are cropped appropriately. N/A

True-size relationships are illustrated. Yes.

Visual effects such as wipes, dissolves, or cuts enhance the production. Yes.

Students, cartoon figures, or other personifications portrayed in the visuals are appropriate for the age level and developmental level of the intended audience. Yes. The only personification used is the skipping stick figure in the Time signatures animation.

The visuals are clear and the contrast is appropriate for lighted classroom viewing at a distance. Yes.

Colours are selected for maximum discrimination when used on non-colour screens. As most of the graphics relate to sheet music, black and white were the two main colours used. Red, blue and green were also used for highlighting purposes and all of the colours translate well. (Monochrome screens aren’t even available anymore).

A good selection of both video segments and collections of stills is available to support the topic, even if the stills have to come on a separate CD-ROM. All such media is incorporated directly into the thesis.

Appendix F

Exemplars for Future Development of Interactive Technologies

PROGRAM OPERATION

The program accepts open-ended, natural language input. No.

The program will easily print, display, and / or electronically transmit assessment components, instructional segments, performance records, learner-created materials, etc. Yes.

There is easy access to any online version of the database or online extensions of the program. Yes.

There is Internet linking capability. Yes.

The program supports both interactivity with, and transferability to, other technology resources. Yes.

Efficient data compression keeps the number of disks to a minimum. The entire thesis is on the web site www.brendanpauljacobs.com. Audio is compressed using the MP3 codec and videos are in Windows Media format.

Local information / data / names can be integrated into the program. N/A

PLATFORMS/OPERATING SYSTEMS/NETWORKS

The program design utilizes new developments in hardware and / or operating systems in an exemplary integration of new and emerging technologies. Yes.

The program will operate on older systems as well as on state-of-the-art systems. Yes.

The program operates identically on multiple platforms. Yes.

The network version appropriately utilizes any available LAN/WAN/Internet capabilities. Yes.

The program can be installed or downloaded to a network via a license rather than purchased on disk and stored in a box on the shelf. N/A

PRODUCTIVITY TOOLS

When the tool is available on multiple platforms, the files translate easily between platforms. Yes.

Flexibility is allowed in customizing tool bar displays. Web browser toolbar setting options are set by the user.

There are multiple layers of “undo.” N/A

An easy-to-use “Bookmark” feature is available for individual learners. Yes.

Any search strategy supports natural language searching. Yes.

VISUAL

The program uses state of the art computer graphics, including customized three-dimensional computer animation. Three dimensional graphics are not helpful when dealing with sheet music.

Multiple points of view are provided for the same visual. No.

Visual graphics are presented in 3-D as appropriate. N/A

Original images have been created specifically for the interactive medium rather than using images form older videos, filmstrips, or other sources. Yes.

The system manager can select screen display colors. Yes.

AUDIO

Multiple audio channels are used effectively for stereo, multiple languages, and / or presentation of information at multiple levels of difficulty. Stereo.

Audio quality is the highest possible for state-of-the-art hardware but program can also be used effectively on older hardware. Yes.

The program includes a sound test function. No.

Audio level can be controlled from the screen. Yes.

Main menu