564-2010-The-List

(Rubric version, 22 July 2010)

Providing Feedback

 * 1) After each response, feedback is immediate.
 * 2) Includes knowledge-of-correct-response feedback (e.g., including response accuracy verification, providing correct answers, etc.)
 * 3) Elaborative feedback is available for low certitude responses
 * 4) Periodic feedback relates tracked data to learner goals (e.g., learning/achievement is definable (either by designer or user, i.e. five correct iterations)
 * 5) Results of learning session are related to learner goals.

Scheduling Sequence and Spacing

 * 1) Sequencing reflects a scheduled framework (e.g., Leitner system)
 * 2) Presentation of each item is discrete and spaced.
 * 3) Design provides for at least one intersession interval of anywhere between one and thirty days (no "cramming").

Motivating and Engaging

 * 1) Design captures learners' interest (e.g., use simple unexpected events like a loud whistle or an upside-down word in a visual, etc.).
 * 2) Design stimulates learners' inquiry (e.g., give mentally stimulating problems that engage a deeper level of curiosity, etc.).
 * 3) Design maintains learners' attention (e.g., utilize variation).
 * 4) Design makes learning outcomes relevant to students (e.g., connect content to learner goals, interests, learning styles, etc.).
 * 5) Design builds learner confidence (e.g., providing examples of acceptable achievement).
 * 6) Design promotes student satisfaction (e.g., provides recognition and evidence of success, practical application, etc.).

Managing Cognitive Load

 * 1) Design takes advantage of verbal (text, narration, etc.) and non-verbal (photographs, illustrations, diagrams, etc.) input channels
 * 2) Design avoids cognitive overload (e.g., text in close spatial proximity to visuals to avoid split attention cognitive load concerns).
 * 3) Design acknowledges and adapts to limitations of audience (i.e. universal design and accessibility)
 * 4) Design enables learner to efficiently "chunk" facts by identifying, connecting (grouping), and sequencing information.

Determining Prior Knowledge

 * 1) Design determines learner's prior knowledge and goals (e.g., pre-assessment, iterations of a Leitner system, etc.)
 * 2) Design facilitates open content (e.g., user-generated content, sharing of content and results, user-user or user-population comparisons of results, etc.)
 * 3) Design provides low prior-knowledge students with response-contingent feedback (e.g., system explains reasons for correct/incorrect responses)
 * 4) Design provides high prior-knowledge students with topic-contingent feedback (e.g., system directs learners to find the correct response or a path to additional information).

Maximizing Academic Learning Time (ALT)

 * 1) Design ensures all instructional activities support desired learning outcomes.
 * 2) Design ensures waiting and transitional time is minimized.

(Edited version, 20 July 2010)

Feedback

 * 1) User-response feedback is immediate.
 * 2) Includes knowledge of correct response feedback
 * 3) This includes response verification (is it right or wrong) AND providing the correct answer
 * 4) Low prior knowledge students need response-contingent feedback.
 * 5) Include why the wrong answer is wrong and why the right answer is right.
 * 6) No searching is necessary for the learner
 * 7) High prior knowledge students need topic-contingent feedback.
 * 8) Directs learners to (or provides additional) information to find the correct response.
 * 9) The learner is responsible for finding the correct answer in the information given.
 * 10) High certitude responses need quick and succinct feedback
 * 11) Low certitude responses need more elaborative feedback
 * 12) Certitude can be measured during the evaluation, dictating the type of feedback, or the learners can have the choice of how elaborative the feedback will be.
 * 13) There should be periodic feedback based on tracked data according to learner goals.
 * 14) There is a way to determine learner goals.
 * 15) Learning/achievement is defined (either by designer or user - Ex. 5 correct iterations)
 * 16) The results of the latest session is related back to the goal.

Spacing effect/lag/schedule/leitner/sequencing

 * 1) Sequencing reflects the Leitner Framework
 * 2) Design should sort flashcards into groups, showing learners items they are least familiar with more often, moving items learners are improving knowledge of into higher "boxes" which present those words/items less frequently.
 * 3) Design should present discrete information
 * 4) at least 2 spaced presentations (note, more presentations appears to increase learning)
 * 5) Design provides for a lag between presentations
 * 6) of at least 1 day.
 * 7) Instructional product contains at least 1 intersession interval of at least 1-day. Note that the ISI can be as long as 30 days.  (Should we combine?)

Motivation ARCS/engaging

 * 1) Design to keep student attention.
 * 2) Capture interest: use simple unexpected events (e.g. a loud whistle, an upside-down word in a visual)
 * 3) Stimulate inquiry: give mentally stimulating problems that engage a deeper level of curiosity
 * 4) Maintain attention: utilize variation
 * 5) Make learning outcomes relevant to students.
 * 6) Connect the content of instruction to important goals of the learners, their past interests, and their learning styles (e.g. relate instructional content to the learners’ future job or academic requirements)
 * 7) Use simulations, analogies, case studies, and examples related to the students' immediate and current interests and experiences
 * 8) The design builds student confidence.
 * 9) Make the objectives clear
 * 10) Provide examples of acceptable achievements
 * 11) If the objective is to have a coherent essay, have a coherent essay available for them to see.
 * 12) Make instruction moderately challenging (communicate high expectations)
 * 13) Design promotes student satisfaction.
 * 14) Give students recognition and evidence of success (they can consist of grades, privileges, promotions or such things as certificates, monogrammed school supplies, or other tokens of achievement)
 * 15) Provide opportunities for students to apply what they have learned
 * 16) Make design fair (amount of work required by the course was appropriate, that there was internal consistency between objectives, content, and tests, and that there was no favoritism in grading)

Managing cognitive load

 * 1) The design takes advantage of multiple sensory input channels, specifically visual (photographs, illustrations, diagrams, etc.) and non-visual (text, narration, etc.) channels.
 * 2) For the most efficacious cognition, use visual images in conjunction with narrated (spoken) non-visual/linguistic information
 * 3) If narration is not being used, keep non-visual input (read: caption or textual information) in close spatial proximity to visuals to avoid split attention cognitive load concerns.
 * 4) Evaluate the use of visual and non-visual input in cases where accessibility is impeded
 * 5) Know limitations of audience and act accordingly (not slaves to the theoretical design)
 * 6) The design enables the efficient chunking of facts by identifying, connecting, and sequencing the information.
 * 7) Create opportunities for the learner to identify salient and distinct chunks of information.
 * 8) Create opportunities for the learner to create relevant connections between chunks of information.
 * 9) Create opportunities for the learner to sequence chunks of information.

Program is Analysis-Driven and Considers Learners' Prior Knowledge

 * 1) Determine learner's prior knowledge.
 * 2) Pre-assess learner's prior knowledge to determine topics and items.  (of topics and items)
 * 3) Pre-assessment may be the first iteration of Leitner system. Prior knowledge can inform second iteration of Leitner system repetitions.  (needed)
 * 4) Facilitates open content:
 * 5) Allows for user-generated content.
 * 6) Allows user sharing of content.
 * 7) Allows user sharing of results.
 * 8) Allows user-user or user-population comparisons of results.
 * 9) Conduct audience analysis and use feedback in design to ensure (ADDIE)
 * 10) objectives meet learner needs,
 * 11) content complements learner prior knowledge
 * 12)  design meets learner abilities/preferences.

Maximize ALT
(Does this section apply to a comp-based program?)
 * 1) Make sure all instructional activities support desired learning outcomes.
 * 2) Make sure there is little to no waiting time
 * 3) e.g. waiting to receive new assignments, waiting for work to be checked, waiting for teacher attention, etc.
 * 4) Keep transitional time short
 * 5) e.g. teacher hands back work, students take supplies out and put them away, etc.
 * 6) Match curriculum to students
 * 7) e.g. take into account individual differences between students, make instruction relevant to students, etc. (repeated?)
 * 8) Structure lessons
 * 9) (announcing where students should be, what they should be doing, and what they are responsible for; giving directions; providing advance organizers; and so forth)
 * 10) Control the attention of the learners
 * 11) (fun, interactive, illustrative, utilize humor, variety, etc.)

(Initial version, 15 July 2010)

Feedback

 * Timing and Type of feedback
 * The user-response feedback should always be immediate. (CBI, 7)
 * "Research has shown, for example, that immediate feedback can be more effective for decision-making and novel information tasks(Jonassen & Hannum, 1987) as well as for lower level, knowledge-based tasks (Gaynor, 1981). For higher level tasks, such as abstract concepts and application/comprehension skills, delayed feedback has proven more effective (Gaynor, 1981; Jonassen & Hannum, 1987). Other research(e.g., Gaynor, 1981; Roper, 1977) indicates that concept acquisition is facilitated through immediate feedback while long-term retention is enhanced with delayed feedback (Bardwell, 1981)."
 * The program should include knowledge of correct response feedback.(CBI, 3)
 * This includes response verification (is it right or wrong) AND providing the correct answer
 * Low prior knowledge students need response contingent feedback. (CBI 3)
 * Include why the wrong answer is wrong and why the right answer is right.
 * No searching is necessary for the learner
 * High prior knowledge students need topic-contingent feedback.(CBI 3)
 * Directs learners to (or provides additional) information to find the correct response.
 * The learner is responsible for finding the correct answer in the information given.
 * High certitude responses need quick and succinct feedback
 * Low certitude responses need more elaborative feedback
 * Certitude can be measured during the evaluation, dictating the type of feedback, or the learners can have the choice of how elobarative the feedback will be.
 * "A potential criticism of extensive elaborative feedback is reduced efficiency of the instructional unit. This problem can be effectively addressed, however, by providing increase learner control over the type and elaboration of the feedback (CBI 8)."
 * There should be periodic feedback based on tracked data according to learner goals.
 * There is a way to determine learner goals.
 * Learning/achievement is defined (either by designer or user - Ex. 5 correct iterations)
 * The results of the latest session is related back to the goal.

Spacing effect/lag/schedule/leitner/sequencing
Here is a link to a summary of specific research for this section.[]
 * 1) learner control
 * 2) Product or instructional program design should sort flashcards into groups, showing learners items they are least familiar with more often, moving items learners are improving knowledge of into higher "boxes" which present those words/items less frequently. (Leitner)
 * 3) Product or instructional event design should present discrete information/facts/vocabulary words in at least 2 (More than 1 second between presentations)spaced presentations (Dempster, p. 627) (note, more presentations appears to increase learning)
 * 4) Product or Instructional event provides for a lag between presentations of at least 1 day. (Dempster, p. 628)
 * 5) Instructional product contains at least 1 intersession interval of at least 1-day. Note that the ISI can be as long as much 30 days.  (Bahrick and Phelps, 1987, as cited in Dempster, p. 629)

Motivation ARCS/engaging
(Keller, pages 2-3)
 * 1) Design to keep student attention.
 * 2) Capture interest: use simple unexpected events (e.g. a loud whistle, an upside-down word in a visual)
 * 3) Stimulate inquiry: give mentally stimulating problems that engage a deeper level of curiosity
 * 4) Maintain attention: utilize variation
 * 5) Make learning outcomes relevant to students.
 * 6) Connect the content of instruction to important goals of the learners, their past interests, and their learning styles (e.g. relate instructional content to the learners’ future job or academic requirements)
 * 7) Use simulations, analogies, case studies, and examples related to the students' immediate and current interests and experiences
 * 8) Build student confidence in design.
 * 9) Make the objectives clear
 * 10) Provide examples of acceptable achievements
 * 11) Make instruction moderately challenging (communicate high expectations - Berliner)
 * 12) Build support of student satisfaction into the design.
 * 13) Give students recognition and evidence of success (they can consist of grades, privileges, promotions or such things as certificates, monogrammed school supplies, or other tokens of achievement)
 * 14) Provide opportunities for students to apply what they have learned
 * 15) Make design fair (amount of work required by the course was appropriate, that there was internal consistency between objectives, content, and tests, and that there was no favoritism in grading)

Managing cognitive load

 * 1) The design leverages (takes advantage of?) multiple sensory input channels, specifically visual (photographs, illustrations, diagrams, etc.) and non-visual (text, narration, etc.) channels.
 * 2) For greatest efficiency, use visual images in conjunction with narrated (spoken) non-visual/linguistic information (Mayer & Moreno, 318; Mayer, 4, 17).
 * 3) If narration is not being used, keep non-visual input (read: caption or textual information) in close spatial proximity to visuals to avoid split attention cognitive load concerns. (Mayer, 17–18)
 * 4) Evaluate/Leverage the use of visual and non-visual input in cases where accessibility is impeded (Mayer & Moreno, 318)
 * 5) The design leverages/enables the efficient chunking of facts by pegging/identifying, linking/connecting, and sequencing the information. (Mayer & Moreno, 312; Miller, 92)
 * 6) Create opportunities for the learner to identify (and discern?) salient and distinct chunks of information.
 * 7) Create opportunities for the learner to identify (and create?) relevant connections between chunks of information.
 * 8) Create opportunities for the learner to (understand and create salient?) sequence(s and order of?) chunks of information.

Summative Criteria

 * 1) Determines learner's prior knowledge.
 * 2) * Pre-assesses learner's prior knowledge to determine topics and items.
 * 3) *Pre-assessment may be the first iteration of Leitner system. Prior knowledge can inform second iteration of Leitner system repetitions. (Wikipedia, Leitner system)
 * 4) Prior knowledge assessment determines 'type' of feedback.
 * 5) * (Mason & Bruning 15-16, 21) (high = topic contingent; low = response contingent; requires categorizing of learners according to pre-assessment, or granular correlation of practice items with pre-assessment items)
 * 6) Determines learner's motivation profile, learning goals, modifiable influences, etc.
 * 7) *(Keller 3-6 ARCS)

Openness

 * 1) Facilitates open content:
 * 2) * Allows for user-generated content.
 * 3) * Allows user sharing of content.
 * 4) * Allows user sharing of results.
 * 5) * Allows user-user or user-population comparisons of results.

Design Critiera

 * 1) Conduct audience analysis to ensure objectives meet learner needs, content complements learner prior knowledge, and design meets learner abilities/preferences.
 * 2) *(ADDIE)

Maximize ALT
("As defined earlier, the concept and variable of ALT is that part of allocated time during which a student is engaged with materials and activities in which a high level of success is attained, and in which the materials and activities are related to outcomes that are valued." Berliner) The above Maximize ALT list of concepts was taken from "What's All the Fuss About Instructional Time?" DAVID C. BERLINER, Arizona State University http://courses.ed.asu.edu/berliner/readings/fuss/fuss.htm
 * 1) Make sure all instructional activities support desired learning outcomes.
 * 2) Make sure there is little to no waiting time (e.g. waiting to receive new assignments, waiting for work to be checked, waiting for teacher attention, etc.).
 * 3) Keep transitional time short (e.g. teacher hands back work, students take supplies out and put them away, etc.).
 * 4) Match curriculum to students (e.g. take into account individual differences between students, make instruction relevant to students, etc.)
 * 5) Structure lessons (announcing where students should be, what they should be doing, and what they are responsible for; giving directions; providing advance organizers; and so forth)
 * 6) Control the attention of the learners (fun, interactive, illustrative, utilize humor, variety, etc.)