Skip to main content
SpringerLink
Log in

An analysis of guidelines for expert reviewers

  • Article
  • Published:
Instructional Science Aims and scope Submit manuscript

Abstract

Forty eight sets of recommendations identified by their authors as practical heuristics for the evaluation and revision of instructional materials were reviewed and consolidated. These guidelines were extracted from professional journals, book chapters and independent publications. Initially, all items were compiled, and then sorted into three categories according to their specific focus: content (subject matter), design, and presentation. In a second sort, identical items were eliminated and semantically equivalent items were grouped together. Three independent judges performed the same tasks for reliability. The outcome is a comprehensive list of 67 items, representing all of the reviewed guidelines. This instrument could be a successful aid in identifying deficiencies of instructional materials in the areas of content, design and presentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ali, M. A. (1981). The use of positive and negative examples during instruction. Journal of Instructional Development, 5(1), 2–7.

    Google Scholar 

  • Ally, M. (1985). A team approach to computer courseware design. Educational Technology, 25(6), 29–30.

    Google Scholar 

  • American Council on Education. (1984). Courseware evaluation checklist. Washington, DC: Author.

  • Ashmore, T. M. (1984, Nov.). Evaluating CAI material for the computer. Paper presented at the Annual Meeting of the Speech Communication Association, Chicago, IL. ED 252 180.

  • AT. & T. (1986, April). Technical training materials evaluation. Paper presented at the Annual Meeting of the National Society for Performance and Instruction, San Francisco, CA.

  • Aversa, F.M. and Forman, D.C. (1978). Issues in the evaluation of educational television programs. NSPI Journal, 17(2), 16–18.

    Google Scholar 

  • Baker, E. L. and Alkin, M. C. (1973). ERIC/AVCR annual review paper: Formative evaluation of instructional development. AV Communications Review, 21(4), 389–418.

    Google Scholar 

  • Bell, M. E. (1985). The role of instructional theories in the evaluation of microcomputer courseware. Educational Technology, 25(3), 36–40.

    Google Scholar 

  • Bell, R. C. and Sullivan, J. L. F. (1981). Student preferences in typography. Programmed Learning and Educational Technology, 18(2), 57–61.

    Google Scholar 

  • Bowler, M. (1978). The making of a textbook. Learning, 6, 8–42.

    Google Scholar 

  • Bracewell, R., Bereiter, C. and Scardemalia, M. (1979). A test of two myths about revision. Paper presented at the Annual Conference of the American Educational Research Association.

  • Briggs, L. J. (1984). Comment: three types of textbooks. Educational Technology, 24(1), 33–34.

    Google Scholar 

  • Burbank, L. and Plett, D. (1986). Designing printed instructional materials. Performance and Instruction, (Oct.), 5–9.

  • Chiesi, H. L., Spilich, G. J. and Voss, J. F. (1979). Acquisition of domain-related information in relation to high and low domain knowledge. Journal of Verbal Learning and Verbal Behavior, 18, 257–273.

    Google Scholar 

  • Clement, F. J. (1983). How an average person can test a software program. Performance and Instruction Journal, 22(5), 27–28.

    Google Scholar 

  • Cowan, J. (1980). Is systematic curriculum design always feasible? Programmed Learning and Educational Technology, 17(2), 115–117.

    Google Scholar 

  • Criswell, E. L. and Sweezey, R. W. (1984). Behavioral learning theory-based computer courseware evaluation. Educational Technology, 24(11), 43–46.

    Google Scholar 

  • Cronbach, L. J. (1963). Course improvement through evaluation. Teachers' College Record, 64, 672–683.

    Google Scholar 

  • Davis, J. E. and Hawke, S. D. (1986). Boards and publishers should collaborate on the design of curriculum materials. American School Board Journal, 183(1), 35–36 and 40.

    Google Scholar 

  • Debert, R. (1979). Revisionary tactics. NSPI Journal, 18(7), 18–21.

    Google Scholar 

  • Dick, W. (1977). Formative evaluation. In L. J., Briggs (Ed.), Instructional design: principles and applications (pp. 311–333). Englewood Cliffs, N.J.: Educational Technology.

    Google Scholar 

  • Dick, W. (1979, March). Current research and issues in formative evaluation. Paper presented at the Annual Conference of the Association of Educational Communications Technology, New Orleans, LA.

  • Drotter, M. T. and Roth, G. L. (1984). Revising poor software documentation: guidelines for modification in the classroom. Educational Technology, 24(12), 47–49.

    Google Scholar 

  • Duchastel, P. C. (1983). Text illustrations. Performance and Instruction Journal, 22(4), 3–5.

    Google Scholar 

  • Duchastel, P. C. and Whitehead, D. (1980). Exploring student reactions to inserted questions in text. Programmed Learning and Educational Technology, 17(1), 41–47.

    Google Scholar 

  • Dupont, D. and Stolovitch, H. D. (1983). The effects of a systematic revision model on revisers in terms of student outcomes. NSPI Journal, 22(2), 33–37.

    Google Scholar 

  • Duquette, C. (1985). Formative evaluation of courseware: one instrument. Educational Technology, 25(2), 20–23.

    Google Scholar 

  • Eash, M. J. (1969). Assessing curriculum materials: a preliminary instrument. Educational Product Report, 2(5), 18–22.

    Google Scholar 

  • Eiss, A. F. (1970). Evaluation of instructional systems. New York: Gordon and Breach.

    Google Scholar 

  • English, R. (1986). Can social studies textbooks have scholarly integrity? Social Education, 50(1), 46–48.

    Google Scholar 

  • Faulkner, L. and Hinds, M. (1981). Planning and perception: principles for designing the printed page. NSPI Journal, 20(2), 26–28.

    Google Scholar 

  • Faust, S.M. (1980). Instructional developer as content specialist: three case studies utilizing the instructional development operations research model. Educational Technology, 20(9), 5–12.

    Google Scholar 

  • Fetter, W.R. (1978). An evaluation instrument for instructional material. Educational Technology, 18 (10), 55–56.

    Google Scholar 

  • Foshay, W.R. (1984). QA and QC: a training vendor's vision of the formative/summative evaluation distinction. Performance and Instruction Journal, 23(10), 15–17.

    Google Scholar 

  • Frase, L. E. De, Gracie, J. S. and Poston, W. K.Jr. (1974). Product validation: pilot test or panel review? Educational Technology, 14(8), 32–35.

    Google Scholar 

  • Geis, G. L. (1986, April). Student input in materials development. Paper presented at the Annual Conference of the American Educational Research Association, San Francisco, CA.

  • Geis, G. L. (1987). Formative evaluation: developmental testing and expert review. Performance and Instruction Journal, 26(4), 1–8.

    Google Scholar 

  • Geis, G.L., Burt, C. and Weston, C.B. (1984). Instructional development: developmental testing. Paper presented at the Annual Conference of the American Educational Research Association, New Orleans, LA.

  • Glass, G.V. (1972). Educational product evaluation. Educational Researcher, 1(1), 7–10 and 16.

    Google Scholar 

  • Golas, K.C. (1983). The effectiveness and cost of alternate models of formative evaluation for printed instructional materials. (Doctoral dissertation, Florida State University, 1982). Dissertation Abstracts International, 43, 2873-A.

  • Goodman, R. I. (1984). Evaluation scripts for instructional media programs: a structured script evaluation procedure. Educational Technology, 24(3), 25–30.

    Google Scholar 

  • Gropper, G. L. (1975). Diagnosis and revision in the development of instructional materials. Englewood Cliffs, N.J.: Educational Technology.

    Google Scholar 

  • Hartley, J. (1978). Designing instructional text. New York: Nichols.

    Google Scholar 

  • Hartley, J. (1981). Eighty ways of improving instructional text. IEEE Transactions on Professional Communication, PC 24(1), 17–27.

    Google Scholar 

  • Hartley, J. and Burnhill, P. (1977). Fifty guidelines for improving instructional text. Programmed Learning and Educational Technology, 14, 65–73.

    Google Scholar 

  • Hartley, J. and Rooum, D. (1983). Sir Cyril Burt and typography: a re-evaluation. British Journal of Psychology, 74, 203–212.

    Google Scholar 

  • Hartley, J., Trueman, M. and Burnhill, P. (1980). Some observations on producing and measuring readable writing. Programmed Learning and Educational Technology, 17(3), 164–174.

    Google Scholar 

  • Hayes, J. (1986). If it's clear to me, it must be clear to them. Presentation at the Annual CCC Meeting, New Orleans, LA.

  • Henderson, E. S. and Nathenson, M. B. (1976). Developmental testing: an empirical approach to course improvement. Programmed Learning and Educational Technology, 13(4), 31–42.

    Google Scholar 

  • Indermill, K. (1986). What to do if you've got SME sclerosis. Performance and Instruction Journal, 25(2), 14–15 and 19.

    Google Scholar 

  • Jereb, B. (1986). Plain English on the plant floor. Visible Language, XX(2), 219–225.

    Google Scholar 

  • Kandaswamy, S. (1980). Sequential model for appraising instructional superiority of revised material. Educational Communication and Technology Journal, 28(3), 186–193.

    Google Scholar 

  • Kandaswamy, S., Stolovitch, H. D. and Thiagarajan, S. (1976). Learner verification and revision: an experimental comparison of two methods. AV Communication Review, 24(3), 316–338.

    Google Scholar 

  • Kinross, R. (1979). Review of Designing Instructional Text. Instructional Science, 8, 275–289.

    Google Scholar 

  • Klare, G. R. (1976). Judging readability. Instructional Science, 5, 55–61.

    Google Scholar 

  • Kline, M. R. (1984). Social influences in textbook publishing. Educational Forum, 48, Winter, 223–234.

    Google Scholar 

  • Komoski, P.K. (1971). Testimony. Washington, D.C.: U.S. House of Representatives, House Select Committee on Education and Labor.

    Google Scholar 

  • Komoski, P. K. and Woodward, A. (1985). The continuing need for the learner verification and revision of textual material. In D. H., Jonassen (Ed.), The technology of text. Englewood Cliffs, N.J.: Educational Technology.

    Google Scholar 

  • Lawson, P. (1986). Effective instructional materials are more than content. Performance and Instruction, Oct, 3–4.

  • MacDonald-Ross, M. (1978). Language in texts. In L. S. Schulman (Ed.), Review of Research in Education, 6, 229–275.

  • Markle, S. M. (1979). Evaluating instructional programs: how much is enough? NSPI Journal, 15(2), 22–24.

    Google Scholar 

  • Martinetz, C. F. (1986). A checklist for course evaluation. Performance and Instruction Journal, 25 (5), 12–19.

    Google Scholar 

  • Maxwell, J. (1985). The future textbooks — can they help individualize education? NASSP, 69, May, 68–74.

    Google Scholar 

  • McCutcheon, G. (1982). Bait/Rebate. English Journal, 71(7), 16 and 19.

    Google Scholar 

  • McLaughlin, J. A. and Trlica, J. S. (1976). Teacher evaluation of instructional material. Educational Technology, 16(3), 51–54.

    Google Scholar 

  • Merrill, M. D. (1979). Elaboration theory and cognitive theory. Paper presented at the Annual Convention of the Association for Educational Communications and Technology, New Orleans, LA.

  • Merrill, M. D., Richards, R. E., Schmidt, R. V. and Wood, N. D. (1979). The instructional strategy diagnostic profile training manual. San Diego, CA.: Courseware, Inc.

    Google Scholar 

  • Nathenson, M.B. and Henderson, E.S. (1977). Problems and issues in developmental testing. NSPI Journal, 16(1), 9–10.

    Google Scholar 

  • Nevo, D. (1977). A model for the utilization of formative evaluation in the process of developing instructional materials. Programmed Learning and Educational Technology, 14(2), 127–133.

    Google Scholar 

  • Nickols, F. W. (1981). Generalist to specialist. Whom do I consult? NSPI Journal, XX(8), 23–24.

    Google Scholar 

  • Niedermeyer, F. C. (1976). Developing classroom based instructional products: an evolving set of guidelines. Educational Technology, 16(16), 55–57.

    Google Scholar 

  • O'Donnell, H. (1985). Improving textbooks-who is responsible? Journal of Reading, 29(3), 268–270.

    Google Scholar 

  • Osborn, J. H., Jones, B. F. and Stein, M. (1985). The case for improving textbooks. Educational Leadership, 42(7), 9–16.

    Google Scholar 

  • Parsons, J., Treat, K., Burnette, D., Foster, B. and Stockert, T.C. (1976). Criteria for selecting, evaluating or developing learning modules. Educational Technology, 16(2), 31–32.

    Google Scholar 

  • Pflieger, J.L., Chomienne, M., Bordeleau, P. and Stolovitch, H.D. (1979). Enquête sur 095 l'éducation des documents audiovisuels dans la Province de Québec (1978–79). GERDAVE, rapport no. 8. Montréal: Université de Montréal, Section Technologie Educationelle.

    Google Scholar 

  • Popham, W.J. (1969). Curriculum material. Review of Educational Research, 39(3), 319–338.

    Google Scholar 

  • Poulton, E.C. (1969). How efficient is print? New Society, 349, 869–871.

    Google Scholar 

  • Reigeluth, C.M. and Darwazeh, A. (1982). The elaboration theory's procedure for designing instruction. Journal of Instructional Development, 5(3), 22–32.

    Google Scholar 

  • Roberts, W.K. (1979). Some considerations for evaluating instruction using the analytical approach. Educational Technology, 19(2), 39–41.

    Google Scholar 

  • Robinson, T. J. (1972). Replicable training in revision techniques. (Doctoral dissertation, University of California, Los Angeles, 1972). Dissertation Abstracts International, 33, 1573-A.

  • Roblyer, M. D. (1981). When is it “Good courseware”? Problems in developing standards for micro-computer courseware. Educational Technology, 21(10), 47–54.

    Google Scholar 

  • Rogers, M. A. and Volpe, S. (1984). Evaluation of vendor courses. Performance and Instruction Journal, 23(10), 18–22.

    Google Scholar 

  • Rosenberg, M. J. and Honig, P. K. (1979). Developing effective instructor's manuals to accompany audio-visual materials. Educational Technology, 19(1), 35–36.

    Google Scholar 

  • Ross, P. C. (1976). Exchanging ideas on evaluation: a model for evaluating off-the-shelf training. NSPI Journal, XV(10), 4–5.

    Google Scholar 

  • Sari, F. and Reigeluth, C. M. (1982). Writing and evaluating textbooks: contributions from instructional theory. In D. M., Jonassen (Ed.), The technology of text: principles for structuring, designing, and displaying text (pp. 53–90). Englewood Cliffs, N.J.: Educational Technology.

    Google Scholar 

  • Scriven, M. (1967). The methodology of evaluation. Monograph of the American Educational Research Association on Curriculum Evaluation, 1. Chicago: Rand McNally.

    Google Scholar 

  • Sommers, N. (1980). Revision strategies of student writers and experienced adult writers. College Composition and Communication, 31(4), 378–388.

    Google Scholar 

  • Stakenaes, R. G. and Mayer, H. (1983). Formative evaluation in vocational education. Performance and Instruction Journal, 22(5), 23–26.

    Google Scholar 

  • Steinberg, E. R. (1985). A program for improving documentation (Tech. Rep. No. 6). Pittsburgh: Camegie-Mellon University, Communications Design Center.

    Google Scholar 

  • Stevens, W. W.Jr. and Morrissett, I. (1967–68). Apsystem for analyzing social science curricula. Educational Product Report, 1 (4–5), 10–15.

    Google Scholar 

  • Stolovitch, H. D. (1982). Application of the intermediate technology of learner verification and revision (LVR) for adapting international instructional resources to meet local needs. NSPI Journal, XXI(7), 16–22.

    Google Scholar 

  • Stolovitch, H. D. (1976). How to produce audiovisual modules: evaluating your product. NSPI Journal, XV(4), 3–5.

    Google Scholar 

  • Thomas, D. (1984). A high school evaluates software (with an evaluation form). Educational Technology, 24(9), 21–23.

    Google Scholar 

  • Tillema, H. (1982). Sequencing of text materials in relation to information processing strategies. British Journal of Educational Psychology, 52, 170–178.

    Google Scholar 

  • Tillman, M. (1985). Job aids in text design. Performance and Instruction Journal, 24(1), 24–25.

    Google Scholar 

  • Truett, C. (1984). Field testing educational software: are publishers making the effort?. Educational Technology, 24(5), 7–12.

    Google Scholar 

  • Tulley, M. A. (1985). A descriptive study of the intents of state-level textbook adoption procedures. Educational Evaluation and Policy Analysis, 7(3), 289–308.

    Google Scholar 

  • Tyler, R. (1949). Basic principles of curriculum and instruction. Chicago: University of Chicago.

    Google Scholar 

  • Wade, T. E.Jr. (1980). Evaluating computer instructional programs and other teaching units. Educational Technology, 20(11), 32–35.

    Google Scholar 

  • Weston, C. B. (1986). Formative evaluation of instructional materials: an overview of approaches. Canadian Journal of Educational Communications, 15(1), 5–11.

    Google Scholar 

  • Wright, P. (1977). Presenting technical information: a survey of research findings. Instructional Science, 6, 93–134.

    Google Scholar 

  • Wright, P. and Barnard, P. (1975). Just fill in this form: a review for designers. Applied Ergonomics, 6(4), 213–220.

    Google Scholar 

  • Yelon, S. L. (1974). Constructive evaluation. Unpublished manuscript.

Download references

Author information

Authors and Affiliations

  1. Department of Educational Psychology, McGill University, 3700 McTavish Street, Montreal, Quebec, Canada

    Alenoush Saroyan

  2. Institute for Studies in Education, Ontario, Canada

    George L. Geis

Authors
  1. Alenoush Saroyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George L. Geis
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Saroyan, A., Geis, G.L. An analysis of guidelines for expert reviewers. Instr Sci 17, 101–128 (1988). https://doi.org/10.1007/BF00052698

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00052698

Keywords

Search

Navigation