Skip to main content

Advertisement

SpringerLink
Log in

Intraclass correlation metrics for the accuracy of algorithmic definitions in a computerized decision support system for supportive cancer care

  • Short Communication
  • Published:
Supportive Care in Cancer Aims and scope Submit manuscript

Abstract

As part of the development of a computerized clinical decision support system for anemia management in cancer patients, we applied psychometric principles and techniques to assess the accuracy of the algorithmic operationalizations of a set of evidence-based practice guidelines. In an iterative rating process, five medical and nursing experts rated 27 algorithmic sets derived from 18 guidelines, the objective being an intraclass coefficient (ICC) exceeding 0.90. The first round of review yielded an ICC of 1.00 for 22 sets. After revision and resubmission to the expert panel, an ICC of 1.00 was obtained for the additional five sets. The evolving decision support system is based on algorithms that accurately specify evidence-based guidelines for anemia management in cancer patients.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk I et al (2003) Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 10:523–530

    Article  PubMed  Google Scholar 

  2. Bokemeyer C, Aapro MS, Courdi A, Foubert J, Link H, Ősterborg A et al (2004) EORTC guidelines for the use of erythropoietic proteins in anaemic patients with cancer. Eur J Cancer 40:2201–2216

    Article  PubMed  CAS  Google Scholar 

  3. Bokemeyer C, Aapro MS, Courdi A, Foubert J, Link H, Ősterborg A et al (2006) EORTC guidelines for the use of erythropoietic proteins in anaemic patients with cancer: 2006 update. Eur J Cancer 43(2):258–270

    Article  PubMed  Google Scholar 

  4. Cella D (1998) Factors influencing quality of life in cancer patients: anemia and fatigue. Sem Oncol 25(3 Suppl 7):43–46

    CAS  Google Scholar 

  5. Eddy DM (2005) Evidence-based medicine: a unified approach. Health Aff 24:9–17

    Article  Google Scholar 

  6. Kawamoto K, Houlihan CA, Balas EA, Lobach DF (2005) Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. Br Med J 330:775–782

    Article  Google Scholar 

  7. Ludwig H, Van Belle S, Barrett-Lee P, Birgegard G, Bokemeyer C, Gascon P et al (2004) The European Cancer Anaemia Study (ECAS): a large, multinational, prospective study defining the prevalence, incidence, and treatment of anaemia in cancer patients. Eur J Cancer 40:2293–2306

    Article  PubMed  Google Scholar 

  8. Nunnally JC, Bernstein I (1994) Psychometric theory, 3rd ed. McGraw-Hill, New York

    Google Scholar 

  9. Park HM, Jung HW (2003) Evaluating interrater agreement with intraclass correlation coefficient in SPICE-based software process assessment. In: IEEE proceedings of the third international conference on quality software. IEEE, Washington, DC

    Google Scholar 

  10. Rizzo JD, Lichtin AE, Woolf SH, Seidenfeld J, Bennett CL, Cella D et al (2002) Use of epoetin in patients with cancer: evidence-based clinical practice guidelines of the American Society of Clinical Oncology and the American Society of Hematology. J Clin Oncol 20:4083–4107

    Article  PubMed  CAS  Google Scholar 

  11. Shiffman RN, Brandt CA, Liaw Y, Corb GJ (1999) A design model for computer-based guideline implementation based on information management services. J Am Med Inform Assoc 6:99–103

    PubMed  CAS  Google Scholar 

  12. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H et al (2001) Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 8:527–534

    PubMed  CAS  Google Scholar 

  13. Timmermans S, Mauck A (2005) The promises and pitfalls of evidence-based medicine. Health Aff 24:18–28

    Article  Google Scholar 

Download references

Acknowledgment

Supported by a research contract from F. Hoffmann-La Roche (Basel, Switzerland). The authors thank Matthew Abraham for the independent editorial and proofreading support.

Disclosures

Section I: Contributions to the project

Project concept—methodological aspects and study paradigm: I. Abraham, K. MacDonald, M. Turner, M. Muenzberg

Project concept—application to oncology: M. Aapro, P. Soubeyran, J. Foubert, C. Bokemeyer, J. Van Erps, M. Turner, M. Muenzberg

Project concept—application to evidence-based medicine: M. Aapro, P. Soubeyran, J. Foubert, C. Bokemeyer, J. Van Erps, M. Turner, M. Muenzberg, K. MacDonald, I. Abraham

Study design: I. Abraham, M. Aapro, K. MacDonald, M. Turner

Supervision of study implementation: K. MacDonald, I. Abraham, M. Turner, M. Muenzberg

Quality assurance: K. MacDonald, I. Abraham

Statistical plan: I. Abraham

Manuscript preparation: I. Abraham

Critical review of manuscript for intellectual content: M. Aapro, J. Van Erps, P. Soubeyran, J. Foubert, C. Bokemeyer, M. Turner, M. Muenzberg, K. MacDonald, I. Abraham

Section II: Conflict of interest

M. Aapro has consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies: Roche, Amgen, and Novartis. He declares no conflict with regard to the work described in this manuscript.

Abraham and K. MacDonald have consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies and, as applicable, their subsidiaries: Novartis, Johnson & Johnson (including Centocor, Ortho-Biotech, Janssen Pharmaceutica, Janssen-Cilag, and Janssen-Ortho), Eli Lilly, Roche, Pfizer, Amgen, Merck, Bristol-Myers Squibb, Schering-Plough, Astra-Zeneca, Bayer, GlaxoSmithKline, Lundbeck, and Innogenetics (including Xcellentis). Matrix45 has been contracted by sponsor to provide support with project conceptualization, project design, protocol development, development of project materials, training, project management and implementation, development of statistical plan, and quality assurance. Per company policy, I. Abraham and K. MacDonald are barred from holding equity in any client companies and are subjected to internal and external review of their work to assure objectivity and transparency. They have taken the necessary steps to assure independence and do not declare a conflict of interest with regard to the work described in this manuscript.

P. Soubeyran has consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies: Roche, Amgen, Johnson & Johnson, Sanofi-Aventis, Schering AG, Schering-Plough, Pfizer, Chugai, and Baxter Oncology. He declares no conflict with regard to the work described in this manuscript.

J. Foubert has consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies: Amgen. Johnson & Johnson, Novartis, Roche, and Merck. He declares no conflict with regard to the work described in this manuscript.

C. Bokemeyer has consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies: Roche, Amgen, and Johnson & Johnson. He declares no conflict with regard to the work described in this manuscript.

J. Van Erps has consulted with, received research grants and contracts from, and/or served as a sponsored speaker for the following companies: Roche and Novartis. She declares no conflict with regard to the work described in this manuscript.

M. Turner and M. Muenzberg are employees of F. Hoffmann–La Roche and its subsidiaries. They have refrained from undue influence throughout the project and manuscript preparation.

Section III: Sponsor and role of sponsor

Sponsor: F. Hoffmann–La Roche AG (Basel, Switzerland) and its subsidiaries provide financial support for the project. The sponsor has also committed internal resources to support the project.

Role of the sponsor: The study paradigm was developed independently and proposed to sponsor by I. Abraham and K. MacDonald. The application to cancer-related anemia was identified by the sponsor. The independent oncology experts (M. Aapro, J. Foubert, P. Soubeyran, C. Bokemeyer) were proposed by the sponsor. Collectively, all authors constituted the project team; employees of F. Hoffmann–La Roche AG and its subsidiaries listed as authors participated as members of the team. The manuscript was developed by members of the research team not affiliated with the sponsor. The sponsor had right of review and comment. All final decisions regarding the content of the manuscript were made by the external members of the team. See also Section I supra.

Section IV: Role of the medical writer or editor

No medical writer or editor was retained in the preparation of this manuscript. M. Abraham provided proofreading support and was funded under the manuscript preparation subcontract by the sponsor to Matrix45.

Author information

Authors and Affiliations

  1. Institut Multidisciplinaire d’Oncology, Clinique de Genolier, 1 Route du Muids, CH-1272, Genolier, Switzerland

    Matti Aapro

  2. Matrix45, 620 Frays Ridge Road, Earlysville, VA, 22936, USA

    Ivo Abraham & Karen MacDonald

  3. Center for Health Outcomes and Policy Research, Leonard Davis Institute of Health Economics and Institute of Aging, University of Pennsylvania, 418 Curie Boulevard, Philadelphia, PA, 19104, USA

    Ivo Abraham

  4. Institut Bergonié, 229, cours de l’Argonne and Université Victor Segalen Bordeaux 2, 146 rue Léo Saignat, 33076, Bordeaux, France

    Pierre Soubeyran

  5. Erasmushogeschool, Laarbeeklaan 101, B-1090, Brussel, Belgium

    Jan Foubert

  6. Universitätsklinikum Hamburg Eppendorf, Martinistraße 52, 20246, Hamburg, Germany

    Carsten Bokemeyer

  7. F. Hoffmann–La Roche AG, Grenzacherstrasse, Bau 74, CH-4070, Basel, Switzerland

    Michael Muenzberg

  8. Afdeling Oncologie en Hematologie, Algemeen Stedelijk Ziekenhuis Aalst, Merestraat 80, B-9300, Aalst, Belgium

    Joanna Van Erps

  9. Roche Products, 6 Falcon Way, Shire Park, Welwyn Garden City, AL7 1TW, Hertfordshire, UK

    Matthew Turner

Authors
  1. Matti Aapro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivo Abraham
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karen MacDonald
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pierre Soubeyran
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jan Foubert
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carsten Bokemeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Muenzberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Joanna Van Erps
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Matthew Turner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivo Abraham.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Aapro, M., Abraham, I., MacDonald, K. et al. Intraclass correlation metrics for the accuracy of algorithmic definitions in a computerized decision support system for supportive cancer care. Support Care Cancer 15, 1325–1329 (2007). https://doi.org/10.1007/s00520-007-0246-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00520-007-0246-7

Keywords

Search

Navigation