Skip to main content
SpringerLink
Log in

Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1–3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach’s alpha for this single factor (i.e. all six mini-CEX items) was ≥0.86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Brennan, R. L. (2001). Generalizability theory. New York: Springer.

    Google Scholar 

  • Cook, D. A., & Beckman, T. J. (2009). Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Advances in Health Sciences Education: Theory and Practice, 14, 655–664.

    Article  Google Scholar 

  • Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009). Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine, 24, 74–79.

    Article  Google Scholar 

  • Donato, A. A., Pangaro, L., Smith, C., Rencic, J., Diaz, Y., Mensinger, J., et al. (2008). Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Medical Education, 42, 1234–1242.

    Article  Google Scholar 

  • Durning, S. J., Cation, L. J., Markert, R. J., & Pangaro, L. N. (2002). Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Academic Medicine, 77, 900–904.

    Article  Google Scholar 

  • Fernando, N., Cleland, J., McKenzie, H., & Cassar, K. (2008). Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Medical Education, 42, 89–95.

    Google Scholar 

  • Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.

    Article  Google Scholar 

  • Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Greenburg, D. L., Durning, S. J., Cohen, D. L., Cruess, D., & Jackson, J. L. (2007). Identifying medical students likely to exhibit poor professionalism and knowledge during internship. Journal of General Internal Medicine, 22, 1711–1717.

    Article  Google Scholar 

  • Haber, R. J., & Avins, A. L. (1994). Do ratings on the American Board of internal medicine resident evaluation form detect differences in clinical competence? Journal of General Internal Medicine, 9, 140–145.

    Article  Google Scholar 

  • Harvill, L. M. (1991). NCME instructional module: Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33–41.

    Article  Google Scholar 

  • Hatala, R., Ainslie, M., Kassen, B. O., Mackie, I., & Roberts, J. M. (2006). Assessing the mini-clinical evaluation exercise in comparison to a national specialty examination. Medical Education, 40, 950–956.

    Article  Google Scholar 

  • Herbers, J. E., Jr., Noel, G. L., Cooper, G. S., Harvey, J., Pangaro, L. N., & Weaver, M. J. (1989). How accurate are faculty evaluations of clinical competence? Journal of General Internal Medicine, 4, 202–208.

    Article  Google Scholar 

  • Hill, F., Kendall, K., Galbraith, K., & Crossley, J. (2009). Implementing the undergraduate mini-CEX: A tailored approach at Southampton University. Medical Education, 43, 326–334.

    Article  Google Scholar 

  • Hojat, M., Paskin, D. L., Callahan, C. A., Nasca, T. J., Louis, D. Z., Veloski, J., et al. (2007). Components of postgraduate competence: analyses of thirty years of longitudinal data. Medical Education, 41, 982–989.

    Article  Google Scholar 

  • Holmboe, E., Fiebach, N., Galaty, L., & Huot, S. (2001). Effectiveness of a focused educational intervention on resident evaluations from faculty. Journal of General Internal Medicine, 16, 427–434.

    Article  Google Scholar 

  • Holmboe, E. S., Huot, S., Chung, J., Norcini, J., & Hawkins, R. E. (2003). Construct validity of the miniclinical evaluation exercise (miniCEX). Academic Medicine, 78, 826–830.

    Article  Google Scholar 

  • Kogan, J. R., Bellini, L. M., & Shea, J. A. (2003). Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Academic Medicine, 78(10 Suppl), S33–S35.

    Article  Google Scholar 

  • Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, 302, 1316–1326.

    Article  Google Scholar 

  • Kroboth, F. J., Hanusa, B. H., & Parker, S. C. (1996). Didactic value of the clinical evaluation exercise. Missed opportunities. Journal of General Internal Medicine, 11, 551–553.

    Article  Google Scholar 

  • Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine, 81(10 Suppl), S56–S60.

    Article  Google Scholar 

  • Nasca, T. J., Gonnella, J. S., Hojat, M., Veloski, J., Erdmann, J. B., Robeson, M., et al. (2002). Conceptualization and measurement of clinical competence of residents: A brief rating form and its psychometric properties. Medical Teacher, 24, 299–303.

    Article  Google Scholar 

  • Ney, E. M., Shea, J. A., & Kogan, J. R. (2009). Predictive validity of the mini-clinical evaluation exercise (mCEX): Do medical students’ mCEX ratings correlate with future clinical exam performance? Academic Medicine, 84(10 suppl), S21–S24.

    Google Scholar 

  • Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123, 795–799.

    Google Scholar 

  • Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138, 476–481.

    Google Scholar 

  • O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, & Computers, 32, 396–402.

    Google Scholar 

  • Shavelson, R. L., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park: Sage Publications.

    Google Scholar 

  • Sidhu, R. S., Hatala, R., Barron, S., Broudo, M., Pachev, G., & Page, G. (2009). Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians. Academic Medicine, 84(10 suppl), S113–S115.

    Article  Google Scholar 

  • Silber, C. G., Nasca, T. J., Paskin, D. L., Eiger, G., Robeson, M., & Veloski, J. J. (2004). Do global rating forms enable program directors to assess the ACGME competencies? Academic Medicine, 79, 549–556.

    Article  Google Scholar 

  • Srinivasan, M., Hauer, K. E., Der-Martirosian, C., Wilkes, M., & Gesundheit, N. (2007). Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination. Medical Education, 41, 857–865.

    Article  Google Scholar 

  • Thomas, P. A., Gebo, K. A., & Hellmann, D. B. (1999). A pilot study of peer review in residency training. Journal of General Internal Medicine, 14, 551–554.

    Article  Google Scholar 

  • Volkan, K., Simon, S. R., Baker, H., & Todres, I. D. (2004). Psychometric structure of a comprehensive objective structured clinical examination: A factor analytic approach. Advances in Health Sciences Education: Theory and Practice, 9, 83–92.

    Article  Google Scholar 

  • Weller, J. M., Jolly, B., Misur, M. P., Merry, A. F., Jones, A., Crossley, J. G. M., et al. (2009). Mini-clinical evaluation exercise in anaesthesia training. British Journal of Anaesthesia, 102, 633–641.

    Article  Google Scholar 

  • Wilkinson, T. J., & Frampton, C. M. (2003). Assessing performance in final year medical students. Can a postgraduate measure be used in an undergraduate setting? Medical Education, 37, 233–240.

    Article  Google Scholar 

Download references

Acknowledgments

Financial support

No external funding.

Authorship

All authors were involved in the planning and execution of this study and in the drafting and revising of this manuscript.

Ethical approval

Judged exempt by our Institutional Review Board.

Conflicts of interest statement

The authors have no affiliation with an organization with a financial interest in the subject matter, and are not aware of any conflicts of interest.

Author information

Authors and Affiliations

  1. Division of General Internal Medicine and Office of Education Research, Mayo Clinic College of Medicine, Baldwin 4-A, 200 First Street SW, Rochester, MN, 55905, USA

    David A. Cook & Thomas J. Beckman

  2. Division of Biomedical Statistics and Informatics, Mayo Clinic College of Medicine, 200 First Street SW, Rochester, MN, 55905, USA

    Jayawant N. Mandrekar & V. Shane Pankratz

Authors
  1. David A. Cook
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas J. Beckman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jayawant N. Mandrekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. V. Shane Pankratz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David A. Cook.

Appendix

Appendix

Part 1. Factor analysis accounting for multiple observations on each resident

Code is for SAS 9.1. Original dataset for the post-workshop data is “cex_post” with data columns:

  • fac_id (unique preceptor identification code),

  • res_id (unique resident identification code),

  • rep_id (unique encounter identifier for preceptor-resident pairs with more than one observation; if preceptor A observed resident B three times, rep_id values would be 1 for the first encounter, 2 for the second, and 3 for the third)

  • ratings for each mini-CEX domain: couns, ex, human, hx, judg, org.

Step 1. Create an adjusted correlation matrix using mixed linear models with repeated measures on preceptors and residents.

  1. (a)

    First reformat the dataset for proc mixed:

  2. (b)

    Then create the adjusted correlation matrix. Note that you must be careful to determine the order of the variables in the resultant matrix (the item labels are not part of the matrix output).

  3. (c)

    The results of this analysis will appear as output on the screen. Again, it is essential to correctly identify which variable matches with each column in this matrix (column order is the same as the order of the variables in the parent dataset, in this case “cex_mixed”).

Step 2. Create a correlation data set for subsequent analysis

The values derived above can be manually used to create a data set of type “CORR” for subsequent analysis as shown below. Means and standard deviations can be determined using proc means. We estimated N (effective sample size) from the number of discrete preceptor-resident pairs (which can be found by counting the number of observations with rep_id = 1).

Step 3. Perform factor analysis on this adjusted correlation matrix:

Part 2. Additional tables: principal components factor analysis and adjusted/unadjusted correlation matrices

Table 5 Principal components factor analysis of mini-CEX scores (Set 2)
Full size table
Table 6 Factor loading for mini-CEX domains following principal components analysis (Set 2)
Full size table
Table 7 Comparison of correlation matrices for adjusted and unadjusted factor analysis
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cook, D.A., Beckman, T.J., Mandrekar, J.N. et al. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv in Health Sci Educ 15, 633–645 (2010). https://doi.org/10.1007/s10459-010-9224-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-010-9224-9

Keywords

Search

Navigation