Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Deep learning-based classification of mesothelioma improves prediction of patient outcome

Nature Medicine volume 25, pages 1519–1525 (2019)Cite this article

Subjects

Abstract

Malignant mesothelioma (MM) is an aggressive cancer primarily diagnosed on the basis of histological criteria1. The 2015 World Health Organization classification subdivides mesothelioma tumors into three histological types: epithelioid, biphasic and sarcomatoid MM. MM is a highly complex and heterogeneous disease, rendering its diagnosis and histological typing difficult and leading to suboptimal patient care and decisions regarding treatment modalities2. Here we have developed a new approach—based on deep convolutional neural networks—called MesoNet to accurately predict the overall survival of mesothelioma patients from whole-slide digitized images, without any pathologist-provided locally annotated regions. We validated MesoNet on both an internal validation cohort from the French MESOBANK and an independent cohort from The Cancer Genome Atlas (TCGA). We also demonstrated that the model was more accurate in predicting patient survival than using current pathology practices. Furthermore, unlike classical black-box deep learning methods, MesoNet identified regions contributing to patient outcome prediction. Strikingly, we found that these regions are mainly located in the stroma and are histological features associated with inflammation, cellular diversity and vacuolization. These findings suggest that deep learning models can identify new features predictive of patient survival and potentially lead to new biomarker discoveries.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: MesoNet layout.
Fig. 2: Comparison of the performance between MesoNet and other histology-based models to predict MM patient overall survival.
Fig. 3: Extraction of extremal tiles associated with prognosis to identify regions of interest.
Fig. 4: Histological characterization and localization of tiles of interest in EMM patients with a poor prognosis.

Similar content being viewed by others

Data availability

The external validation TCGA dataset is publicly available at the TCGA portal (https://portal.gdc.cancer.gov). We provide a manifest linking to the sample IDs considered in the study (Supplementary Table 4). The MESOBANK/MESOPATH dataset that supports the findings of this study is available from the Centre Léon Bérard, but restrictions apply to its availability (used with permission for the current study), and so it is not publicly available. The data, or a test subset, may be available from the Centre Léon Bérard subject to ethical approvals.

Code availability

The code used for training the models has a large number of dependencies on internal tooling and its release is therefore not feasible. However, all experiments and implementation details are described thoroughly in the Methods so that it can be independently replicated with non-proprietary libraries.

References

  1. Galateau-Sallé, F., Churg, A., Roggli, V. & Travis, W. D. The 2015 World Health Organization classification of tumors of the pleura: advances since the 2004 classification. J. Thorac. Oncol. 11, 142–154 (2016).

    Article  Google Scholar 

  2. Galateau-Sallé, F. et al. New insights on diagnostic reproducibility of biphasic mesotheliomas: a multi-institutional evaluation by the International Mesothelioma Panel from the MESOPATH reference center. J. Thorac. Oncol. 13, 1189–1203 (2018).

    Article  Google Scholar 

  3. Noonan, C. W. Environmental asbestos exposure and mesothelioma. Ann. Transl. Med. 5, 234 (2017).

    Article  Google Scholar 

  4. Lacourt, A. et al. Dose–time-response association between occupational asbestos exposure and pleural mesothelioma. Occup. Environ. Med. 74, 691–697 (2017).

    Article  Google Scholar 

  5. Robinson, B. W. S. & Lake, R. A. Advances in malignant mesothelioma. N. Engl. J. Med. 353, 1591–1603 (2005).

    Article  CAS  Google Scholar 

  6. Yap, T. A., Aerts, J. G., Popat, S. & Fennell, D. A. Novel insights into mesothelioma biology and implications for therapy. Nat. Rev. Cancer 17, 475–488 (2017).

    Article  CAS  Google Scholar 

  7. Opitz, I. et al. A new prognostic score supporting treatment allocation for multimodality therapy for malignant pleural mesothelioma: a review of 12 years’ experience. J. Thorac. Oncol. 10, 1634–1641 (2015).

    Article  Google Scholar 

  8. Kindler, H. L. et al. Treatment of malignant pleural mesothelioma: American Society of Clinical Oncology clinical practice guideline. J. Clin. Oncol. 36, 1343–1373 (2018).

    Article  Google Scholar 

  9. Brcic, L., Vlacic, G., Quehenberger, F. & Kern, I. Reproducibility of malignant pleural mesothelioma histopathologic subtyping. Arch. Pathol. Lab. Med. 142, 747–752 (2018).

    Article  Google Scholar 

  10. Hmeljak, J. et al. Integrative molecular characterization of malignant pleural mesothelioma. Cancer Discov. 8, 1548–1565 (2018).

    Article  Google Scholar 

  11. Shrestha, R. et al. BAP1 haploinsufficiency predicts a distinct immunogenic class of malignant peritoneal mesothelioma. Genom. Med. 11, 8 (2019).

    Article  Google Scholar 

  12. Yu, K. H. et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 7, 12474 (2016).

    Article  CAS  Google Scholar 

  13. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1090–1098 (2012).

    Google Scholar 

  14. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  Google Scholar 

  15. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  CAS  Google Scholar 

  16. Hou, L. et al. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of 2016 IEEE Conference Computer Vision and Pattern Recognitition (IEEE, 2016); https://doi.org/10.1109/CVPR.2016.266

  17. Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559–1567 (2018).

    Article  CAS  Google Scholar 

  18. Mobadersany, P. et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl Acad. Sci. USA 115, 2970–2979 (2018).

    Article  Google Scholar 

  19. Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).

    Article  CAS  Google Scholar 

  20. Schaumberg, A. J. et al. Large-scale annotation of histopathology images from social media. Preprint at https://doi.org/10.1101/396663 (2018).

  21. Nagpal, K. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. Npj Digit. Med. 2, 48 (2019).

    Article  Google Scholar 

  22. Courtiol, P., Tramel, E. W., Sanselme, M. & Wainrib, G. Classification and disease localization in histopathology using only global labels: a weakly-supervised approach. Preprint at https://arxiv.org/abs/1802.02212 (2018).

  23. Zarella, M. D. et al. A practical guide to whole slide imaging. Arch. Pathol. Lab. Med. 143, 222–234 (2019).

    Article  Google Scholar 

  24. Mukhopadhyay, S. et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology. Am. J. Surg. Pathol. 42, 1 (2018).

    Google Scholar 

  25. Galateau-sallé, F. et al. [The French mesothelioma network from 1998 to 2013]. Ann. Pathol. Elsevier Masson 34, 51–63 (2014).

    Google Scholar 

  26. Baas, P. et al. Malignant pleural mesothelioma: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann. Oncol. 21, 126–169 (2015).

    Google Scholar 

  27. Kadota, K. et al. Pleomorphic epithelioid diffuse malignant pleural mesothelioma: a clinicopathological review and conceptual proposal to reclassify as biphasic or sarcomatoid mesothelioma. J. Thorac. Oncol. 6, 896–904 (2011).

    Article  Google Scholar 

  28. Junttila, M. R. & De Sauvage, F. J. Influence of tumour micro-environment heterogeneity on therapeutic response. Nature 501, 346–354 (2013).

    Article  CAS  Google Scholar 

  29. Dacic, S. et al. Prognostic significance of p16/cdkn2a loss in pleural malignant mesotheliomas. Virchows Arch. 453, 627–635 (2008).

    Article  Google Scholar 

  30. Pillai, K., Pourgholami, M. H., Chua, T. C. & Morris, D. L. Prognostic significance of Ki67 expression in malignant peritoneal mesothelioma. Am. J. Clin. Oncol. Cancer Clin. Trials 38, 388–394 (2015).

    Article  CAS  Google Scholar 

  31. Beck, A. H. et al. Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med. 3, 108ra113 (2011).

    Article  Google Scholar 

  32. Ujiie, H. et al. The tumoral and stromal immune microenvironment in malignant pleural mesothelioma: a comprehensive analysis reveals prognostic immune markers. Oncoimmunology 4, 1–9 (2015).

    Article  Google Scholar 

  33. Rosen, L. E. et al. Nuclear grade and necrosis predict prognosis in malignant epithelioid pleural mesothelioma: a multi-institutional study. Mod. Pathol. 31, 598–606 (2018).

    Article  Google Scholar 

  34. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference 234–241 (Springer, 2015); https://doi.org/10.1007/978-3-319-24574-4_28

    Google Scholar 

  35. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2016); https://doi.org/10.1109/CVPR.2016.90

  36. Wang, D., Khosla, A., Gargeya, R., Irsha, H. & Beck, A.H. Deep learning for identifying metastatic breast cancer. Preprint at https://arxiv.org/abs/1606.05718 (2016).

  37. Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B. & Wei, L. J. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat. Med. 30, 1105–1117 (2011).

    PubMed  PubMed Central  Google Scholar 

  38. Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (ACM Press, 2016); https://doi.org/10.1145/2939672.2939785

Download references

Acknowledgements

We thank the Owkin Lab for interesting discussions and the Centre Léon Bérard for providing access to the MESOBANK/MESOPATH dataset. We thank the MESOPATH consortium—I. Abd-Alsamad, H. Begueret, E. Brambilla, F. Capron, M.-C. Copin, C. Danel, A.-Y. de Lajartre, E. Foulet-Roge, F. Galateau-Sallé, L. Garbe, O. Groussard, S. Giusiano, V. Hofman, S. Lantejuoul, J.-M. Piquenot, I. Rouquette, S. Sagan, F. Thivolet-Bejui and J.-M. Vignaud—for the histological typing of the mesothelioma samples. We also thank the TCGA network for providing the external validation set (http://cancergenome.nih.gov/).

Author information

Author notes

  1. These authors contributed equally: Pierre Courtiol, Charles Maussion, Françoise Galateau-Sallé, Gilles Wainrib, Thomas Clozel.

Authors and Affiliations

  1. Owkin Lab, Owkin, Inc., New York, NY, USA

    Pierre Courtiol, Charles Maussion, Matahi Moarii, Elodie Pronier, Samuel Pilcer, Meriem Sefta, Pierre Manceron, Sylvain Toldo, Mikhail Zaslavskiy, Gilles Wainrib & Thomas Clozel

  2. Department of Biopathology, MESOPATH/MESOBANK Cancer Center Léon Bérard, Lyon, France

    Nolwenn Le Stang & Françoise Galateau-Sallé

  3. Université de Lyon, Université Claud Bernard Lyon 1, Lyon, France

    Nicolas Girard

  4. Institut du Thorax Curie-Montsouris, Institut Curie, Paris, France

    Nicolas Girard

  5. Department of Physiology and Biophysics, Institute for Computational Biomedicine and Caryl and Israel Englander Institute for Precision Medicine, WorldQuant Initiative for Quantitative Prediction, Weill Cornell Medicine, New York, NY, USA

    Olivier Elemento

  6. Department of Histopathology, Royal Brompton and Harefield Hospitals NHS Foundation Trust, and National Heart and Lung Institute, Imperial College, London, UK

    Andrew G. Nicholson

  7. Department of Medical Oncology, Centre Léon Bérard, Lyon, France

    Jean-Yves Blay

Authors
  1. Pierre Courtiol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles Maussion
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matahi Moarii
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elodie Pronier
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Samuel Pilcer
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Meriem Sefta
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Pierre Manceron
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sylvain Toldo
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mikhail Zaslavskiy
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Nolwenn Le Stang
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Nicolas Girard
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Olivier Elemento
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Andrew G. Nicholson
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Jean-Yves Blay
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Françoise Galateau-Sallé
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Gilles Wainrib
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Thomas Clozel
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., N.L.S., N.G., O.E., A.G.N., J.-Y.B., F.G.-S., G.W. and T.C. conceived the idea for this paper. P.C., C.M. and M.M. implemented the analysis. A.G.N. and F.G.-S. reviewed the tiles of interest. P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., N.L.S., N.G., O.E., A.G.N., J.-Y.B., F.G.-S., G.W. and T.C. contributed to the writing of the manuscript.

Corresponding author

Correspondence to Thomas Clozel.

Ethics declarations

Competing interests

The authors declare the following competing interests. Employment: P.C., C.M., M.M., E.P., S.P., M.S., P.M., S.T., M.Z., T.C. and G.W. are employed by Owkin. Advisory: O.E. is on the scientific advisory board of Owkin and equity holder in the company. Financial: Owkin has submitted a patent application entitled ‘Systems and methods for mesothelioma feature detection and enhanced prognosis or response to treatment’ listing J.-Y.B., F.G.-S., T.C., G.W., P.C. and C.M. as the co-inventors.

Additional information

Peer Review Information Javier Carmona was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 MesoNet layout.

Our prognosis prediction is composed of five steps. (1) The ‘tiling’ process divides the whole slide image into small tiles of 224 × 224 pixels. (2) Features are then extracted from each tile using ResNet-50. (3) Features are then auto-encoded to reduce the dimension to 512 features per tile. (4) We use a convolutional 1D layer to score each tile. (5) Tiles associated with the largest and lowest scores are then retained to train a multilayer perceptron (MLP) to predict overall survival of the patient.

Extended Data Fig. 2 Comparison of the performance between MesoNet and models including additional non-pathology variables such as age and sex to predict MM patient overall survival.

The c index distribution for different predictive models. For the box plots, whiskers represent the minima and maxima. The middle line within the box represents the median. The upper and lower boundaries of the whiskers represent the 25th and 75th percentiles, respectively. A two-sided t-test was performed to determine significance.

Extended Data Fig. 3 Robustness of MesoNet to tissue sampling.

Distributions of variability of the MesoNet prediction for the 167 patients where multiple biopsies were available. The boxplot on the left side represents the distribution of maximal intra-individual variability, that is, the absolute difference between the minimum score and maximum score predicted for all the slides from a single patient for all patients with multiple biopsies (n = 167 patients). The boxplot on the right side represents the inter-individual variability, that is, the absolute difference between the mean of the scores for all the slides of a given patient to the mean of the score for all patients with multiple biopsies (n = 167 patients). For the two boxplots, whiskers represent the minima and maxima. The middle line within the box represents the median. The upper and lower boundaries of the whiskers represent the 25th and 75th percentiles, respectively. Significance was calculated using a two-sided t-test (P = 9.66 × 10−20).

Extended Data Fig. 4 Histological review of tiles of interest.

a, Schematic of the reviewing process. b, Repartition of features of interests in low survival tiles (n = 42) and high survival tiles (n = 42). c, Tiles of low survival with a transitional pattern. d, Tiles of unknown significance, TUS. Scale bars, 112 µm.

Extended Data Fig. 5 Illustration of predictive and non-predictive tiles selection.

Similarity of predictive and non-predictive tiles to a given predictive tile of interest, calculated based on the vector of coefficients obtained with ResNet-50. Similar predictive and non-predictive tiles are then reviewed manually for each extremal tile by pathologists. Scale bars, 112 µm.

Extended Data Fig. 6 Comparative histological analysis of predictive and non-predictive tiles.

Histogram of histological features associated with either predictive or non-predictive tiles that are similar to extremal tiles of high and low survival (n = 42). The review was performed independently by two mesothelioma pathologists.

Extended Data Fig. 7 Biological correlates.

Correlation between MesoNet risk score and the Ki67 expression (n = 54 samples), the ploidy level (n = 36 samples), the EMT score (n = 36 samples) and the leukocyte fraction (n = 36 samples) available for the TCGA dataset.

Supplementary information

Supplementary Tables

Supplementary Tables 1–4

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Courtiol, P., Maussion, C., Moarii, M. et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat Med 25, 1519–1525 (2019). https://doi.org/10.1038/s41591-019-0583-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-019-0583-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer