Skip to main content
SpringerLink
Log in

Challenges and Benchmarks in Bioimage Analysis

  • Chapter
  • First Online:
Focus on Bio-Image Informatics

Part of the book series: Advances in Anatomy, Embryology and Cell Biology ((ADVSANAT,volume 219))

Abstract

Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI). This review summarizes recent progress in this respect and describes the general process of designing a bioimage analysis benchmark or challenge, including the proper selection of datasets and evaluation metrics. It also presents examples of specific target applications and biological research tasks that have benefited from these challenges with respect to the performance of automatic image analysis methods that are crucial for the given task. Finally, available benchmarks and challenges in terms of common features, possible classification and implications drawn from the results are analysed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • 2DSNEMI Challenge (2012) MIT. http://brainiac2.mit.edu/isbi_challenge/. Accessed 17 May 2015

  • 3DSNEMI Challenge (2013) MIT. http://brainiac2.mit.edu/SNEMI3D/. Accessed 17 May 2015

  • AMIDA (2013) University Medical Center Utrecht. http://amida13.isi.uu.nl/. Accessed 17 May 2015

  • BBBC (2008) Broad Institute of Harvard and MIT. http://www.broadinstitute.org/bbbc/. Accessed 17 May 2015

  • BigNeuron (2015) Allen Institute for Brain Science. http://bigneuron.org. Accessed 17 May 2015

  • Buck TE, Li J, Rohde GK, Murphy RF (2012) Toward the virtual cell: automated approaches to building models of subcellular organization “learned” from microscopy images. Bioessays 34:791–799

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Carpenter A, Kamentsky L, Eliceiri KW (2012) A call for bioimaging software usability. Nat Methods 9(7):666–670

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • CCDB (2002) University of California, San Diego. http://ccdb.ucsd.edu/. Accessed 17 May 2015

  • CellOrganizer (2012) Carnegie Mellon University, Pittsburgh. http://cellorganizer.org/. Accessed 17 May 2015

  • Chenouard N et al (2014) Objective comparison of particle tracking methods. Nat Methods 11(3):281–289

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Coelho LP, Shariff A, Murphy RF (2009) Nuclear segmentation in microsope cell images: a hand-segmented dataset and comparison of algorithms. In: Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging (ISBI 2009), IEEE, Boston, 28 June–1 July 2009, pp 518–521

    Google Scholar 

  • Computer Vision Homepage (1994) School of Computer Science, Carnegie Mellon University. http://www.cs.cmu.edu/~cil/vision.html. Accessed 17 May 2015

  • CIL (2010) American Society for Cell Biology. http://www.cellimagelibrary.org/. Accessed 17 May 2015

  • CTC (2013) University of Navarra. http://www.codesolorzano.com/celltrackingchallenge/. Accessed 17 May 2015

  • CytoPacq (2008) Masaryk University, Brno. http://cbia.fi.muni.cz/simulator/. Accessed 17 May 2015

  • Deconvolution Benchmark Datasets (2010) EPFL. http://bigwww.epfl.ch/deconvolution/. Accessed 17 May 2015

  • DIADEM Challenge (2010) Howard Hughes Medical Institute. http://diademchallenge.org/. Accessed 17 May 2015

  • DMC (2013) EPFL. http://bigwww.epfl.ch/deconvolution/challenge/. Accessed 17 May 2015

  • Drelie Gelasca E, Obara B, Fedorov D, Kvilekval K, Manjunath BS (2009) A biosegmentation benchmark for evaluation of bioimage analysis methods. BMC Bioinformatics 10:368

    Article  PubMed  PubMed Central  Google Scholar 

  • Ferko MC, Patterson BW, Butler PJ (2006) High-resolution solid modeling of biological samples imaged with 3D fluorescence microscopy. Microsc Res Tech 69(8):648–655

    Article  PubMed  PubMed Central  Google Scholar 

  • Foggia P, Percannella G, Soda P, Vento M (2013) Benchmarking HEp-2 cells classification methods. IEEE Trans Med Imaging 32(10):1878–1889

    Article  PubMed  Google Scholar 

  • Gillette TA, Brown KM, Ascoli GA (2011) The DIADEM metric: comparing multiple reconstructions of the same neuron. Neuroinformatics 9(2-3):233–245

    Article  PubMed  PubMed Central  Google Scholar 

  • GLAS (2015) Department of Computer Science, University of Warwick. http://www2.warwick.ac.uk/fac/sci/dcs/research/combi/research/bic/glascontest/. Accessed 17 May 2015

  • Griffa A, Garin N, Sage D (2010) Comparison of deconvolution software in 3D microscopy: a user point of view, part I and part II. G.I.T. Imaging Microscopy 1:43–45

    Google Scholar 

  • Gurcan MN, Madabhushi A, Rajpoot N (2010) Pattern recognition in histopathological images: an ICPR 2010 contest. In: Ünay D, Çataltepe Z, Aksoy S (eds) Recognizing patterns in signals, speech, images and videos. Lecture notes in computer science, vol 6388. Springer, Heidelberg, pp 226–234

    Google Scholar 

  • HEp-2 Images Dataset (2012) Mivia Lab, University of Salerno. http://mivia.unisa.it/datasets/biomedical-image-datasets/hep2-image-dataset/. Accessed 17 May 2015

  • Hill E (2008) Announcing the JCB DataViewer, a browser-based application for viewing original image files. J Cell Biol 183:969–970

    Article  CAS  PubMed Central  Google Scholar 

  • JCB DataViewer (2008) Rockefeller University Press. http://jcb-dataviewer.rupress.org/. Accessed 17 May 2015

  • Jannin P, Grova C, Maurer C (2006) Model for defining and reporting reference-based validation protocols in medical image processing. Int J CARS 1(2):63–73

    Article  Google Scholar 

  • Kozubek M, Matula P (2000) An efficient algorithm for measurement and correction of chromatic aberrations in fluorescence microscopy. J Microsc 200(3):206–217

    Article  CAS  PubMed  Google Scholar 

  • Kwan RK-S, Evans AC, Pike GB (1999) MRI simulation-based evaluation of image-processing and classification methods. IEEE Trans Med Imaging 18(11):1085–1097

    Article  CAS  PubMed  Google Scholar 

  • Lehmussola A, Selinummi J, Ruusuvuori P, Niemist A, Yli-Harja O (2005) Simulating fluorescent microscope images of cell populations. In: Proceedings of the 27th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC’05), IEEE, Shanghai, 17–18 Jan 2006, pp 3153–3156

    Google Scholar 

  • Ljosa V, Sokolnicki KL, Carpenter AE (2012) Annotated high-throughput microscopy image sets for validation. Nat Methods 9(7):637

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • LMC (2013) EPFL. http://bigwww.epfl.ch/smlm/challenge2013/. Accessed 17 May 2015

  • Lockett SJ, Sudar D, Thompson CT, Pinkel D, Gray JW (1998) Efficient, interactive, and three-dimensional segmentation of cell nuclei in thick tissue sections. Cytometry A 31:275–286

    Article  CAS  Google Scholar 

  • Malm P, Brun A, Bengtsson E (2015) Simulation of bright-field microscopy images depicting Pap-Smear specimen. Cytometry A 87A:212–226

    Article  Google Scholar 

  • Martone ME, Zhang S, Gupta A, Qian X, He H, Price DL, Wong M, Santini S, Ellisman MH (2003) The cell-centered database: a database for multiscale structural and protein localization data from light and electron microscopy. Neuroinformatics 1(4):379–395

    Article  PubMed  Google Scholar 

  • Martone ME, Tran J, Wong WW, Sargis J, Fong L, Larson S, Lamont SP, Gupta A, Ellisman MH (2008) The Cell Centered Database project: an update on building community resources for managing and sharing 3D imaging data. J Struct Biol 161(3):220–231

    Article  PubMed  PubMed Central  Google Scholar 

  • Maška M et al (2014) A benchmark for comparison of cell tracking algorithms. Bioinformatics 30(11):1609–1617

    Article  PubMed  PubMed Central  Google Scholar 

  • McNally JG, Cogswell CJ, Fekete PW, Conchello JA (1997) Comparison of 3D microscopy methods by imaging a well characterized test object. In: Cogswell CJ, Conchello JA, Wilson T (eds) Three-dimensional microscopy: image acquisition and processing IV, San Jose, 8 Feb 1997. Proc SPIE, vol 2984, pp 52–63

    Google Scholar 

  • MITOS-ATYPIA (2014) Consortium for Open Medical Image Computing. http://mitos-atypia-14.grand-challenge.org/. Accessed 17 May 2015

  • Murphy Lab Data (1999) Carnegie Mellon University. http://murphylab.web.cmu.edu/data/. Accessed 17 May 2015

  • OCCISC (2014) University of Adelaide. http://cs.adelaide.edu.au/~zhi/isbi15_challenge/. Accessed 17 May 2015

  • Open Bio Image Alliance (2015) http://www.openbioimage.org/. Accessed 17 May 2015

  • Orloff DN, Iwasa JH, Martone ME, Ellisman MH, Kane CM (2012) The cell: an image library-CCDB: a curated repository of microscopy data. Nucleic Acids Res 41:D1241–D1250

    Article  PubMed  PubMed Central  Google Scholar 

  • Ortiz-de-Solórzano C, Muñoz-Barrutia A, Meijering E, Kozubek M (2015) Toward a morphodynamic model of the cell: signal processing for cell modeling. IEEE Signal Proc Mag 32(1):20–29

    Article  Google Scholar 

  • Peng H, Hawrylycz M, Roskams J, Hill S, Spruston N, Meijering E, Ascoli GA (2015) BigNeuron: large-scale 3D neuron reconstruction from optical microscopy images. Neuron 87(2):252–256

    Article  CAS  PubMed  Google Scholar 

  • Price K (1986) Anything you can do, I can do better (no you can’t). Comput Vision Graph 36:387–391

    Article  Google Scholar 

  • Rajaram S, Pavie B, Hac NE, Altschuler SJ, Wu LF (2012) SimuCell: a flexible framework for creating synthetic microscopy images. Nat Methods 9(7):634–635

    Article  CAS  PubMed  Google Scholar 

  • Rosenberg C (1996) The Lenna Story. http://www.lenna.org. Accessed 17 May 2015

  • Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Le Naour G, Gurcan MN (2013) Mitosis detection in breast cancer histological images: an ICPR 2012 contest. J Pathol Inform 4:8

    Article  PubMed  PubMed Central  Google Scholar 

  • Sage D, Kirshner H, Pengo T, Stuurman N, Min J, Manley S, Unser M (2015) Quantitative evaluation of software packages for single-molecule localization microscopy. Nat Methods 12(8):717–724

    Article  CAS  PubMed  Google Scholar 

  • Sage D, Kirshner H, Vonesch C, Lefkimmiatis S, Unser M (2013) Benchmarking image-processing algorithms for biomicroscopy: reference datasets and perspectives. In: Proceedings of the 21st European Signal Processing Conference (EUSIPCO), IEEE, Marrakech, 9–13 Sept 2013, pp 1–4

    Google Scholar 

  • Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, Matsui M, Fujita H, Kodera Y, Doi K (2000) Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR Am J Roentgenol 174(1):71–74

    Article  CAS  PubMed  Google Scholar 

  • SimuCell (2012) Altschuler & Wu laboratories, University of California, San Francisco. http://awlab.ucsf.edu/Web_Site/SimuCell/documentation.html. Accessed 17 May 2015

  • Svoboda D, Kozubek M, Stejskal S (2009) Generation of digital phantoms of cell nuclei and simulation of image formation in 3D image cytometry. Cytometry A 75A:494–509

    Article  Google Scholar 

  • UCSB Biosegmentation Benchmark (2008) University of California, Santa Barbara. http://bioimage.ucsb.edu/research/bio-segmentation. Accessed 17 May 2015

  • van Ginneken B, Kerkstra S (2015) Grand challenges in biomedical image analysis. http://grand-challenge.org/. Accessed 17 May 2015

  • Veta M et al (2015) Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 20(1):237–248

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The author would like to thank Martin Maška and David Svoboda for their feedback and useful comments. This work was supported by the Czech Science Foundation (Grant No. 302/12/G157).

Author information

Authors and Affiliations

  1. Faculty of Informatics, Centre for Biomedical Image Analysis, Masaryk University, Botanická 68a, Brno, 60200, Czech Republic

    Michal Kozubek

Authors
  1. Michal Kozubek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michal Kozubek .

Editor information

Editors and Affiliations

  1. Dept. of Veterinary Sciences, University of Antwerp, Antwerpen, Antwerpen, Belgium

    Winnok H. De Vos

  2. Dept. of Human Genetics, KU Leuven, Leuven, Belgium

    Sebastian Munck

  3. University of Antwerp, Antwerp, Belgium

    Jean-Pierre Timmermans

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Kozubek, M. (2016). Challenges and Benchmarks in Bioimage Analysis. In: De Vos, W., Munck, S., Timmermans, JP. (eds) Focus on Bio-Image Informatics. Advances in Anatomy, Embryology and Cell Biology, vol 219. Springer, Cham. https://doi.org/10.1007/978-3-319-28549-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-28549-8_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-28547-4

  • Online ISBN: 978-3-319-28549-8

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation