Skip to main content
SpringerLink
Log in

A multi-branch deep convolutional fusion architecture for 3D microwave inverse scattering: stored grain application

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this paper, a multi-branch deep convolutional fusion architecture is proposed to solve electromagnetic inverse scattering problems. The conventional methods for solving inverse problems face various challenges, including strong ill-conditioning, expensive computational cost, and unavoidable intrinsic nonlinearity. To overcome these difficulties, we designed a novel multi-branch convolutional neural network (CNN) to reconstruct the 3D images of the moisture distribution in stored grain. Inspired by objective-function techniques for solving the electromagnetic inverse scattering problems, the proposed CNN architecture takes in the scattered-field data and prior information to produce 3D images of the moisture content. With the aim of using inputs of different formats, i.e., a complex-valued vector of scattered-field data and a 3D image of the background moisture distribution as prior information, we propose a multi-branch architecture consisting of decoder-only, and encoder–decoder, convolutional branches. The two branches are later fused to produce the final reconstructed 3D image. To train the CNN, we use the true numerical grain moisture distribution image, which were synthetically generated. The reconstructed moisture distribution images produced by the proposed CNN show that the network is not only able to reconstruct the 3D moisture distribution images directly from measured scattered-field data for high contrast objects-of-interest, but it also achieves a higher imaging quality compared with traditional inversion techniques in microwave imaging. Quantitative evaluations are reported using receiver operating characteristics curves for the hotspot detectability and RMS error. The proposed approach opens a novel path for the deep learning-based real-time quantitative microwave imaging.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Meaney PM, Fanning MW, Li D, Poplack SP, Paulsen KD (2000) A clinical prototype for active microwave imaging of the breast. IEEE Trans Microw Theory Tech 48(11):1841–1853

    Article  Google Scholar 

  2. Lambot S, Slob EC, van den Bosch I, Stockbroeckx B, Vanclooster M (2004) Modeling of ground-penetrating radar for accurate characterization of subsurface electric properties. IEEE Trans Geosci Remote Sens 42(11):2555–2568

    Article  Google Scholar 

  3. Asefi M, Jeffrey I, LoVetri J, Gilmore C, Card P, Paliwal J (2015) Grain bin monitoring via electromagnetic imaging. Comput Electron Agric 119:133–141

    Article  Google Scholar 

  4. Kurrant D, Baran A, LoVetri J, Fear E (2017) Integrating prior information into microwave tomography part 1: impact of detail on image quality. Med Phys 44(12):6461–6481

    Article  Google Scholar 

  5. Baran A, Kurrant D, Fear E, LoVetri J (2017) Integrating prior information into microwave tomography part 2: impact of errors in prior information on microwave tomography image quality. Med Phys 44(12):6482–6503

    Article  Google Scholar 

  6. Golnabi AH, Meaney PM, Geimer SD, Paulsen KD (2019) 3-d microwave tomography using the soft prior regularization technique: evaluation in anatomically realistic mri-derived numerical breast phantoms. IEEE Trans Biomed Eng 66(9):2566–2575

    Article  Google Scholar 

  7. Odle TG (2015) Breast imaging artifacts. Radiol Technol 87:65M–87M

    Google Scholar 

  8. Mojabi P, LoVetri J (2016) Composite tissue-type and probability image for ultrasound and microwave tomography. IEEE J Multiscale Multiphys Comput Tech 1:26–35

    Article  Google Scholar 

  9. Hughson M, LoVetri J, Jeffrey I (2019) Microwave breast imaging incorporating material property dependencies. In: IEEE MTT-S international microwave symposium (IMS), vol 2019, pp 1450–1453

  10. Jayasand DS, Whiteand NDG, Muirand WE (eds) (1995) The stored-grain ecosystem. Stored grain ecosystems, vol 1. M. Dekker, New York, pp 1–32

    Google Scholar 

  11. LoVetri J, Asefi MA, Gilmore C, Jeffrey I (2020) Innovations in electromagnetic imaging technology: the stored-grain-monitoring case. IEEE Antennas Propag Mag

  12. Gilmore C, Asefi M, Paliwal J, LoVetri J (2017) Industrial scale electromagnetic grain bin monitoring. Comput Electron Agric 136:210–220

    Article  Google Scholar 

  13. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  14. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. CoRR, vol. abs/1505.04597. [Online]. http://arxiv.org/abs/1505.04597

  15. Wang G, Li W, Zuluaga MA, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T (2018) Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans Med Imaging 37(7):1562–1573

    Article  Google Scholar 

  16. Shin H, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35(5):1285–1298

    Article  Google Scholar 

  17. McCann MT, Jin KH, Unser M (2017) Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal Process Mag 34(6):85–95

    Article  Google Scholar 

  18. Xie Y, Xia Y, Zhang J, Song Y, Feng D, Fulham M, Cai W (2019) Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest ct. IEEE Trans Med Imaging 38(4):991–1004

    Article  Google Scholar 

  19. June-Goo L, Sanghoon J, Young-Won C, Hyunna L, Guk Bae K, Joon Beom S, Namkug K (2017) Deep learning in medical imaging: general overview. KJR 18(4):570–584

    Google Scholar 

  20. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems, vol 25. Curran Associates, Inc., Red Hook, pp 1097–1105

    Google Scholar 

  21. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  23. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. CoRR, vol. abs/1409.4842, [Online]. http://arxiv.org/abs/1409.4842

  24. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation

  25. Schlemper J, Caballero J, Hajnal JV, Price AN, Rueckert D (2018) A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans Med Imaging 37(2):491–503

    Article  Google Scholar 

  26. Lundervold AS, Lundervold A (2019) An overview of deep learning in medical imaging focusing on mri. Zeitschrift für Medizinische Physik, 29(2):102 – 127. special Issue: Deep Learning in Medical Physics. [Online]. http://www.sciencedirect.com/science/article/pii/S0939388918301181

  27. Khoshdel V, Ashraf A, LoVetri J (2019) Enhancement of multimodal microwave-ultrasound breast imaging using a deep-learning technique. Sensors

  28. Khoshdel V, Asefi M, Ashraf A, LoVetri J (2020) Full 3d microwave breast imaging using a deep-learning technique. J Imaging 6(8):80

    Article  Google Scholar 

  29. Chen G, Shah P, Stang J, Moghaddam M (2020) Learning-assisted multimodality dielectric imaging. IEEE Trans Antennas Propag 68(3):2356–2369

    Article  Google Scholar 

  30. Yao HM, Sha WEI, Jiang L (2019) Two-step enhanced deep learning approach for electromagnetic inverse scattering problems. IEEE Antennas Wirel Propag Lett 18(11):2254–2258

    Article  Google Scholar 

  31. Zhu B, Liu JZ, Cauley S.F, Rosen B. R, Rosen M. S. (2018) Image reconstruction by domain-transform manifold learning. Nature 555:487–492

    Article  Google Scholar 

  32. Zhao W, Wang H, Gemmeke H, van Dongen KWA, Hopp T, Hesser J (2017) Ultrasound transmission tomography image reconstruction with fully convolutional neural network. IEEE Trans Med Imaging

  33. Li L, Wang LG, Teixeira FL, Liu C, Nehorai A, Cui TJ (2019) Deepnis: deep neural network for nonlinear electromagnetic inverse scattering. IEEE Trans Antennas Propag 67(3):1819–1825

    Article  Google Scholar 

  34. Golnabi AH, Meaney PM, Epstein NR, Paulsen KD (2011) Microwave imaging for breast cancer detection: advances in three-dimensional image reconstruction. In: Conference of proceedings IEEE engineering in medicine and biology society, pp 5730–5733

  35. van den Berg PM, Kleinman RE (1997) A contrast source inversion method. Inverse Probab 13(6):1607

    Article  MathSciNet  Google Scholar 

  36. Nelson SO (1991) Dielectric properties of agricultural products-measurements and applications. IEEE Trans Electr Insul 26(5):845–869

    Article  Google Scholar 

  37. Asefi M, Baran A, LoVetri J (2019) An experimental phantom study for air-based quasi-resonant microwave breast imaging. IEEE Trans Microw Theory Tech 67(9):3946–3954

    Article  Google Scholar 

  38. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: In Proceedings of the international conference on artificial intelligence and statistics (AISTATS–10). Society for Artificial Intelligence and Statistics

  39. Kingma, DP, Ba J (2014) Adam: a method for stochastic optimization [Online]. http://arxiv.org/abs/1412.6980

  40. Abubakar A, van den Berg PM, Mallorqui JJ (2002) Imaging of biomedical data using a multiplicative regularized contrast source inversion method. IEEE Trans Microw Theory Tech 50(7):1761–1771

    Article  Google Scholar 

  41. Zakaria A, Gilmore C, LoVetri J (2010) Finite-element contrast source inversion method for microwave imaging. Inverse Probl 26(11):115010

    Article  MathSciNet  Google Scholar 

  42. Edwards K, Krakalovich K, Kruk R, Khoshdel V, LoVetri J, Gilmore C, Jeffrey I (2020) The implementation of neural networks for phaseless parametric inversion. In: URSI GASS 2020

Download references

Acknowledgements

Funding was provided by Natural Sciences and Engineering Research Council of Canada and Canadian Cancer Society.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Manitoba, Winnipeg, MB, Canada

    Vahab Khoshdel, Mohhamad Asefi, Ahmed Ashraf & Joe LoVetri

Authors
  1. Vahab Khoshdel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohhamad Asefi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Ashraf
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joe LoVetri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahab Khoshdel.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khoshdel, V., Asefi, M., Ashraf, A. et al. A multi-branch deep convolutional fusion architecture for 3D microwave inverse scattering: stored grain application. Neural Comput & Applic 33, 13467–13479 (2021). https://doi.org/10.1007/s00521-021-05970-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05970-3

Keywords

Search

Navigation