Elsevier

Neurocomputing

Volume 443, 5 July 2021, Pages 356-368
Neurocomputing

Micro-expression spotting: A new benchmark

https://doi.org/10.1016/j.neucom.2021.02.022Get rights and content
Under a Creative Commons license
open access

Abstract

Micro-expressions (MEs) are brief and involuntary facial expressions that occur when people are trying to hide their true feelings or conceal their emotions. Based on psychology research, MEs play an important role in understanding genuine emotions, which leads to many potential applications. Therefore, ME analysis has become an attractive topic for various research areas, such as psychology, law enforcement, and psychotherapy. In the computer vision field, the study of MEs can be divided into two main tasks, spotting and recognition, which are used to identify positions of MEs in videos and determine the emotion category of the detected MEs, respectively. Recently, although much research has been done, no fully automatic system for analyzing MEs has yet been constructed on a practical level for two main reasons: most of the research on MEs only focuses on the recognition part, while abandoning the spotting task; current public datasets for ME spotting are not challenging enough to support developing a robust spotting algorithm. The contributions of this paper are threefold: (1) we introduce an extension of the SMIC-E database, namely the SMIC-E-Long database, which is a new challenging benchmark for ME spotting; (2) we suggest a new evaluation protocol that standardizes the comparison of various ME spotting techniques; (3) extensive experiments with handcrafted and deep learning-based approaches on the SMIC-E-Long database are performed for baseline evaluation.

Keywords

Micro-expression spotting
Benchmark
Evaluation protocol

Cited by (0)

Thuong-Khanh Tran received his B.S. degree in Mathematics and Computer Science from the University of Science-VNUHCM, Vietnam, in 2010, and his M.S degree in Electronics Engineering from Chonnam National University, South Korea, in 2015. He is currently Ph.D candidate in Center for Machine Vision and Signal Analysis of University of Oulu, Finland. His research topics focus on computer vision, emotional analysis and micro-expression.

Quang-Nhat Vo received his B.S. degree in Information Technology from the University of Science-VNUHCM, Vietnam, in 2010, and his M.S. and Ph.D. degree in Electronics and Computer Engineering from Chonnam National University, Republic of Korea, in 2017. He worked as Postdoc researcher at Center for Machine Vision and Signal Analysis, University of Oulu, Finland. He is currently working as A.I scientist in Silo.AI. His study interests are multimedia and image processing, facial expression analysis, and pattern recognition.

Xiaopeng Hong received his Ph.D. degree in computer application and technology from Harbin Institute of Technology, P. R. China, in 2010. He is currently working as Distinguished Research Fellow at Xi’an Jiaotong University, China. Dr. Hong has published over 30 articles in mainstream journals and conferences such as IEEE T-PAMI, T-IP, CVPR and ACM UbiComp. His current research interests include multi-modal learning, affective computing, intelligent medical examination, and human-computer interaction, etc. His research has been reported by global media including MIT Technology Review and Daily Mail.

Xiaobai Li received her B.Sc degree in Psychology from Peking University, M.Sc degree in Biophysics from the Chinese Academy of Science, and Ph.D. degree in Computer Science from University of Oulu. She is currently an assistant professor in the Center for Machine Vision and Signal Analysis of University of Oulu. Her research interests include spontaneous vs. posed facial expression comparison, micro-expression and deceitful behaviors, and heart rate measurement from facial videos.

Guoying Zhao (IEEE Senior member 2012, IAPR Fellow) received the Ph.D. degree in computer science from the Chinese Academy of Sciences, Beijing, China, in 2005. Then she worked as senior researcher since 2005 and an Associate Professor since 2014 with the Center for Machine Vision and Signal Analysis, University of Oulu, Finland. She is currently a full professor with University of Oulu, Finland and a visiting professor with Northwest University, China. In 2020, she was selected to the prestigious Academy Professor position. She was Nokia visiting professor in 2016. She has authored or coauthored more than 250 papers in journals and conferences. Her papers have currently over 14900 citations in Google Scholar (h-index 55). She is co-program chair for ACM International Conference on Multimodal Interaction (ICMI 2021), was co-publicity chair for FG2018, General chair of 3rd International Conference on Biometric Engineering and Applications (ICBEA 2019), and Late Breaking Results Co-Chairs of 21st ACM International Conference on Multimodal Interaction (ICMI 2019), has served as area chairs for several conferences and is associate editor for Pattern Recognition, IEEE Transactions on Circuits and Systems for Video Technology, and Image and Vision Computing Journals. She has lectured tutorials at ICPR 2006, ICCV 2009, SCIA 2013 and FG 2018, authored/edited three books and eight special issues in journals. Dr. Zhao was a Co-Chair of many International Workshops at ICCV, CVPR, ECCV, ACCV and BMVC. Her current research interests include image and video descriptors, facial-expression and micro-expression recognition, emotional gesture analysis, affective computing, and biometrics. Her research has been reported by Finnish TV programs, newspapers and MIT Technology Review.