Behavioral and gaze-data analysis with deep learning neural networks

I've joined the Pattern Recognition Company in Lübeck, Germany since January 1st, 2022. And I am enrolled as a PhD student at Lübeck University. Here I am supervised by Prof. Erhardt Barth. 

Personal Background and Interest:

I have spent 22 years living in China, and I have got my Bachelor’s Degree in Electrical Engineering and Automation from School of Mechatronic Engineering and Automation, Shanghai University. Then I went to Singapore and got my Master’s Degree in Mechanical Engineering from National University of Singapore. Since then, my research interests have become computer vision and deep learning for medicine. I would like to analyze medical data and design medical toolboxes by using artificial intelligence.

Aim of the project:

The objectives of my project are to 1) perform a comprehensive evaluation of different deep-learning approaches for analyzing behavioral and gaze-data and, 2) develop a machine-learning toolbox that can be used to discover diagnostic or rehabilitation relevance in these data.

Current activities:

To date, I have used machine learning to reveal task-relevant patterns and features in the gaze-data for videos. In particular, I have designed a network architecture that can extract temporal features from gaze-data. This architecture has been applied to the Weizmann dataset with a near 100% accuracy. On the larger UCF101 dataset, an improved slicing architecture increased the accuracy from 61% to 81%. 

To further the accuracy for the UCF101 dataset, I used UniFormerV2 (a Transformer architecture) with the accuracy to 94.61%. Currently, I’m trying to improve the architecture to reach even higher accuracy.

In collaboration with ESR7 (Safa), I applied deep learning networks on VR navigation data to find some facts and features of glaucoma behavior that can improve the experiment. Once a best performing architecture is obtained, it can be used to extract new patterns and features to predict a patient’s functional vision or other diseases.

Furthermore, I have collaborated with Lübeck University. Here I developed a machine learning method to classify First-Episode Psychosis, Clinic High Risk of Psychosis and Healthy Control based on brain Magnetic Resonance Imaging (MRI) images.

I have also started my secondment in IIT, Italy where together with ESR6 (Kurt) we collected audio-spatial compression data from individuals with scotomas.

Future directions:

In the near future I’ll continue my collaboration with ESR6 and analyze the audio-spatial compression data. I’ll also continue the  collaboration with ESR7 and find the differences between the daily life behavior of health control and glaucoma patients. And I will further design the brain-MRI model to make it not only diagnose psychosis, but also predict the transition of clinical high risks and localize the most related brain region to psychosis. I will still further improve the accuracy of video recognition architecture.