Yong-Jin Liu

Yong-Jin Liu

Yong-Jin Liu is a tenured Full Professor with the Computer Science Department at Tsinghua University. He obtained his Ph.D. degree in 2004 from Hong Kong University of Science & Technology. His research interests include computational geometry, computer graphics, computer aided design and pattern analysis.

Multimodal Contrastive Learning for Cybersickness Recognition Using Brain Connectivity Graph Representation

Published in IEEE Transactions on Visualization and Computer Graphics , 2025

Abstract: Cybersickness significantly impairs user comfort and immersion in virtual reality (VR). Effective identification of cybersickness leveraging physiological, visual, and motion data is a critical prerequisite for its mitigation. However, current methods primarily employ direct feature fusion across modalities, which often leads to limited accuracy due to inadequate modeling of inter-modal relationships. In this paper, we propose a multimodal contrastive learning method for cybersickness recognition. First, we introduce Brain Connectivity Graph Representation (BCGR), an innovative graph-based representation that captures cybersickness-related connectivity patterns across modalities. We further develop three BCGR instances: E-BCGR, constructed based on EEG signals; MV-BCGR, constructed based on video and motion data; and S-BCGR, obtained through our proposed standardized decomposition algorithm. Then, we propose a connectivity-constrained contrastive fusion module, which aligns E-BCGR and MV-BCGR into a shared latent space via graph contrastive learning while utilizing S-BCGR as a connectivity constraint to enhance representation quality. Moreover, we construct a multimodal cybersickness dataset comprising synchronized EEG, video, and motion data collected in VR environments to promote further research in this domain. Experimental results demonstrate that our method outperforms existing state-of-the-art methods across four critical evaluation metrics: accuracy, sensitivity, specificity, and the area under the curve. Source code: https://github.com/PEKEW/cybersickness-bcgr.

Download paper here

More information

Recommended citation: P. Wang, M. Li, Z. Wang, Y. -J. Liu and L. Wang, “Multimodal Contrastive Learning for Cybersickness Recognition Using Brain Connectivity Graph Representation,” in IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 11, pp. 10080-10089, Nov. 2025, doi: 10.1109/TVCG.2025.3616797.