← Back to Course Home

Week 12: Multi-modal Integration

No single imaging modality provides millisecond temporal precision alongside whole-brain millimeter spatial coverage. This week examines multi-modal data fusion—such as simultaneous EEG-fMRI and PET-MR—addressing hardware safety, mutual artifact contamination, and asymmetric data integration.

Interactive Demos & Activities

• Cross-Modality Temporal Comparison

  Comparative Analysis · Fast Electrical vs. Slow Hemodynamic

  Compare the millisecond electrical events simulated in the EEG simulator (../../neuroengineering/eeg.html) directly with the 5–6 second hemodynamic delay in the fMRI simulator, demonstrating the fundamental challenge of simultaneous recording.

• Kalman Filter Sensor Fusion Simulator

  State-Space Modeling · Sensor Integration

  Tune process noise $Q$ versus measurement noise $R$ to understand how state-space filters combine noisy multi-sensor streams into an optimal hidden state estimate.

• NBL 425 Modality Game Center

  Unified Modality Hub · Comparative Review

  Play through all four modality games (MRI, PET, EEG, MEG) to synthesize physical principles, hardware components, and clinical strengths across modalities.

Topics Covered

• Simultaneous EEG-fMRI: Gradient artifacts (switched magnetic fields) and ballistocardiogram artifacts (pulsatile blood flow in the magnetic field) and their correction.
• Integrated PET-MRI: Avalanche photodiodes and silicon photomultipliers (SiPMs) operating within high magnetic fields; MR-based attenuation correction (MRAC).
• Asymmetric & Symmetric Data Fusion: fMRI-constrained EEG/MEG source localization, joint independent component analysis (jICA), and representational similarity analysis (RSA).