← Back to Week 12 · All Course Demos

Multi-Modal Fusion & Representational Similarity Analysis (RSA)

Multi-Modal Data Integration · Bridging Millisecond EEG Dynamics & Millimeter fMRI Spatial Patterns via Second-Order Isomorphism

The Integration Dilemma: fMRI provides millimeter spatial resolution but sluggish hemodynamics (~6 s). EEG provides millisecond temporal resolution but smeared scalp topography. Direct asymmetric fitting (e.g. using fMRI as a hard EEG source constraint) risks mislocalization due to neurovascular decoupling. Representational Similarity Analysis (RSA) solves multi-modal fusion without forcing disparate modalities into the same coordinate system. By comparing condition-by-condition Representational Dissimilarity Matrices (RDMs), we track when spatial representations in specific brain structures emerge in millisecond electrophysiology.

EEG-fMRI Cross-Modal Fusion Trajectory \(\rho(t)\)

V1 (Early Visual Cortex) · Peak ~85 ms

LOC (Object Shape) · Peak ~135 ms

FFA (Face Category) · Peak ~170 ms (N170)

EEG RDM at 170 msSensor Pattern

fMRI V1 Spatial RDMVoxel Pattern

fMRI FFA Spatial RDMVoxel Pattern

Post-Stimulus Latency (t) 170 ms

Scrub to inspect evolving RDM geometry

Spatial Cortical Localization:

6 Experimental Categories:

1. Human Faces

2. Animal Faces

3. Human Bodies

4. Natural Scenes

5. Handheld Tools

6. Abstract Shapes

Second-Order Isomorphism & Representational Geometries

1. Why Direct Fusion Fails:

Combining fMRI and EEG directly into a single joint cost function faces fundamental biological barriers. Hemodynamics reflect metabolic demand (synaptic astrocyte uptake of glutamate), which can dissociate from instantaneous postsynaptic electrical field potentials. Forcing EEG sources into fMRI statistical hotspots ignores hemodynamic delays and introduces dangerous spatial bias for invisible or transient neural events.

2. Second-Order Isomorphism:

Representational Similarity Analysis establishes equivalence at the level of information geometry. For \(K\) conditions, each modality defines a symmetric \(K \times K\) Representational Dissimilarity Matrix (RDM):

D_{ij} = 1 - r(\vec{y}_i, \vec{y}_j)

Because distance matrices are coordinate-free, an fMRI RDM (computed across voxels) and an EEG RDM (computed across electrodes) exist in the exact same representational space, allowing direct Spearman rank correlation:

\rho(t) = \text{Spearman}\left(\text{vec}(\text{RDM}_{\text{EEG}}(t)), \, \text{vec}(\text{RDM}_{\text{fMRI}})\right)

3. Spatiotemporal Cascades of Object Vision:

Tracking \(\rho(t)\) over time reveals the hierarchical flow of sensory information:

Early Peak (~80 ms): EEG correlates with Early Visual Cortex (V1). The RDM geometry reflects low-level retinotopic features (edge energy, spatial contrast).
Mid Peak (~135 ms): Correlation shifts to the Lateral Occipital Complex (LOC), reflecting structural shape boundaries.
Categorical Peak (~170 ms): Strong correlation with the Fusiform Face Area (FFA), producing the classic face-selective N170 waveform and categorical separation of animate vs. inanimate objects.

4. Advantages in Clinical & Cognitive Research:

RSA multi-modal fusion does not require simultaneous EEG-fMRI acquisition; datasets recorded in separate sessions can be fused seamlessly. Furthermore, computational models (deep convolutional neural networks, semantics) can be tested against the same brain RDMs, bridging human neuroscience and artificial intelligence.