Decoding the Neural Processing of Speech

3 aims of Sensory Neuroengineering:

1. Understand the neurobiology of (multi-)sensing

2. Diagnose impairments in sensory processing

3. Restore sensory impairments

Hearing & communication disorders:

• Affect 20% of people worldwide, more than 5% in children
• Progress with age → major problem in aging society
• Types: sensorineural hearing loss, hidden hearing loss, auditory processing disorder (APD), language disorders (e.g., after stroke)

Building blocks of speech (timescales):

Brain oscillations & speech synchronization:

• The brain synchronizes (entrains) to the rhythm of speech
• Measured via EEG (oscillatory brain activity)
• Key bands: delta (1–4 Hz), theta (4–8 Hz), alpha (8–12 Hz)
• Theta band → entrains to speech envelope (syllable-level rhythm)
• Attentional modulation found in delta but NOT in theta band

Two models for decoding speech from EEG:

• Forward model: predict EEG from acoustic stimulus
• Backward model: reconstruct speech features from EEG

What can be decoded from brain activity:

• Clarity (acoustic quality of speech)
• Comprehension (cognitive understanding)
• Auditory attention — which speaker a person is attending to (real-time)

Auditory Attention Decoding — the vision:

• Hearing aids + cochlear implants struggle in noisy environments ("cocktail party problem")
• EEG can decode _which_ speaker the user is attending to
• This enables neurofeedback-driven hearing instruments that amplify the attended speaker

GAN for audiovisual speech:

• A Generative Adversarial Network (GAN) generates realistic talking-face video from speech + a still image
• Turing test result: humans cannot differentiate between real and synthetic videos
• Synthetic videos improve speech-in-noise comprehension (but not as much as natural videos)

Vibrotactile stimulation:

• Pulses delivered to fingertips at the rhythm of syllables (theta rhythm)
• Enhances speech comprehension in noise
• EEG confirms the brain's multisensory integration of tactile + auditory signals

Summary (from slides):

• Brain waves synchronize to speech rhythms
• Speech comprehension AND attention can be decoded from this synchronization
• Visual and tactile signals can enhance speech comprehension in noise