IEEE JBHI 2024

Deep Representation Learning for Open Vocabulary
Electroencephalography-to-Text Decoding

Hamza Amrani, Daniela Micucci, Paolo Napoletano
University of Milano-Bicocca
Paper arXiv Code
42.75
BLEU-1
+1.40 vs. SoTA
33.28
ROUGE-1-F
+2.59 vs. SoTA
53.86
BERTScore-F
+3.20 vs. SoTA

TL;DR — We present an end-to-end architecture that decodes natural language directly from non-invasive EEG brain recordings, combining subject-dependent representation learning with a pre-trained BART language model and GPT-4 refinement to set a new state-of-the-art on the ZuCo benchmark.

End-to-end EEG-to-Text pipeline: raw EEG signals are processed by the Brain Module and decoded into text by the Language Module, with optional GPT-4 refinement.

Figure 1. Overview of our approach at inference time. Raw word-level EEG signals are processed by the Brain Module to extract subject-dependent representations, which are then decoded into natural language by a fine-tuned BART model. An optional GPT-4 refinement step further improves fluency. In this example, the model correctly recovers key entities such as "member of the Bush family" and "President George W. Bush" from brain activity alone.

Abstract

Decoding language directly from brain activity is a long-standing goal at the intersection of neuroscience and artificial intelligence. While recent work has shown that pre-trained language models can decode open-vocabulary EEG signals acquired through non-invasive brain-computer interfaces, critical questions remain unanswered: How should EEG signals be embedded in the context of language models? and How does inter-subject variability affect decoding accuracy? Furthermore, existing evaluation metrics are predominantly syntactic and fail to capture whether the decoded output is genuinely comprehensible to humans.

We address these challenges with an end-to-end deep learning architecture that makes three key contributions:

  1. A subject-dependent representation learning module that encodes raw EEG signals into language-aligned embeddings, combined with a fine-tuned BART language model for text generation and a GPT-4 refinement module for improved fluency and coherence.
  2. A comprehensive evaluation protocol incorporating BERTScore to assess semantic similarity beyond surface-level n-gram overlap, offering a measure closer to human judgment of comprehensibility.
  3. An extensive ablation study that isolates and quantifies the contribution of each architectural component, providing a clear roadmap for future brain-to-text research.

Evaluated on two publicly available datasets — ZuCo v1.0 and v2.0 — comprising EEG recordings from 30 subjects during natural reading tasks, our approach achieves new state-of-the-art results across all metrics.

Method

Our architecture operates in two training stages designed to bridge the gap between neural signals and natural language.

Stage 1 — Brain Module Training. We learn to project raw word-level EEG signals into the BART token embedding space. The Brain Module consists of four components: (i) a bidirectional GRU that captures temporal dynamics of variable-length EEG segments, (ii) a learnable subject-specific layer that adapts to individual neural signatures, (iii) a multi-layer Transformer encoder with multi-head self-attention that captures long-range dependencies across the EEG sequence, and (iv) a projection MLP that maps representations into BART's embedding space, optimized via MSE loss against frozen BART token embeddings.

Stage 2 — Language Model Fine-tuning. With the Brain Module frozen, we fine-tune the BART encoder-decoder to generate coherent text sentences from the aligned EEG representations using cross-entropy loss. At inference, an optional GPT-4 post-processing step refines the output for improved grammatical correctness and semantic coherence.

Brain Module architecture: bidirectional GRU, subject layer, Transformer encoder, and MLP projection

Figure 2. Brain Module architecture. Raw word-level EEG signals (variable length, 105 channels) pass through four components: a learnable EEG feature block with bidirectional GRU, a subject-dependent representation layer, a multi-layer Transformer encoder, and a projection MLP that maps to BART's 1024-dimensional token embedding space.

Two-stage training pipeline

Figure 3. Two-stage training pipeline. Stage 1 (left): the Brain Module learns to align EEG representations to frozen BART embeddings via MSE loss. Stage 2 (right): the BART encoder-decoder is fine-tuned to generate text from aligned representations using cross-entropy loss.

Results

We evaluate on the ZuCo benchmark (v1.0 + v2.0) using the same teacher-forcing evaluation protocol as prior work for fair comparison. Our model establishes new state-of-the-art results, outperforming all prior methods on every metric.

State-of-the-Art Comparison

Method BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-1-F BERTScore-F
EEG-To-Text [Wang et al.] 40.10 23.10 12.50 6.80 30.10 50.66
DeWave [Duan et al.] 41.35 24.15 13.92 8.22 30.69 —
Ours 42.75 25.90 15.66 9.56 33.28 53.86
Ours + GPT-4 40.87 24.43 14.53 8.82 32.61 55.34

Table 1. Comparison on ZuCo v1.0 & v2.0 test set. Our model surpasses all prior work across metrics. GPT-4 refinement trades syntactic similarity for improved semantic coherence (BERTScore +1.48).

Qualitative Examples

Beyond aggregate metrics, we inspect individual decoded sentences to assess comprehensibility. Our model correctly recovers named entities unseen during training (e.g., "George W. Bush", "Puerto Rico") and produces significantly fewer grammatical errors and repetitions compared to prior work. GPT-4 refinement further improves readability while preserving semantic content.

Exact match Semantic match Error
Example 1 — Natural Reading, ZuCo v1.0
Ground Truth
He is a prominent member of the Bush family, the younger brother of President George W. Bush…
EEG-To-Text
was a former member of the American family, and son brother of President George W. Bush…
Ours + GPT-4
He was a member of the American Bush family, brother of President George W. Bush…
Example 2 — Natural Reading, ZuCo v2.0
Ground Truth
Raymond Arrieta (born March 26, 1965 in San Juan, Puerto Rico) is considered by many to be one of Puerto Rico's greatest comedians.
EEG-To-Text
mond wasaga, 19 in 17, 18) New Francisco, Puerto Rico) is a one many to be the of the Rico's greatest poets.
Ours + GPT-4
Ramon Wasaga (born April 17, 1946, in New Francisco, Puerto Rico) is one of the many to be considered as one of the most prominent artists of Puerto Rico.
Example 3 — Sentiment Reading, ZuCo v1.0
Ground Truth
It's not a particularly good film, but neither is it a monstrous one.
EEG-To-Text
was a a bad good story, but it is it bad bad. one.
Ours + GPT-4
It's a bad good movie, but is it a bad one.

Table 2. Decoded EEG-to-Text examples on ZuCo unseen test sentences. Our model produces more coherent and accurate decodings, with GPT-4 refinement further reducing grammatical errors while preserving semantic content.

Ablation Study

We isolate the contribution of each architectural component by systematically removing one module at a time from the full model. The results reveal a clear hierarchy of importance:

Configuration BLEU-1 ROUGE-1-F BERTScore-F
Full Model 42.75 33.28 53.86
− BART Fine-tuning 28.50 (−14.25) 21.44 (−11.84) 43.10 (−10.76)
− Brain Transformer Encoder 35.51 (−7.24) 27.98 (−5.30) 48.91 (−4.95)
− Subject Layer 41.32 (−1.43) 31.85 (−1.43) 52.50 (−1.36)
− Language Alignment (MSE) 42.44 (−0.31) 32.92 (−0.36) 53.20 (−0.66)
+ GPT-4 Refinement 40.87 (−1.88) 32.61 (−0.67) 55.34 (+1.48)

Table 3. Ablation study. Each row removes one component from the full model. BART fine-tuning is the single most impactful factor (−14.25 BLEU-1); the Brain Transformer and subject layer provide complementary gains. GPT-4 trades syntactic precision for semantic coherence.

BART fine-tuning is the most critical component — without it, performance drops by 14.25 BLEU-1 points, confirming that language model adaptation is essential for coherent sentence generation from neural signals. The Brain Transformer Encoder contributes a 7.24-point BLEU-1 gain, validating that multi-head self-attention effectively captures long-range dependencies in EEG sequences. The subject-dependent layer provides a consistent 1–1.5% improvement across all metrics — while modest in absolute terms, this gain directly validates our hypothesis that modeling inter-subject neural variability is beneficial for EEG decoding. Interestingly, GPT-4 refinement reduces syntactic metrics (BLEU) while substantially boosting BERTScore (+1.48), indicating that it restructures sentences for better semantic coherence at the cost of exact n-gram overlap.

Subject-Dependent Representations

A core design principle of our approach is that EEG signals are inherently subject-dependent — the same thought or reading experience produces different neural patterns across individuals. To validate that our Brain Module successfully captures this variability, we visualize the learned EEG representations using t-SNE and compare them against the pre-calculated features used by prior work.

t-SNE of pre-calculated EEG features — poorly clustered

(a) Pre-calculated EEG features [Wang et al.]. Representations are scattered with minimal subject-level structure.

t-SNE of our learned EEG representations — clearly clustered by subject

(b) Our learned representations. Clear per-subject clustering demonstrates that the Brain Module captures individual neural signatures.

Figure 4. t-SNE visualization of EEG sentence embeddings on the training set. Each color represents a different subject; the red triangle marks the same sentence across subjects. Our representations (b) show markedly better subject-level clustering than pre-calculated features (a), confirming effective modeling of inter-subject variability.

BibTeX

@article{amrani2024deep,
  title     = {Deep Representation Learning for Open Vocabulary
               Electroencephalography-to-Text Decoding},
  author    = {Amrani, Hamza and Micucci, Daniela and Napoletano, Paolo},
  journal   = {IEEE Journal of Biomedical and Health Informatics},
  year      = {2024},
  doi       = {10.1109/JBHI.2024.3416066},
  publisher = {IEEE}
}