Prompting the Mind: EEG-to-Text Translation with Multimodal LLMs and Semantic Control

Prompting the Mind: EEG-to-Text Translation with Multimodal LLMs and Semantic Control Mohammed Salah Al-Radhi1[0000−0003−3094−6916], Sadi Mahmud Shurid1[0009−0007−1594−2614], and Géza Németh1[0000−0002−2311−4858] Department of Telecommunications and Artificial Intelligence, {malradhi,nemeth}@tmit.bme.hu, [email protected] Abstract. We present Prompting the Mind (PTM), an extended EEG-totext translation framework that combines large language models (LLMs) with multimodal alignment to decode human brain signals into natural language. Our system follows a multi-stage pipeline: an EEG encoder first transforms raw neural activity into discriminative embeddings; these are then mapped into a shared vision-language semantic space using CLIP-based cross-modal alignment. Finally, a general-purpose base LLM, DeepSeek-7B-Base, generates descriptive text conditioned on the EEGderived representations through structured prompting. We evaluate the framework on a publicly available EEG-image dataset, comparing its performance with chance-level and alignment-only baselines as well as an instruction-tuned LLM (Mistral-7B). Results on BLEU, METEOR, ROUGE-L, and BERTScore show that while instruction-tuned models yield higher token overlap, our prompt-conditioned base LLM produces shorter, more semantically faithful outputs that better align with the original brain signals. Qualitative examples highlight this trade-off and the practical value of structured prompting for non-invasive neural decoding. All code, prompt templates, and configuration files are shared 1 to promote reproducibility and future extensions of open-weight frameworks for brainto-text communication. 1 Introduction Brain-to-text and brain-to-speech decoding has become a significant research direction within brain-computer interface (BCI) studies, aiming to provide natural communication pathways for individuals with physical or speech impairments [ 1 , 2 , 3 , 4 ]. Non-invasive neural recording methods such as electroencephalography (EEG) offer a practical trade-off between safety and accessibility, yet they remain challenging due to their low signal-to-noise ratio, limited spatial resolution, and high inter-subject variability [ 5 ]. Despite these challenges, recent advances 1https://github.com/Sadi-Mahmud-Shurid/PTM Keywords: Brain-Computer Interface ·Neural Speech Decodin in deep learning and natural language processing have led to renewed interest in mapping brain signals directly to natural language text [6,7]. Large language models (LLMs) have demonstrated notable capabilities in generating fluent, contextually rich text across various domains [ 8 , 9 ]. However, applying LLMs to decode neural signals remains largely unexplored, particularly in non-invasive EEG settings. Prior work in neural speech decoding systems has mostly focused on invasive recordings (e.g., ECoG) and task-specific decoders [ 10 ], while EEG-based studies have generally relied on classical classification pipelines or shallow language models [11,12]. In this work, we propose Prompting the Mind (PTM), an extended EEG-totext translation framework that combines the generalization capacity of LLMs with the representational power of multimodal alignment. Our approach builds upon the pipeline in [ 13 ]: first, an EEG encoder transforms raw neural activity into a meaningful embedding space; second, a vision-language alignment stage maps these embeddings into a shared semantic space using CLIP-based supervision [ 14 ]; and third, a general-purpose LLM, DeepSeek-7B-Base [ 15 ], is adapted to generate descriptive text conditioned on the EEG-derived representations through structured prompting. We evaluate the framework on a publicly available EEG-image dataset [ 16 ], comparing our model against chance-level baselines and prior instruction-tuned LLMs such as Mistral-7B [ 17 ]. We report standard machine translation and semantic similarity metrics, including BLEU, METEOR, ROUGE-L, and BERTScore [18], and provide qualitative examples to illustrate the effectiveness of prompt-based semantic control. Our results demonstrate that aligning brain signals with vision-language representations improves the relevance and quality of generated text, highlighting the potential of LLMs for non-invasive neural speech decoding. In this work, our main contributions are: – We extend the framework in [ 13 ] by systematically testing its performance with a base LLM and analyzing its generalizability beyond instruction-tuned models. – We design and test structured prompting strategies to enable semantic control in EEG-to-text generation using a base LLM, verified through controlled baselines and output comparison. – We provide an open-source implementation, prompt templates, and evaluation scripts to support reproducibility and further research in non-invasive brainto-text communication. The remainder of this paper is organized as follows: Section 2 discusses related work in EEG-to-text decoding, multimodal learning, and LLM prompting. Section 3 details our methodology and system architecture. Section 4 describes the experimental setup, including datasets and evaluation metrics. Section 5 presents quantitative and qualitative results. Finally, Section 6 concludes the paper and outlines future directions. 2 Related Work Decoding natural language directly from neural activity has long been an important goal in brain-computer interface (BCI) research. While significant progress has been made with invasive signals such as electrocorticography (ECoG) for speech or handwriting decoding [ 10 , 6 , 7 , 21 , 22 ], these approaches remain clinically limited. In contrast, non-invasive methods such as electroencephalography (EEG) are safer and more accessible [ 1 , 2 ], but their low signal-to-noise ratio and poor spatial resolution make free-form text generation highly challenging. Previous EEG-based systems have mostly focused on classification tasks, such as recognizing isolated phonemes, imagined speech commands, or visual stimuli [ 19 , 20 ]. While some studies have explored phrase-level classification or keyword spotting [ 23 , 24 , 25 ], the generation of natural language text directly from EEG signals remains largely underexplored. In this context, our work aims to address this gap by combining a robust EEG encoder with large language models (LLMs) to move beyond traditional classification toward open-ended, semantically meaningful text generation. One promising direction that complements EEG decoding is multimodal learning. By combining EEG data with additional modalities, such as visual or linguistic features, researchers aim to enhance the representational power and robustness of neural decoding models [ 26 ]. For example, Spampinato et al. [ 27 ] demonstrated the feasibility of classifying visual stimuli based on EEG responses, laying the groundwork for cross-modal supervision. More recently, studies have explored mapping EEG signals to visual semantic spaces using CLIP [ 14 ], leveraging pre-trained image-text embeddings to anchor noisy EEG data in a richer context. Such multimodal alignment can help bridge the gap between brain signals and natural language output by grounding neural representations in human-interpretable semantic spaces. However, most prior work focuses on classification or retrieval tasks, with few efforts extending this alignment toward free-form language generation. At the same time, large language models have rapidly advanced the state of the art in natural language processing (NLP), achieving impressive results in tasks ranging from text generation and summarization to instruction following and conversation [ 8 , 28 ]. Open-weight models such as LLaMA [ 8 ] and Mistral [ 17 ] have made it feasible to experiment with LLMs for specialized applications, including low-resource and cross-modal settings. In the BCI domain, however, the use of LLMs for direct neural-to-text translation remains largely unexplored. Prior work on neural speech decoding has primarily relied on task-specific decoders or shallow language models integrated with invasive signals such as ECoG [ 10 , 6 ], while non-invasive EEG studies have generally focused on classification or retrieval tasks [29], rather than free-form text generation. Recent advances in prompt engineering and instruction tuning have shown that pre-trained vision-language representations can be combined with LLMs through prompt design or adapter modules, improving generation quality without retraining the entire model [ 30 ]. While this is well established in general NLP, its application to non-invasive BCI pipelines remains underexplored. Overall, although neural decoding, multimodal alignment, and LLMs have each seen significant progress individually, there remains a clear need for reproducible frameworks that bridge non-invasive brain signals and open-weight language generation models while ensuring that alignment and semantic control work together effectively. These open challenges motivate the development of frameworks that combine an EEG encoder, vision-language alignment, and prompt-conditioned LLM generation in an end-to-end pipeline, which we propose in this paper. 3 Methodology In this section, we describe our extended EEG-to-text framework, Prompting the Mind (PTM), which builds upon the open-source baseline [ 13 ] to investigate its generalizability when using an open-weight base LLM. Our focus is on analyzing the impact of structured prompting strategies and LLM instruction-tuning status on output quality. The architecture consists of three key components: an EEG encoder that transforms raw neural signals into discriminative embeddings, a cross-modal alignment module that projects these embeddings into a shared vision-language representation space, and a prompt-conditioned base LLM that generates natural language text from the aligned EEG features. We preserve the encoder and alignment modules with minimal changes, but introduce novel prompt templates and control experiments to systematically compare an instruction-tuned LLM (Mistral-7B) with a base LLM (DeepSeek-7B-Base). We also formalize the core training and inference processes with clear objective functions to improve reproducibility and transparency. Figure 1 illustrates the overall PTM framework, showing the three stages and their connections. The following subsections describe each component in detail. 3.1 EEG Data Preprocessing and Representation The first stage of our framework processes raw EEG signals to generate meaningful embeddings for downstream alignment and text generation. Following common practice for visual-evoked EEG datasets [ 27 ], each raw EEG trial XEEG is recorded with Cchannels over Ttime points: XEEG ∈RC×T(1) We apply standard preprocessing steps, including bandpass filtering (1–40 Hz), normalization, and artifact rejection to improve the signal-to-noise ratio. Each trial is segmented relative to stimulus onset and downsampled if needed to reduce computational complexity. The preprocessed EEG is then encoded using a convolutional neural network (CNN)-based feature extractor fenc ( · ), which maps the multichannel time series into a fixed-length representation: z=fenc(XEEG)∈Rd(2) Fig. 1. Overview of the Prompting the Mind (PTM) framework. Here, d denotes the dimensionality of the latent EEG embedding. The encoder architecture follows the open-source baseline [ 13 ] without major modification to ensure consistency for our comparative analysis. Encoder weights are trained jointly with the alignment module during the cross-modal alignment phase described in Section 3.2. 3.2 Cross-Modal Alignment with Vision-Language Embeddings To mitigate the low signal-to-noise ratio and limited spatial resolution of EEG, we adopt a cross-modal alignment stage that anchors EEG embeddings in a shared semantic space with visual features. Following prior work [ 27 , 14 ], we use CLIP [14] to provide pre-trained vision-language embeddings for supervision. Given a batch of EEG trials and their associated image captions, we encode each visual stimulus using CLIP to obtain an image embedding vi∈Rd . The goal is to align the EEG-derived embedding zi with its corresponding visual embedding viusing a symmetric InfoNCE-style contrastive loss: Lalign =−1 N N X i=1 "log exp sim(zi, vi)/τ PN j=1 exp sim(zi, vj)/τ + log exp sim(vi, zi)/τ PN j=1 exp sim(vi, zj)/τ#,(3) where sim(·,·)denotes cosine similarity, τis a temperature hyperparameter, and N is the batch size. This objective encourages each EEG embedding to match its paired image embedding while pushing apart mismatched pairs within the batch. During training, the CLIP visual encoder is frozen to preserve its pre-trained representation, while the EEG encoder is optimized jointly with the alignment module. After training, only the EEG encoder is used for downstream generation, ensuring that inference requires EEG signals only. 3.3 Prompt-Conditioned Base LLM Adaptation After cross-modal alignment, each EEG trial is represented by a latent embedding z∈Rd that reflects its semantic relationship to the visual stimulus. To generate descriptive text, we adapt an open-weight base large language model (LLM), DeepSeek-7B-Base [15], which has not been instruction-tuned. Instruction tuning refers to the process of further fine-tuning a pre-trained LLM on a large collection of curated instruction–response pairs so that it can reliably follow natural-language requests. For example, an instruction-tuned model such as Mistral-7B-Instruct is trained to handle prompts like “Describe the scene in this image” or “Summarize the following text,” producing coherent and instruction-compliant outputs without additional task-specific engineering. In contrast, a base model such as DeepSeek-7B-Base has only been pre-trained on large-scale text corpora without this alignment stage, and therefore relies more heavily on carefully crafted prompts to elicit the desired behaviour. Instruction tuning is considered costly because it requires (i) collecting or licensing large, diverse instruction datasets, (ii) substantial computational resources to fine-tune billions of parameters, and (iii) rigorous quality control to prevent overfitting or performance regressions. By comparing the instructiontuned Mistral-7B with the prompt-conditioned DeepSeek-7B-Base under identical EEG-conditioning, we assess whether structured prompting alone can yield competitive results without incurring the high financial and computational cost of instruction tuning. We design structured prompt templates that incorporate the aligned EEG representation z as an additional conditioning signal. Specifically, for each input, the final prompt P concatenates a reusable natural language template with a linear projection of the EEG embedding: P= [p;Wpz],(4) where p is a fixed prompt prefix, Wp∈Rk×d is a learned projection matrix that maps the EEG embedding to the LLM embedding dimension k , and [ · ; · ]denotes concatenation. During training, the base LLM is frozen and the projection parameters Wp are optimized to align the EEG-conditioned prompt with the target captions. The objective is to minimize the negative log-likelihood of the ground truth text ygiven the prompt-conditioned input: Lgen =−E(z,y)hlog p(y|P)i.(5) We experiment with several prompt templates, varying in length and instruction style, to test their impact on semantic relevance and fluency. At inference time, only the EEG-derived embedding and the chosen template are needed to generate free-form text. We compare this base LLM approach against an instruction-tuned model (Mistral-7B [ 17 ]) under the same conditions to evaluate the role of pre-training and prompting in non-invasive neural-to-text decoding. A detailed comparison of methodological similarities and differences with the baseline in [11], along with discussion of how our design achieves competitive performance without fine-tuning, is provided at the start of Section 5. 3.4 End-to-End Training and Inference Pipeline The full PTM framework integrates the EEG encoder, cross-modal alignment module, and prompt-conditioned LLM into a coherent system. During training, the EEG encoder is optimized jointly with the cross-modal alignment stage to learn embeddings that are semantically consistent with vision-language representations. The prompt-conditioned LLM stage uses these aligned embeddings to generate descriptive text. The overall training objective combines the alignment loss (Eq. 3) and the generation loss (Eq. 5): Ltotal =Lalign +λLgen,(6) where λ is a scalar hyperparameter that balances the two objectives. In practice, we tune λ to ensure that the EEG embeddings remain well-aligned while still producing fluent text outputs. At inference time, the image input is not required. The trained EEG encoder maps unseen EEG signals to embeddings, which are then projected into the prompt template for the base LLM. The LLM generates free-form text conditioned solely on the EEG-derived representation and the reusable prompt. This setup demonstrates the feasibility of non-invasive brain-to-text translation without reliance on external visual stimuli during deployment. Our implementation supports reusable prompt templates, batch inference, and ablation toggles for alignment and prompt conditioning. These design choices make our proposed approach straightforward to extend or adapt to other noninvasive neural recording modalities. 4 Experimental Setup and Evaluation 4.1 Dataset and Preprocessing We evaluate our framework on the publicly available visual EEG dataset originally presented by Spampinato et al. [ 27 ]. This dataset consists of EEG signals recorded from subjects viewing a set of 40 object categories spanning 2,000 images in total. EEG was recorded from 128 channels using the Biosemi ActiveTwo system at a sampling rate of 128 Hz. Each image is displayed for 0.5 seconds with an interstimulus interval of 0.5 seconds. Following prior work, each EEG trial is segmented relative to stimulus onset and preprocessed to reduce noise and artifacts. We apply bandpass filtering in the 1–40 Hz range, channel-wise z-score normalization, and artifact removal to improve the signal-to-noise ratio. Trials are downsampled to 128 Hz if needed, and epochs are cropped to a fixed window length that captures the relevant visual evoked potentials. For training the cross-modal alignment, we pair each EEG trial with its corresponding image and its CLIP-encoded embedding. During inference, only the EEG signal and reusable prompt template are required to generate descriptive text, as the image and CLIP supervision are used solely for training (see Fig. 1). 4.2 Model Configurations The EEG encoder module follows a lightweight CNN-based architecture. It consists of three convolutional layers with batch normalization and ReLU activation, followed by a fully connected layer to produce a d-dimensional embedding (d= 512). For the cross-modal alignment stage, we use the pre-trained CLIP-ViT-B/32 model [ 14 ] as the vision-language supervision source. The CLIP visual encoder is frozen during training, and the EEG encoder is trained to align its embeddings with the CLIP image embeddings in a shared semantic space. The prompt-conditioned LLM module uses DeepSeek-7B-Base [ 15 ] as the base model. We do not fine-tune the LLM weights; instead, a lightweight projection layer maps the EEG embedding to the LLM embedding space, which is then combined with reusable prompt templates for generation. For comparison, we also evaluate an instruction-tuned variant (Mistral-7B [ 17 ]) under the same input conditions. The entire architecture is modular, allowing for easy replacement of the encoder, alignment module, or LLM for further experiments. The balancing hyperparameter λfor the combined loss is described in Section 3. 4.3 Training Details All experiments are conducted on a single NVIDIA RTX 3090 GPU using PyTorch. The EEG encoder and the projection parameters Wp for the prompt-conditioning are trained jointly with the cross-modal alignment module. The LLM weights (DeepSeek-7B-Base and Mistral-7B) remain frozen throughout. For optimization, we use the Adam optimizer with an initial learning rate of 1 × 10 −4 for the encoder and 5 × 10 −5 for the projection layer. A batch size of 32 is used for alignment training. The temperature parameter τ in the InfoNCE loss is set to 0.07. The balancing factor λ for the combined loss (Eq. 6) is set to 1.0, which was found to maintain a good trade-off between alignment and generation objectives. Each training run lasts for up to 50 epochs with early stopping based on validation BERTScore to prevent overfitting. The CLIP encoder remains frozen to preserve its pre-trained semantic space, and image stimuli are used only during training. During inference, only the EEG encoder and prompt-conditioned LLM are used to generate text. All hyperparameters and configurations are shared in the open-source repository to ensure reproducibility. 4.4 Evaluation Metrics We evaluate the quality of the generated text using a combination of standard machine translation and semantic similarity metrics. Specifically, we report: – BLEU [ 31 ]: Measures the precision of n-gram overlaps between generated text and reference captions, commonly used for translation tasks. – METEOR [ 32 ]: Considers both precision and recall, with additional synonym matching and stemming, providing a more robust measure of fluency. – ROUGE-L [ 33 ]: Computes the longest common subsequence between generated and reference texts, capturing overall sentence-level similarity. – BERTScore [ 34 ]: Uses contextual embeddings from a pre-trained BERT model to measure semantic similarity between generated and reference sentences, going beyond surface-level token overlap. In addition to these quantitative metrics, we present selected qualitative examples to illustrate the practical differences between baseline variants and the impact of structured prompting. We follow a consistent cross-subject evaluation split for all experiments to assess generalization performance. 4.5 Baseline and Evaluation Protocol To assess the impact of prompt conditioning and LLM instruction-tuning, we design comparative baselines that isolate key components of our framework. Our primary baselines include: – Chance-level baseline: EEG embeddings are replaced with random vectors to measure whether the model produces meaningful text beyond random noise. – Aligned-only baseline: The cross-modal alignment is used without prompt conditioning to test the contribution of the structured prompts. 16 Mohammed Salah Al-Radhi, Sadi Mahmud Shurid, and Géza Németh 34. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evaluating text generation with BERT. In: International Conference on Learning Representations (ICLR), pp. 1–43 (2020). http://www.openreview.net/pdf?id=SkeHuCVFDr 35. Al-Radhi, M.S., Németh, G., Gerazov, B.: MiSTR: Multi-Modal iEEG-to-Speech Synthesis with Transformer-Based Prosody Prediction and Neural Phase Reconstruction. In: Proceedings of Interspeech 2025, Rotterdam, The Netherlands, pp. 1–5 (2025).