Open reviews
August 11, 2026 — Open review of Yoon and colleagues:
Yoon, S., Avansino, D. T., Madugula, S., Levin, A. D., Fan, C., Abramovich Krasa, B., … & Willett, F. R. (2026). Neural decoding of speech using deep neural ensembles. bioRxiv. DOI
Yoon and colleagues test a family of deep ensembling methods for brain-to-text decoding in a speech-BCI clinical trial participant. These methods have been shown to improve performance in offline benchmark datasets, but have not been adequately vetted for real-time performance “in the wild.” They explore several different parameters related to computational resources, time constraints, training data volume, and introduce a pseudoensembling method that achieves moderate gains with less computing overhead. We get a lot of neat practical insights about which factors most affect latency and overall performance. This is a fairly straightforward and tightly scoped methodological paper (despite the heavy-duty engineering effort detailed in the Methods section). The writing is clear, the figures are well designed, and the results make sense. The gains with this method (reduced word error rates) are not huge, but I think this is a useful exercise for identifying where future work should focus. Most of my comments are relatively minor clarification questions and could be addressed in a minor revision.
My top level comment is that, in reading this, I don’t feel like I really got an intuition as to why ensembling is helpful in this context. How does “combin[ing] predictions from multiple independently trained models” actually help? I assume it mitigates the noisiness and limited samples of the neural recordings somehow? Just seems like a single model should be able to ultimately achieve the same performance. I assume there’s nothing particularly special about the models that provide the best hypotheses other than a different random initialization? Any intuition the authors could give readers as to why ensembling makes a difference (perhaps just from machine learning theory) would be helpful.
I was a bit confused which error rate from this real-world application should be compared to the previously reported benchmark results. Am I correct in understanding that the error rates from this real-time experiment are much higher than the error rates from the benchmark datasets (e.g. Figure 1d)? Is the 26% you report in the abstract directly comparable to the ~5–7% performance we see in Figure 1d? If there really is a big decrement in performance like this, can you provide a little further explanation as to why this might be the case? Is it mostly due to overfitting of the benchmark data?
Line 121: You mention that the ensemble predictions were submitted to “a ‘merger’ LLM [that] produced a coherent final prediction.” Can you add an additional sentence here to give us a better sense of what this merger LLM is actually doing without having to dig through the methods?
Line 196: I was confused about the mention of “unquantized” here. My understanding from the Methods section was that quantization should speed things up? But here you indicate a speedup without quantization?
Line 247: “intelligence”—you mean “intelligibility”?
Line 515: “To adapt to nonstationarities in neural features, a day-specific trainable input layer was added before the RNN.” Can you say a little bit more about how this day-specific layer is trained? You include a little more about this in the “RNN online recalibration” and “Real-time evaluation session design” sections below, but I still had a hard time following the logic. Maybe just up-fronting the real-time session design a bit more would help resolve some of this confusion?
Line 542: “We used a pruned version of the original 125,000-word 5-gram model to accelerate real-time inference.” Can you help me understand the implications of this a bit more? Is 5-gram just an arbitrary/convenient choice? Will this approach scale to open-ended speech?
Line 560: “We also appended the immediately preceding hypothesis set to the prompt as an in-context learning (ICL) example.” Do you supply the preceding hypothesis set with the ground-truth, or just the hypotheses? Am I correct in understanding that the preceding hypothesis set should be essentially unrelated to the present hypothesis set given the experimental design of the sentence prompts? I’m struggling to understand why this helps. If it helps because the LLM benefits from having some (unrelated) examples of hypotheses, why not prompt with more than just the previous set?
I have what may be a naive question: I’m looking at the plots in Figure 6 a–d where it seems like the amount of fine-tuning data yields increasing gains. Do the authors think it could be feasible to learn some kind of transformation or alignment across individuals so that they could leverage data from other subjects to improve performance in the target patient? I understand that any particular implanted grid will sample a unique subset of neural activity in a unique brain. However, if there’s some latent relational structure among phonemes or speech actions encoded at the neural population level, these latents could be shared in part across individuals. For example, in the fMRI literature, and more recently in ECoG, we use methods like hyperalignment to find a transformation from one subject’s idiosyncratic neural activity space to another subject’s neural activity space (e.g., Haxby et al., 2020; Bhattacharjee et al., 2026). In perception, we typically use neural activity synchronized to naturalistic stimulus like a movie or spoken story to learn the alignment, which provides strong generalization to other tasks. However, I think using slightly fancier modeling, you could learn an alignment across brains based on spontaneous, unconstrained speech. If this were possible, you could potentially build a large normative sample of patients (or even healthy participants), all aligned to some shared latent space of neural activity during speech. When a new patient comes in, you would need to work them through some task to learn a decent alignment to the shared space—but once you have that, you would be in a position to train on other subjects and transfer to the target patient. Anyway, sorry for the tangent; just curious whether something like this might be feasible in the future.
References:
Bhattacharjee, A., Zada, Z., Wang, H., Aubrey, B., Doyle, W., Dugan, P., … & Nastase, S. A. (2026). Aligning brains into a shared space improves their alignment with large language models. Nature Computational Science, 6(2), 169–178. DOI
Haxby, J. V., Guntupalli, J. S., Nastase, S. A., & Feilong, M. (2020). Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. eLife, 9, e56601. DOI