Open reviews
September 19, 2026 — Open review of Ismail and colleagues:
Ismail, T., Chavez, A. G., Yan, X., Zhu, H., Franch, M., Belanger, J., … & Hayden, B. Y. (2026). Estimation of neuronal tuning for word meaning from passively recorded naturalistic speech. bioRxiv. DOI
Ismail and colleagues develop a pipeline for processing a massive dataset of 24/7 intracranial recordings and evaluate encoding/decoding models semantic content across many days worth of speech/listening. This is a somewhat unusual paper in that its main contribution is to extend an established form of semantic encoding to a real-world dataset of unconstrained language use. I think this is a genuinely interesting scientific problem—one that is often undervalued in experimental work—and is critical for ultimately developing technologies that work at scale in the real world. (Not to mention this is an incredible dataset!) The authors compare their automated pipeline for large-scale data analysis against ground-truth data using manual alignment, and compare modeling results for the large-scale incidental data against two smaller datasets. The methodology is a tour de force and will provide useful guidelines for others trying to do similar work. My main concern is with the analyses of drift over the course of days, which I try to unpack in the first three comments below. In some places, the writing and/or methodology is not very clear, so I include a list of relatively minor questions/corrections as well.
Major comments:
My main concern has to do with the analyses of data volume and generalization over time (drift), which I split into three related comments. First, I think the authors should be super explicit about the cross-validation procedure for each analysis. For example, at line 656, you say “We used 5-fold outer cross-validation to evaluate held-out performance.” Can you say more about how the heard and spoken speech was split for cross-validation? Did you use a random split or temporally contiguous cross-validation folds? For example, completely random splits, where the test samples are randomly spliced out of the training samples, could allow autocorrelation (in neuronal signal and/or GPT embeddings) to drive prediction performance in the test set (Hadidi et al., 2026). I’m also curious how the splits might relate to speaking versus listening, conversation boundaries, and days in the clinic. Critically, for each of the analyses in Figure 4G, Figure 5, and Figure 6, it would be helpful to know how the cross-validation procedure is set up. For example, for the encoding models, you find that “models trained on individual days performed significantly better than models trained with data from all days,” whereas for the decoding models, you find that “single-day model performance is 9% (3.5%) worse than full multi-day dataset decoding.” But what is the cross-validation structure in these comparisons? Is the number of training samples and the number of test samples changing / increasing as you go from one day to all days? I think if you wanted to strictly compare the effect of training volume, you would want to vary the training volume (e.g., 1 day, 2 days, 3 days, etc) but keep the test set(s) strictly identical (in both volume and content). Similarly, for the drift analyses, we really need to know what’s in the training set and what’s in the test set. I was expecting, for example, that you train a model on one day’s worth of data, and then see how well that fitted model generalizes to other neighboring days (e.g., a temporal generalization analysis; King & Dehaene, 2014)—but it’s hard to tell if that’s what I’m actually looking at.
Second, I’m a little bit suspicious that the drift in encoding model performance is so catastrophic; I mean the encoding model performance crashes to below null performance on every single day other than the training day (Figure 4G, Figure 6A). This negative “below null” performance is on the same scale as the positive above null performance for the same day—yikes! The authors mention that “R2 scores are normalized in reference to null performance, so decreases below zero are meaningful”—but I don’t really see how the full model could meaningfully predict worse than the null submodel, other than serious issues with overfitting and/or data processing. If I saw this result, I would immediately assume that there’s some kind of “bug” in my encoding pipeline. A few things to check: (1) Is your inner cross-validation loop for hyperparameter optimization using a splitting procedure and outcome metric that matches your outer generalization test? (2) Is your hyperparameter optimization sampling a broad enough range of penalty values? For example, you might be hitting the ceiling of available penalty terms in your grid, which will under-penalize the model, increase overfitting, and reduce generalization. (3) Are you matching how you process both your X predictor matrix and your Y target matrix prior to model fitting and evaluation/testing? For example, if you’re z-scoring the columns of your training X, you should be applying the same already-fitted transformation to your test X. If you’re fitting PCA on the X matrix in the training set of each fold (best practice), you should be projecting the test embeddings onto the PCA solution derived from the training set. If you’re z-scoring the columns of your test set Y for the same day, make sure the test sets from other days are being z-scored. If any one of these switches in the encoding pipeline is flipped incorrectly, it could yield the crash in model performance across days we see in Figure 6A for uninteresting reasons. The authors also mention “slight drift in the semantic tuning functions” (i.e., encoding weight vectors) at each unit, but in Figure 6F we see the hour-by-hour cosine similarities, on average, stay at almost precisely 1—which means they are orthogonal. Weird! Again, this seems highly unlikely to me and could result from a pathology in the modeling pipeline. We need to be extremely careful about a result like this because it could be used to argue that encoding models are not viable for longitudinal neuronal recordings (which I suspect is not the case).
Third, when we arrive at the Discussion, the authors try to build a story as to why we see this pattern of drift. I didn’t find this very convincing. Decoding seems robust (relative to encoding), but the decoding task could also just be an easier task (e.g., predict ten discrete categories versus predict precise up-and-down fluctuations in single-unit signal). To explain the drift observed for encoding models, the authors say: “Thus, the same channel across days may not correspond to the same local neural source, making the apparent channel–stimulus relationship unstable even without any change in the underlying neural representation.” But this logic will also apply to the decoding analyses! The authors write: “Population decoding models are expected to be more robust to this form of drift. Unlike single-unit encoding models, a decoder is not required to preserve a fixed relationship between a semantic feature and any one neuron. Instead, it can exploit redundancy across units, down-weight unstable or weakly informative units, and read out information from population dimensions that remain predictive over time.” There are a couple problems with this. First, the coordinate axes (i.e., units, electrodes, voxels) of the decoder must also remain aligned for the decoder to generalize. Granted, the decoder is aggregating across correlated features, so if some features drift, it may still maintain some level of performance based on features that haven’t drifted. But the more features that drift, the worse it should get; and if all features drift or rotate or anything else, the decoder should crash completely. Second, the decoder can only learn to downweight features within its training set—it cannot learn to downweight features that are “unstable” in the move from training to test. Similarly, it can learn to emphasize features that are “predictive over time” within its training set, but has no way to know what features might be predictive for a test set days later. The authors mention “while the relationships among population activity patterns remain relatively stable and stereotype”—but this logic only applies to representational similarity analysis (RSA), not to decoding. You could use RSA to obtain a (dis)similarity matrix within a particular day, the underlying features could then entirely shift from that day to the next, and you could still recover the same similarity matrix on the following day. But that logic does not apply to decoding generalization, unless you retrain the decoder from scratch on each day. For the decoder to actually generalize across days, the coordinate axes (neural features) of the decoder must remain at least somewhat stable—which conflicts with the encoding result suggesting that the neural features are almost entirely unstable.
Overall, I would suggest to (1) detail the cross-validation procedure a bit more clearly and perhaps try to better match training and test sets for each comparison; and (2) make sure nothing weird is going on with the encoding model estimation/evaluation that causes it to crash below null when generalizing across days. I hope these steps will help resolve some of the discrepancy here and make the results easier to interpret.
In section 2.2.1 of the Results, the authors compare encoding performance in the incidental speech dataset to two smaller naturalistic datasets, a podcast listening dataset and a conversation dataset. The authors observe lower encoding performance in the incidental dataset than in the smaller datasets, but find that this discrepancy doesn’t seem to be explained by poorer performance in automated spike thresholding or transcription. I wonder if this difference could be driven by the linguistic content of the smaller datasets relative to the incidental dataset? My hypothesis would be that the smaller datasets are considerably more “information-dense” and linguistically diverse than the incidental recordings. I’m curious if the authors could quantify this somehow. For example, you could use PCA to compute the dimensionality of GPT-2 embeddings (e.g., number of PCs required to explain 95% variance) over the course of the podcast, over the course of the conversation, and in matched-length time segments of the incidental recordings. My guess would be that, for a matched number of words, podcast and conversation datasets are higher-dimensional in terms of contextual linguistic structure than the incidental conversations. You might also be able to directly put numbers on the linguistic diversity (e.g., lexical diversity, number of low-frequency words, syntactic complexity). In general, I think it would be useful to try to characterize the linguistic richness of these datasets in a bit more detail.
In Figure 4, please mark the chance accuracy level on all plots (e.g., a dotted horizontal line at 10%), as this is the meaningful point of comparison. I would consider trimming the minimum value of the y-axis to roughly chance level (i.e., above zero) or use a different kind of non-bar plot: the zero point is not really relevant here and the portion of those bars that visually goes all the way down to zero (below chance) doesn’t really mean anything.
Minor comments:
Abstract: “Together, these findings demonstrate that incidental natural speech is sufficiently processed in the brain to enable the estimation of neural-level embeddings.” Surely it must be! Speech is generated by the brain, so surely it is “sufficiently processed in the brain”; the question is whether we have the data and/or models necessary to quantify this processing.
Line 14: “using encoding models, which relate patterns of brain activity to language model-derived embedding”—this is a confusing description of encoding models. Encoding models map multivariate stimulus/behavior features onto individual neural features (many-to-one mapping from stimulus features to brain features). Encoding models do not really deal with “patterns of brain activity” in the way that term is typically used. Decoding models, on the other hand, map multivariate brain features (patterns of brain activity) onto individual stimulus/behavior features (e.g., face vs house; many-to-one mapping from brain features to stimulus features). Your description of encoding models sounds more like a description of decoding models. Please be very careful/precise about how these two approaches are described (Naselaris et al., 2011.
Line 24: “neural-level robust semantic models”—what does this mean?
Line 40: “we hypothesize that the brain processes natural speech well enough”—of course it does! Similarly to this formulation in the Abstract, there’s no question that the brain processes speech well enough to process speech. It must! The question is whether you can quantify this processing with a given dataset, processing pipeline, model, etc.
Line 42: “Daily life provides an exceptionally rich and diverse linguistic input”—is this really true? Certainly recordings of conversational language in daily life are more representative of our experience in daily life, but I suspect they tend to be less rich and diverse (in terms of vocabulary, syntactic complexity, etc) than a podcast, audiobook, etc.
Line 52: “3000 of total”—missing a word here?
Figure 2: In Panel C, are background boxes corresponding to nighttime? In Panel F, you use the term “timing drift”, which is not used anywhere else… I assume this refers to sentence-level timing error? Maybe adjust the axis label here to match the terminology used in the text.
Line 227: I would not call these “highly controlled tasks”! Podcast-listening and conversations are highly naturalistic tasks compared to the vast majority of other tasks a psychologist or neuroscientist might use (e.g., picture-naming tasks, single-word or sentence-reading tasks)! :)
Line 275: “2.2.3 Comparing brain regions”—doesn’t seem like you’re actually “comparing” brain regions in the section, more just looking at region-specific encoding
Line 281: “higher-level semantic/symbolic”—I wouldn’t use the word “symbolic” here; this word is loaded and I’m not sure there’s anything genuinely symbolic going on in these models or the brain activity
Line 295: I don’t think we actually do any decoding-/BCI-style analyses in the Goldstein et al., 2025 paper.
Line 357: In discussing your results related to “Encoding model performance as a function of dataset size,” I would consider citing work by Antonello et al., 2023, and Hong et al., 2024, that has also pursued this question.
Line 361: “That performance shows marginal decline”—I think you mean the rate of improvement slows? The performance is still going up so it doesn’t show “marginal decline” as far as I can tell.
Line 410: “the proportion of significant models does decrease significantly over time”—what does this mean? I don’t understand what number of decoding models you’re reporting a proportion of here. I understand that we have a separate encoding model per each unit, and that the number of units with significant encoding becomes fewer, but I don’t understand how that extends to decoding. Decoding models aggregate across all available units, so it’s just one decoding model. Do you mean the proportion of significant decoding models across subjects?
Line 420: “while the average decoding model has a positive slope”—I don’t really think this could be meaningfully positive. Positive slope would suggest that the decoder generalizes better to more distant days, with the same amount of training (and test) data? Highly unlikely.
Line 570: “200 words”—context window of 200 words or 200 tokens?
Line 583: I’m having a hard time understanding exactly what these quality metrics can tell us. Am I correct in understanding that you’re using, e.g., an automated wav2vec CTC metric to quality-check an automated Whisper transcription? Feels like there’s no ground truth mentioned here, so maybe this is more a check of whether different automated pipelines tend to agree? Were the quality metric scores correlated? I think I understand that the point here is just to drop low-quality sentences, but I had to reconstruct after reading this section and then reading back through the manuscript.
Line 744: “PCA was recalculated for the newly permuted training set per fold.” I’m not sure this is strictly necessary if you shuffle samples within the same training set that PCA was originally trained on. If I understand correctly, PCA/SVD does not care what order the samples are in (e.g., the covariance matrix can be computed incrementally one sample at a time); the PCA of the original and shuffled training set should be identical.
Line 750: “our downstream ,”—missing a word here?
Line 758: I’m struggling to understand whether the sample sizes going into your statistical tests are subjects, electrodes, test folds… In other words, what’s determining the degrees of freedom in one of your statistical tests?
Line 850: Is the decimation analysis using contiguous time segments of words or random subsets? I think temporally contiguous segments would support a stronger claim, as random subsets of words could sample different topics at greater diversity than they actually occur in real-world language.
References:
Antonello, R., Vaidya, A., & Huth, A. (2023). Scaling laws for language encoding models in fMRI. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 21895–21907). Curran Associates, Inc. DOI
Goldstein, A., Wang, H., Niekerken, L., Schain, M., Zada, Z., Aubrey, B., … & Hasson, U. (2025). A unified acoustic-to-speech-to-language embedding space captures the neural basis of natural language processing in everyday conversations. Nature Human Behaviour, 9(5), 1041–1055. DOI
Hadidi, N., Feghhi, E., Song, B. H., Blank, I. A., & Kao, J. C. (2026). Spurious alignment between large language models and brains can emerge from non-robust methods and overlooked confounds. Nature Communications, 17, 5769. DOI
Hong, Z., Wang, H., Zada, Z., Gazula, H., Turner, D., Aubrey, B., Niekerken, L., Doyle, W., Devore, S., Dugan, P., Friedman, D., Devinsky, O., Flinker, A., Hasson, U., Nastase, S., & Goldstein, A. Y. (2024). Larger language models better align with neural representations of natural language. eLife, 13, RP101204. DOI
King, J. R., & Dehaene, S. (2014). Characterizing the dynamics of mental representations: the temporal generalization method. Trends in Cognitive Sciences, 18(4), 203–210. DOI
Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage, 56(2), 400–410. DOI