Open reviews

December 28, 2023 — Open review of Lin & Kriegeskorte:

Lin, B., & Kriegeskorte, N. (2023). The topology and geometry of neural representations. arXiv. DOI

Lin and Kriegeskorte present a theoretical framework for incorporating topological analysis into the context of representational similarity analysis (RSA). The topological analyses discussed begin by introducing a nonlinearity (e.g. a piecewise linear function with adjustable thresholds) into the distances between multivariate neural activation patterns so as to compress certain (e.g. very small, very large) distances in hope of revealing more meaningful structure in the representational space. This results in a representational geo-topological matrix (RGTM) or representational geodesic-distance matrix (RGDM), both of which are analogous to a vanilla representational dissimilarity matrix (RDM) and can be swapped into the familiar RSA framework. The authors do a great job of situating purely-geometrical RSA (at one end of a spectrum) within the larger family of geo-topological analyses. They walk through some potential benefits of focusing on topology: for example, the lower bound “l” will effectively collapse small distances, which may only differ due to noise; the upper bound “u” will effectively consider large distances as maximally dissimilar, which may be justified given that once two samples are easily discriminable, further increasing their distance won’t make much difference; the middle segment of the piecewise linear function will tend to accentuate intermediate distances. The manuscript is well-written and the figures are very well-designed. My two main concerns are: (1) the examples used here do not actually make a very compelling case for the benefits of focusing on topology; and (2) potential users might benefit from more explicit guidelines on how to apply analyses.

Major comments:

1a. The authors do a great job of motivating the potential benefits of the topological approach, and the textbook simulation-based example (Figure 4) gets the point across well: the RGTMs and RGDMs more faithfully capture the topological distinction between the “flat 8” and the “untangled flat 8”, as well as the topological similarity or the “flat 8” and the “bent 8”, than purely geometrical RDMs. However, the examples using both fMRI data and deep convolutional neural networks (DCNNs) don’t make a very compelling case for the utility of topological analysis. I’m not very familiar with the literature on topological analysis or neural manifolds, but I wonder if a dataset with a relatively low-dimensional task structure in a more specific neural circuit would make the benefits of topological analysis more obvious; for example, the head direction circuit (Chaudhuri et al., 2019; Langdon et al., 2023). In other words, maybe there’s something about these vision datasets where they don’t tend to yield the kind of representational space that these topographic analyses key into. To be clear: I don’t think it’s necessary for the authors to add a new dataset/analysis to the manuscript at this stage; I just wish the benefits of topology were more obvious.

1b. With that in mind, the authors evaluate different RGTMs based on their “region identification accuracy” (RIA) using leave-one-subject-out cross-validation, with the goal of identifying geo-topological structures that differentiate brain areas but are consistent across subjects; similarly, they evaluate layer identification accuracy (LIA) across DCNN instances. This is a nice data-driven approach, and the analyses show that different parts of the geo-topological parameter space are similarly reliable; but I wonder if it obscures potential distinctions between focusing on geometry versus topology. From looking at panel 5b, I’m surprised there’s not more of a difference between e.g. geometry-sensitive (GS) and global extractor (GE) RGTMs panel 5a. I like the approach of bootstrap resampling both subjects and stimuli, but I wonder: are there more notable differences if we consider only population inference across subjects, rather than stimuli? Should we expect the geo-topological structure to actually generalize to the population of stimuli?

1c. With fMRI, do you think we might be able to see interpretable differences between geometry- versus topology-oriented parameter settings in ROI-by-ROI (dis)similarity matrices? (That is, for each type of RGTM, compute the similarity between the off-diagonal triangle of the RGTMs across all pairs of ROIs) Or in the corresponding ROI-level MDS plots (similar to Figure 6c and d for DCNNs; e.g. Visconti di Oleggio Castello et al., 2017, Figure 7)? In any case, maybe it would help to visualize each type of RGTM for each ROI/layer in the supplement. From a different angle: are there any meaningful differences in the benefits of geometry versus topology for different ROIs? For example, I could imagine a processing hierarchy where the representational space is higher-dimensional and more entangled at earlier stages of the hierarchy, and perhaps lower-dimensional and more topologically organized at the later stages (DiCarlo & Cox, 2007).

\2. I think it would be great if the authors could lay out a more practical vision for how other researchers—many of whom will probably not be experts in topological analyses (like me)—can incorporate this into their workflow. For example, how should we choose a pair of thresholds? (a priori? empirically?) I could imagine a pipeline where you perform grid search across a handful of threshold combinations in the training set to select an appropriate geo-topological RGTM in an unbiased way. Basically, I worry about the scenario where researchers get excited about this approach, then just start arbitrarily sliding around the parameters until they get something that looks “good”. Any cautions the authors can articulate to help enthusiastic adopters from falling into the “I tried a bunch of things” trap (Hosseini et al., 2020) would be great. Important note: the authors say these analyses are “implemented in the open-source rsa3toolbox” (great!), but a quick search of the current documentation (https://rsatoolbox.readthedocs.io/) didn’t turn up anything obviously related to this manuscript. The authors may have simply not pushed these changes to the “stable” documentation yet, but I just wanted to make sure the documentation gets up to date.

Minor comments:

In motivating topological analyses, the authors emphasize that certain features of topological analyses—for example, compressing small distances—may yield RGTMs that are more robust to noise. However, as far as I can tell, this is not really what we see in the analyses of DCNNs with varying levels of noise (Figures 6 and 7). At line 390, the authors admit that “geometry-sensitive statistics outperformed topology-sensitive statistics at higher noise levels”—but isn’t this the opposite of the expectation they set up in the Introduction? That is, the geometry-sensitive (GS) parameter settings—which include vanilla RSA!—seem to perform best (in discriminating layers across instances) at higher noise levels. This could be worth discussing in more detail in the Discussion; or it might be worth walking back this motivation in the Introduction a bit if it’s not borne out by the data.

I found it a little hard to follow the Figure 2 caption. It might be worth introducing some panel labels here to make the text-to-figure mapping a little more clear.

In Figure 5, is the colorbar at right the percentile from lowest to highest RIA values? I suppose this is why seemingly large differences in the percentile colormap do not map onto very obvious differences in the bar plot.

On my first read, it wasn’t obvious to me where the distinction between “topological” versus “geo-topological” arises. You mention this at line 76 (and it’s obvious once you get further through the manuscript), but you might want to make it more explicit.

In section 2.5, I’m not sure we need all this detail on noise sources; especially when the empirical examples don’t make it very obvious how different geo-topological parameters choices interact with increasing noise.

Line 197: “Significantly” > “meaningfully”?—or something to avoid the statistical connotation of “significant”

Line 231: “a held out” > “held out”

Line 278: “regions.” > “regions,”

Figure 5 caption: “Region-indetification” > “Region-identification”

Line 394: “emphasize” > “emphasizes”

Line 396: “in low-noise” > “in a low-noise”

Figure 6 caption: make sure this caption refers to DCNN “instances” and not “subjects”

Line 436: I would specify that the rsa3toolbox is a Python toolbox

References:

Chaudhuri, R., Gerçek, B., Pandey, B., Peyrache, A., & Fiete, I. (2019). The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nature Neuroscience, 22(9), 1512–1520. DOI

DiCarlo, J. J., & Cox, D. D. (2007). Untangling invariant object recognition. Trends in Cognitive Sciences, 11(8), 333–341. DOI

Hosseini, M., Powell, M., Collins, J., Callahan-Flintoft, C., Jones, W., Bowman, H., & Wyble, B. (2020). I tried a bunch of things: the dangers of unexpected overfitting in classification of brain data. Neuroscience & Biobehavioral Reviews, 119, 456–467. DOI

Langdon, C., Genkin, M., & Engel, T. A. (2023). A unifying perspective on neural manifolds and circuits for cognition. Nature Reviews Neuroscience, 24, 363–377. DOI

Visconti di Oleggio Castello, M., Halchenko, Y. O., Guntupalli, J. S., Gors, J. D., & Gobbini, M. I. (2017). The neural representation of personally familiar and unfamiliar faces in the distributed system for face perception. Scientific Reports, 7, 12237. DOI