Open reviews
August 8, 2019 — Open review of preprint by Hudson and colleagues: PubPeer
Hudson, M., Seppälä, K., Putkinen, V., Sun, L., Glerean, E., Karjalainen, T., Hirvonen, J., Nummenmaa, L. (2019) Dissociable neural systems for unconditioned acute and sustained fear. bioRxiv, 676650. DOI
Hudson and colleagues use two horror movies to explore the relationship between brain responses and behavioral fear variables (sustained and acute fear) in a naturalistic context. I’m not an expert in affective processing or fear circuitry, so I’ll generally limit my comments to methods and some interpretational issues. Their general results are, for the most part, consistent across both movies (two separate samples of subjects), which certainly increases confidence. Several analytic choices, however, are not sufficiently motivated. I’m also unsure how to interpret these results with regard to head motion and physiological variables, both of which may be highly stimulus-correlated (the term “jump scare” makes me cringe in the context of fMRI). Finally, some of the interpretations are overly speculative and need to be re-written to minimize speculative reverse inference. I detail these larger-scale concerns first, then provide a list of minor comments.
Major comments:
There were a few analytic choices along the way that seemed poorly motivated. For example, the FDR thresholds range from .05 to .001 corrected. Were these thresholds chosen to yield sensible maps? (This seems like a great opportunity to post unthresholded maps to https://neurovault.org so folks can see everything.) I assume the cluster extent of 10 is set arbitrarily to clean up isolated significant voxels? Another example: It seems like you’re using a high-pass Savitzky-Golay filter with 240 s cutoff, but then also using a high-pass filter with 128 s cutoff for acute fear, then a high-pass filter with 256 s cutoff for sustained fear. I can infer the motivation for using a more permissive filter for slow-evolving sustained fear, but this should be explicitly stated. (Also, I’m wary of this modular application and re-application of filtering via Lindquist et al., 2019.) Why do you use sliding-window correlations to examine dynamic ISCs, but use phase synchronization to examine dynamic ISFCs? I would also include another sentence to better motivate the use of intraclass correlations.
I generally trust the statistics reported here, but it seems that very aggressive statistical thresholding (e.g., FDR = .001) is needed to yield reasonable maps; and even then, the ISC map in Figure 4 is significant for essentially the entire brain (although FDR = .05 is used here). I’m curious how the authors interpret this in light of work by Chen et al., 2016. (I appreciate that ISCs computed for long movie stimuli are often wildly significant.) I assume the authors are computing “pairwise” ISCs rather than “leave-one-out” ISCs (cf. Nastase et al., 2019)? I have a hard time articulating what null hypothesis is tested when using the circular time-shifting randomization. I suppose the null hypothesis is that there is no consistent temporal alignment among pairs of subjects? What kind of “population” inference does this afford? Again, I generally believe the results—I just think these things are worth considering and in some cases explicitly unpacking for the reader.
I suspect the audience would be extremely curious about how head motion and physiological variables (e.g., heart rate, respiration) impact these results, or naturalistic suspense/horror stimuli in general. I assume the authors did not collect heart rate or respiratory data. As far as I can tell, the authors don’t analyze or regress out either head motion or e.g. CSF time series. These things usually aren’t a big problem in intersubject correlation analyses under the assumption that they are not correlated across subjects via the stimulus. But a suspenseful horror film seems like the perfect stimulus to induce stimulus-correlated motion and physiological responses. These variables may also correlate with behavioral ratings of sustained fear and/or jump scares. I really think readers would benefit from an analyses of these confound variables; i.e., the extent to with they are correlated across subjects or with behavioral variables. I’m also curious how the results change if these variables are regressed out. I understand, however, that this is a thorny issue because the physiological variables such as heart rate and respiration may be tightly correlated with or even a fundamental component of the signal of interest. Regardless, this should be discussed.
The interpretation of your results in the discussion involves a lot of reverse inference (cf. Poldrack, 2006, 2011). Some degree of this reasonable and common, but in general it’s best to avoid speculating on what additional processes are engaged based solely on significant activation in region X. I would reel back the reverse inference in the discussion. I would also condense or remove the final section of the discussion on social transmission / contagion of fear, as this is very speculative and not directly addressed by the current data.
Minor comments:
At some point in the discussion, I got a little confused between all the terms used for the fear variants; e.g., “anticipatory fear”, “sustained fear”, “looming”, “jump-scares”, “acute fear”, “experienced fear”; as well as “pre-encounter / anticipatory” and “post-encounter / reactionary”. Make sure to use consistent terminology.
I assume it’s not possible to share stimuli due to copyright restrictions? But it would be great to share the code for segmenting the stimuli so others can precisely recreate them.
Page 10: I found Table 2 a bit confusing; a little more explanation would help here.
Page 11: “Pre-processing.” > “Pre-processing:”
Page 11: This format for pixel bandwidth seems strange to me, but maybe that’s my gap in knowledge. Make sure to fulfill COBIDAS requirements (http://www.humanbrainmapping.org/files/2016/COBIDASreport.pdf).
Page 11–12: I’m confused about the segmentation of the scanner runs (4 segments for Conjuring, 3 for Insidious; ~25 min segments); shouldn’t this match up with the 13 min segments described at page 9–10? Is the first segmentation only used for the behavioral experiments?
Page 12: “scans” > “volumes”
Page 12: You mention that the luminance and sound intensity regressors do not appreciably affect the results; however on page 19 (and Figure 3) it seems like including these regressors do substantially sparsify the results.
Page 12: Reword “Spatial smoothing used isotropic Gaussian kernel whose full width at half maximum (FWHM) was 8 mm.”
Page 12: You might point to Pajula and Tohka, 2014, as a precedent for using the 8 mm smoothing kernel.
Page 15: Does the ISBPS analysis require such a strict a strict bandpass filter? 0.04–0.07 Hz seems extreme.
Page 16: Does “low mean CI” indicate the width of the CI around the mean?
Page 16: Seems like the increase in ISC of fear ratings over time could also simply be due to the subjects becoming more comfortable with the task (providing continuous behavioral ratings of, e.g., fear isn’t a very naturalistic task). This is also not obvious from the CIs in Figure 1—why?
Page 17: “Koo & li, 2016” > “Koo & Li 2016”
Page 18: “parrahippocampus” > “parahippocampus”
Page 18: “cerebellar lingua” > “cerebellar lingula”?
Page 22: In Figure 4, the color bar label “z-score” does not make sense. I assume this is intended to be Fisher z-transformed correlation values.
Page 25: I find these network plots in Figure 6 a bit inscrutable. How are we supposed to interpret such dense network results?
Page 28: Don’t use the term “information transfer” for functional connectivity in fMRI.
Supplementary Material (SM): You use the term “interclass” instead of “intraclass” when describing ICC in the supplementary text and figures.
SM page 3: “high-pass band filter” > “high-pass filter”
SM page 4: The ROI labels on the thresholded coefficient matrices are too small / blurry to read.
SM page 4: “fea” > “fear”
SM page 5: When discussing intersubject phase synchrony in supplementary material, you might also point to Simony et al., Nat Commun, 2016, to situate the related method—but not mandatory.
SM page 7: You use the term “fear” in isolation; be more specific.
SM page 8: The thresholded ISBPS matrices look very different to me—almost anti-correlated. However, the ICC of the unthresholded matrices is .95? How is this possible? It seems like either the thresholding or the ICC values are misleading; hard to reconcile otherwise—and makes me less confident in the method.
SM page 9–14: The survey data seem interesting and may be useful to other researchers; I suppose most of the useful information is already reflected in your tables; but would it be feasible to share anonymized raw data?
References:
Chen, G., Shin, Y. W., Taylor, P. A., Glen, D. R., Reynolds, R. C., Israel, R. B., & Cox, R. W. (2016). Untangling the relatedness among correlations, part I: nonparametric approaches to inter-subject correlation analysis at the group level. NeuroImage, 142, 248–259.
Lindquist, M. A., Geuter, S., Wager, T. D., & Caffo, B. S. (2019). Modular preprocessing pipelines can reintroduce artifacts into fMRI data. Human Brain Mapping, 40(8), 2358–2376.
Nastase, S. A., Gazzola, V., Hasson, U., & Keysers, C. Measuring shared responses across subjects using intersubject correlation. Social Cognitive and Affective Neuroscience, 14(6), 667–685.
Pajula, J., & Tohka, J. (2014). Effects of spatial smoothing on inter-subject correlation based analysis of FMRI. Magnetic Resonance Imaging, 32(9), 1114–1124.
Poldrack, R. A. (2006). Can cognitive processes be inferred from neuroimaging data? Trends in Cognitive Sciences, 10(2), 59–63.
Poldrack, R. A. (2011). Inferring mental states from neuroimaging data: from reverse inference to large-scale decoding. Neuron, 72(5), 692–697.