Open reviews
February 6, 2021 — Open review of preprint by Naumann and colleagues: PubPeer
Naumann, S., Byrne, M. L., de la Fuente, L. A., Harrewijn, A., PhD, Nugiel, T., Rosen, M. L., Dziobiek, I., van Atteveldt, N., & Matusz, P. J. (2020). Assessing the degree of ecological validity of your study: Introducing the Ecological Validity Assessment (EVA) Tool. DOI
Naumann and colleagues introduce a questionnaire for assessing the ecological validity of various aspects of a research project. In general, I think this is a neat idea and would be a useful tool if the authors can convince the community to adopt it. However, I think convincing the community will require an easy-to-use software tool or web interface (which I don’t see referenced here), and a bit more work on motivating the particular questions in the EVA battery. I follow up with a few lower-level comments.
Major comments:
Is there a software tool or web interface accompanying this? On page 7, you say “When you finish, EVA will produce a compass plot that consists of a triangle with three points on a circle”; and on page 15, you say “After submitting the form, a summary with your answers and descriptions accompanied by a compass plot will appear.” I was expecting to find a link where I could fill out the questionnaire and get my summary, but I don’t see any URL in the manuscript. I think you should also make it clear how responses to the questions will be “graded” with regard to final assessment. For example, presumably question 1 cannot be graded automatically on a simple 1–3 scale. Another example: does question 11 factor directly into the score? If a web-based automated assessment tool doesn’t already exist, it should be relatively straightforward to create.
In the Appendix, some of the questions seem overlapping or underspecified. For example, in question 4 the term “experimental approach” seems overly general (and overlaps with e.g. questions 3, 11, etc). Maybe this question should more explicitly specify task conditions, experimental manipulation, etc? Question 6 on “measurement” also seems to overlap with question 3. In general, I think the authors should devote some of the text to motivating each of these questions with reference to the literature, and ensure that each question is specified as clearly as possible.
Regarding Figure 1, I wonder if it would be worthwhile to also automatically generate the “other” radial plot where there are three expanding radii—CLR, PNLRA, NRWRA—and ~11 points on the star representing each question (or maybe 8 points depending on how responses to questions are evaluated). This would allow us to quickly look at the plot and say “okay this project is using PNLRA stimuli, but only a CLR sample of subjects” or something along those lines. Seems like it’d be useful to be able to look at the two radial plots side by side.
On page 2, you define ecological validity as a “quality of representing well the specific real-world behavior the entity aims to represent.” I wonder if this could be better-specified. What does “entity” refer to? I assume it could refer to both an experimental manipulation or a sample of subjects or… what else? This definition also seems to focus on “behavior” and I’m wondering if it’s also worth working in representative sampling of the environment (e.g. stimuli). Maybe something like “the quality of capturing the specific real-world behavioral and environmental factors the experimental design aims to represent.”
The point you make regarding Holleman at the top of page 3 about how ecological validity “should always be well-defined” is an important motivation for this whole piece. Ecological validity is not monolithic and different features of any given study can be more or less ecologically generalizable. One of the benefits of EVA is breaking down a research project into constituent elements and assessing each element’s ecological validity. I think you could emphasize this motivation a bit more and earlier on; e.g. you could use it as a framing device as early as the abstract.
I think providing an example (page 8) is a very useful exercise, but where does this particular example come from? Is this one of the authors’ actual research projects? Or is this a hypothetical example? Or is this a post mortem of some published project? I would make this clear at the outset of introducing the example on page 8.
On page 8, you mention “construct validity”—but this needs a bit more explanation (or reference to the literature). What is construct validity and how does it relate to ecological validity? (I realize there may not be a very clear answer to this question, but if you think it’s worth introducing the term, then we need a little more exposition.) This also makes me wonder whether there are similar questionnaires or tools for assessing other forms of validity e.g. construct validity.
Minor comments:
Page 1, line 21: “study’s characteristics” > “a study’s characteristics”
Page 3, line 50: “Atteveldt” > “van Atteveldt”
Page 4, line 7: The phrasing “use more ecologically valid paradigms” reads strangely to me; I would go with something like “use paradigms with greater ecological validity.”
There are some articles (“the” or “a/an”) missing here and there.
Appendix: Formatting is a bit inconsistent across questions (hyphens, indents, etc).
Appendix: Do questions 1, 2, 9, and 11 factor into the score?
Appendix page 4, line 31: Re-word the title of question 7.