An intergalactic journey to discover the gap between psychological theory and data.
I celebrated my eighth birthday while visiting my aunt and uncle in Minneapolis one summer, and received the most interesting gift. The morning began with my uncle describing the "strange lights" he saw in the sky the previous night. A family trek into the nearby timber revealed an artifact from outer space. At the base of a pine tree was a simple tin can, such as might contain beans or soup, with a picture of a grey alien printed on its label along with the ominous phrase, "Alien in a can." Giving in to my natural impulses, I immediately picked up the can and shook it. It rattled. Something was inside. The tin can was sealed, and I would need a can opener to open it.
I noticed a small paperback book lying in the grass next to the can. This was an instruction book that covered the feeding and care for the alien, who I learned was alive and well inside the can. The most important piece of information, printed in large block letters on the cover, was that the alien cannot breathe Earth's oxygen-rich atmosphere, so under no circumstances should the can ever be opened. Needless to say, I put any further thoughts of a can opener out of my mind.
I spent the rest of the morning reading the instruction book on the couch, with my alien in the can sitting next to me on the coffee table. I learned that the alien fed itself from my brainwaves. Just thinking good thoughts near the can would provide all the sustenance my alien needed. The book also gave several warnings. One of the most memorable was that the alien must never learn that it is in Cleveland, Ohio. If the alien discovered this, it would destroy Cleveland, - although the book didn't describe exactly how. At the time, I had never been to Cleveland and it seemed a rather remote possibility given my limited means of transportation. To be on the safe side, I decided to never mention the word "Cleveland" near the can, lest Des Moines, IA, or some other Midwestern U.S. city, be accidentally destroyed while on other family trips. Despite these dangers, I was enjoying spending time with my newly adopted alien, occasionally giving the can a little shake to hear it rattle. As an adult, I occasionally reflect on my alien in the can. At the time of this writing, Cleveland is still here, an improving football team and a not too distant NBA championship to its credit. I like to think that my careful alien stewardship had something to do with this. What I find especially memorable about my alien is how well it illustrates a fundamental gap between scientific theory and empirical phenomena.
The alien in the can presents a classic inference problem. Shaking the can, or similarly interacting with it, produces an observable result, a rattling noise. This noise is easily verified and reliably replicated. The alien that produces the rattling noise cannot be observed. It's safely sealed away inside the can. At the time, I was consumed with understanding the nature of the alien – what, exactly, was producing the rattling in the can?!? With no way to directly observe the alien, I had to try to infer its nature solely by the rattling noise it could produce.
This is parallel to many questions within psychological research. Human behavior is observable, impactful, and can often be replicated. As researchers, we often claim that behavior is causally linked with latent, unobservable constructs, often ones relating to cognitive processing, memory, social views, and so on. We cannot "open the can" to directly observe these constructs, so, like my eight-year-old self, we are forced to work with what is observable, the "rattles" so to speak.
In particular, the alien in the can presents a problem of coordination (Kellen et al., 2021). We can define a coordination as a precise, unambiguous description of how a set of unobservable variables relate to observable ones. How does the alien's state relate to the rattling noise? Critically, we cannot determine this relationship in a purely empirical fashion, as we can only observe (or hear) the ratting noise and never the alien's true state inside the can. Any empirical approach to discover the actual coordination between the alien and the rattles would require joint observations of both, and the alien can never be directly observed. Likewise, to interpret psychological data, we require coordinations that explicitly link unobservable cognitive constructs to observable human behavior. As with the alien in the can, we can only define these relationships, not empirically discover them. We can run studies testing memory processes, but we can only observe participants telling us they did or did not recognize a stimulus - we cannot observe "memory" itself.
In many ways, the instruction book for my alien provided a compelling theory. It stated that an alien lives inside the can, feeds on my brainwaves, and has a chip on its shoulder for a medium-sized, Midwestern U.S. city for reasons I couldn’t fathom. As informative as the instruction book was, it did not provide a coordination. It never explicitly stated how the alien produced the rattling noise when I shook the can. Any "alien in a can" experiment I could devise would require one. For instance, I could test whether depriving the alien of brainwaves by keeping the can isolated in my closet for a week would make the alien weak and hungry. To connect this hypothesis to data, I would need to come up with a coordination, such as: the can will rattle less if the alien is weak and hungry. This coordination explicitly links the observable data (rattles) to the unobservable alien state (weak and hungry), but it is also entirely arbitrary, which is exactly the point, as I can't use data to determine it.
Suppose I carried out this experiment and found that the can rattled just as much after a weeklong isolation in my closet as before. What are we to infer? Logically, we are testing the conjunction of two things: (i) the general theory that an alien lives inside my can and (ii) my choice of coordination between the alien's hunger state and the rattling of the can. Based on the results of this experiment, either the theory is wrong or my coordination is. It is possible that there is indeed an alien inside the can, but the relationship between the rattles and its state of hunger may not be a simple one. Pre-registering my hypotheses, using sophisticated model selection methods, and replicating the experiment over and over doesn't help solve this problem.
The coordination problem is ubiquitous in science, see Chang (2004) for a great example in thermometry, but is a particular challenge for psychologists, especially when the relationship between theory and coordination is not treated carefully. Kellen et al. provide several cases where the problem of coordination takes center stage, including the "face-inversion" effect, where an individual's ability to recognize images of human faces is more impacted by orientation (e.g., image shown upside down) than images of other objects. Prior research on this topic has almost exclusively used simple linear functions to define the coordination between memory (unobservable construct) and participants' responses indicating whether or not they recognized an image. Claims that our memory processes treat human faces in a unique way may depend heavily on this choice of coordination, with less restrictive coordinations leading to alternative conclusions (Kellen et al., 2021).
Whether due to habit or training, coordinations sometimes reflect a researcher's intent to run an ANOVA, or similar statistical analysis, - often at the expense of more substantive concerns. In this regard, advances in computational modeling, order-constrained inference, and general theory construction can be useful in specifying and evaluating more nuanced, substantively-motivated coordinations. There can also be merit in considering multiple plausible coordinations and evaluating whether there is general agreement among them - akin to consistency tests described by Regnault (see Chang, 2004). The basic idea is that if each coordination is measuring the same latent construct they should behave in similar ways.
The problem of coordination is an extremely difficult one. Yet, thinking carefully about our coordination choices, and how they relate to our empirical tests, is important for making sound inferences. We don’t want to reject valuable theoretic perspectives based on poor, or overly restrictive, coordination choices.
I was home for about two weeks before I fished a can opener out of the kitchen drawer. The alien's can opened as easily as a can of soup. Peering inside, there was only a rolled-up piece of yellow paper bound with a red ribbon. The paper was placed upright in the can and was tall enough that it couldn't fall over. Shaking the can made the end of the tightly rolled-up paper strike the bottom, producing the rattling noise.
I untied the ribbon and unrolled the document. The text was in English and was just a few paragraphs long. It was the last will and testament of my alien, who was described as a prominent and well-respected member of their alien society. All of my alien's extensive properties and belongings were to be left to friends and family back on their home planet. The document clearly stated that I was to receive only the tin can, as I was directly responsible for the alien's untimely demise by opening it.
So, the theory was correct and my coordination was wrong. Science is difficult.
-Clintin P. Davis-Stober