Hello all.
I apologize in advance: This post is far too long. I will cut it down next time.
I have no formal cognitive science, or really any kind of "scientific" training whatsoever, so I may be way off base in the following. I hope to learn from this, and from you.
I was curious about a methodological point, especially in the context of cognitive or other "human" sciences. My concern is what I will naively call "population norming." Crandall emphasized my concern in the section in which he quotes Heraclitus, "you cannot step in the same river twice." The notion underlying this quote is that the river is different from moment to moment, and so literally speaking, by the time you step in "it" again it is a different river. This is applied especially to human beings - we are historical, and we change in time (it seems that Schmidt acknowledges this through the term "irreversible units) - and thus we cannot test the same exact subject twice. One concern with this notion is that you can extend it to very small time-scales: within one experiment, a person is different moment to moment, and so we cannot even test the same person
once (if an experiment takes more than some relevantly small amount of time). That is not my primary concern.
My concern is whether or not this concern with a change in identify of subjects can be mitigating by
norming a population, or taking subjects from a wide range of varying populations: academics, professionals, white/blue collar, the unemployed, oppressed persons, persons of privilege, various genders, etc. Both articles seemed to focus on defining one population and then maintaining in subsequent experiments, with the exception of certain kinds of concept replication (henceforth "CR") experiments, in which the population may be expanded. But it seems that in "social psychology" or any other kind of science in which the subject matter is explicitly human, you would want a wide range of subjects so as to make the most general claims.
Of course there are practical concerns, such as funding, time, access to populations, etc.
I was curious about Crandall's claim that CR is not meant to be exact, but I think that Schmidt explicated this well: There is an intentional change in some variable or other, rather than trying to map the original experiment exactly.
It seems that both authors address what is called a "publication bias," and recommend encouraging that various kinds of replication (for instance, CR) be more readily published. Schmidt seems to raise the point, however, that we don't actually learn very much from a failed CR experiment. Crandall nevertheless would like to see such experiments more often published; I am not sure where Schmidt stands in that regard.
What is pilot testing?
Schmidt utilizes a Dilworth quote that is seemingly inspired by Hume. Hume insists that we ought be careful, though, in ascertaining some "ultimate cause" behind the constant conjunctions we see. In that sense, for Hume, direct replication might not be theory confirming (consistent with the articles), but - except in a potentially deflationary sense - CR might not be either.
Replication is meant, in part, to increase confidence in some experiment, methodology, or perhaps even theory. However, it seems that Schmidt implies confidence that some experimental results
would be replicable is good enough. This seems worrysome.
Why is publishing "mere" replication discouraged in the social sciences but apparently not the natural sciences?
Schmidt points out that "interesting" science challenges assumptions of the audience, or of some general scientific paradigm or worldview. He also makes the claim that, though replication is demanded of these views, it is seldom delivered (this demand, by the way, seems related to Dr. Braasch's research presented last week). Thomas Kuhn argues, from a a philosophy and sociology of science standpoint, that persons operating in different scientific paradigms are destined to, to some extent, talk past one another. His considerations might give us grounds for thinking that replication might not be possible between paradigms, or that what counts as replication may be different between paradigms.
Schmidt introduces the notion of "tacit knowledge." I wonder about "tacit bias." I especially wonder how much a mechanism analogous to "Gettier Cases" function in scientific research. The idea is that someone can have a justified true belief without it counting as knowledge, in my interpretation because what justifies the belief and what makes it true can come apart. The standard example:
Smith and Jones apply for a job. Smith has good reason to believe Jones will get the job: The interviewer tells him he will hire Jones. Earlier, Smith counted the number of coins in Jones' pocket: 10. Smith forms the belief that "the man who will get the job has 10 coins in his pocket," and this belief is justified. In the end however, Smith is the one who gets the job. And, unbeknownst to Smith, he too has 10 coins in his pocket. So, his justified belief that "the man who will get the job has 10 coins in his pocket" ends up being true. But, we seem to think, Smith did not
know that.
I have yet to work it out, but it seems like replication has the potential to replicate bias, and the important point is this: Just because an experiment is badly designed, or in other words does not map the way things really are, does not mean its results cannot be replicated.
