What’s even cooler is the PLoS editors have collated all of that information and released it as one monstrous Excel spreadsheet. So you can now run off and do a quick regression analysis to determine whether or not having longer paper titles leads to more citations, if you’re so inclined.

One can only hope that the for-profit publishers follow the PLoS lead and start putting some of this information on-line. Given the widespread availability of things like download counts, bibliometricians (is that a word?) could develop novel measures of impact that go beyond simple citation counts (or derivative measures such as the H-index). You can imagine, for example, a metric that quantifies the “reach” of an article, which could take into account the number of views and downloads but not necessarily the citation count. Or, as some have suggested, one could simply use the download data as a more readily available proxy for citation rates, since it turns out that download counts within the first few months of publication are a strong predictor of citation rates several years later.

Of course, skepticism is probably warranted given the for-profit publishers’ track record of not exactly being eager to do what’s in the best interest of the scientific community. But perhaps other open-access providers (e.g., the Frontiers series) will follow suit…

On its face, writing a book doesn’t seem to be a high priority in psychology. Very few psychologists can claim to have an authored book on their vita, and these select few individuals still typically list books under a separate heading well below the almighty “Peer-Reviewed Journal Articles” section. I can count on one hand (well, maybe two) the number of times another psychologist has said something to me along the lines of “you should really read so-and-so’s book”. Books don’t play much of a role in day-to-day psychological research, and if anything, many researchers seem to harbor a slight contempt for them. Psychologists typically write books for laypersons, so that research-wise, the level of detail often leaves much to be desired. When psychologists want to know about fancy new experiments, they read fancy new research articles in journals like JEP and Psych Science; when they want to get a bird’s-eye view of a field, they read review articles in journals like Psych Review and Psych Bulletin. Books like Stumbling on Happiness or The Blank Slate may make for great reading before bed, but they’re rarely cited as a primary source in research articles (with a few notable exceptions, e.g., Antonio Damasio’s “Descartes’ Error”, which everyone and their grandmother cites).

Now, given academic psychologists’ general apathy toward authored books, you might expect that the authors of popular books on psychology would tend to be writers first and foremost, and that well-known researchers would rarely if ever take time out of their schedule to write a 400-page volume. That would be my intuition, at least; but it turns out to be the wrong one. In fact, a disproportionate number of popsci psychology books are written by very eminent researchers. People like Dan Wegner (The Illusion of Conscious Will), Dan Gilbert (Stumbling on Happiness), Dan Schacter (Searching for Memory), Paul Bloom (Descartes’ Baby), Steven Pinker (a zillion bestselling books on language and/or evolution), Antonio Damasio (several books nominally about old dead white guys but really about emotion), Michael Gazzaniga (The Mind’s Past), and Joseph LeDoux (The Emotional Brain) have all had extremely productive, well-respected research careers independently of their popular output.

The interesting question, of course, is why. Why are popular psychology books more likely to be written by eminent researchers?  Broadly speaking, I think there are three classes of explanations (I’m sure I’m leaving many out, though). One possibility is that popular books aren’t actually more likely to be written by famous psychologists; rather, psychologists are more likely to become famous if they’re written popular books. This would be interesting if true inasmuch as it would suggest that the conventional wisdom is wrong: rather than focusing single-mindedly on publishing peer-reviewed journal articles, young academics might do better to divert at least some of their time to popularizing psychology by writing full-length books (then again, most of the aforementioned authors wrote their first book after receiving tenure). Of course, this still wouldn’t explain why psychologists become famous after writing popular books. Perhaps there’s a familiarity effect: psychologists who publish popular books are likely to have their names repeated widely and often, which might subsequently bias researchers to assign more weight to those authors’ empirical research, regardless of its actual merit. Or perhaps hiring committees at schmancy universities use popular fame as an explicit criterion when evaluating candidates (though that seems unlikely, because many of the people listed above–e.g., Wegner, Gilbert, and Schacter, all currently at Harvard–published their major popular works after moving to elite universities).

Another possibility is that there’s a kind of selection effect: lots of psychologists publish (or try to publish) popular books, but only books by famous psychologists are widely read. Other things being equal, one would expect that books by Harvard professors sell more copies than books by professors at third-tier schools, so a bias may emerge at either the publishing stage (famous psychologists are more likely to get book deals) or the consumer stage (people are more likely to buy books that say Harvard or Yale on the cover).

The final possibility, which I personally find most interesting, is that there’s something characterological about good researchers–or at least, a subset of good researchers–that makes them more likely to publish popular works. There are a number of traits that come to mind here. One is simply intelligence: while popular science books are often maligned for their lack of depth, synthesizing a broad research literature into a clear, readable package can be a considerable feat of intellect. Another relevant dimension is creativity and/or the ability to see the big picture. Researchers who are good at integrating diverse ideas maybe both more likely to produce good research and more motivated to paint a discipline with broad strokes in a popular book. Or, it could be a matter of drive: writing a book takes persistence and hard work, and persistent, hard-working people are likely to be more productive in general. Of course, one could also cast this all in a more negative light, as simply a result of egomania: if you’re highly driven to be respected and admired by your academic peers, you might also be driven to show the public at large how cleverly you can write a book.

If I had to put money on it, I’d guess the reason book authors tend to be respected researchers is a little of column B (selection) and a little of column C (character). In theory this is a pretty easily testable hypothesis (the question being, essentially, what factor(s) mediate the relationship between (a) book authorship and (b) research eminence)–in fact, there’s a sizeable literature on the personality of highly successful scientists (e.g., Dean Simonton’s work). In practice, you’d probably be hard-pressed to get Steven Pinker to sit down with you for two hours of psychological testing. Which is fine, because it’s really not that informative a question anyway. The bottom line is just that there’s a interesting discrepancy between what many academic psychologists think about popular psychology books (negatively, or not at all) and what they think about the people who tend to write those books (positively).

* * *

As always, there are too many posters to see. The CNS schedule of events doesn’t begin to approach SFN standards–the latter consisting of a CD’s worth of fully indexed and searchable abstracts, and five different books (one per day)–but if you do any sort of neuroimaging work, a much higher proportion of the abstracts are likely to interest you. I start every conference I go to by spending half an hour meticulously checking off all the posters I want to see at the next session. Then when that session rolls around I promptly discard my notes and drift aimlessly from aisle to aisle.

* * *

There are a lot of complaints this year about the quality of the poster halls here at the Sheraton New York. The halls are (a) maze-like; (b) dark; and (c) warm. It’s a safe bet that some small proportion of attendees enjoys this environment, but for those of us who (a) don’t have an exquisitely-tuned spatial navigation system, (b) aren’t vampires, or (c) don’t suffer from hyperthyroidism, it’s a little bit uncomfortable.

* * *

The drinks last night at the welcome reception started at $6.50 for a soft drink. $11.50 for a beer. When I asked the bartender why I couldn’t just have a cup of tap water for free, he shrugged in antipathy. I suppose it was more polite than saying “because it would undercut our bottom line, schmuck.” So I went down the street, bought a bucket, filled it with ice and water, and gave away free refreshments to all the thirsty neuroscientists. No, just kidding. I bent over and took it just like everyone else.

* * *

Memo to presenters: that signup sheet next to your poster isn’t real. Etiquette requires that after someone’s finished being bored by the intimate details of your presentation for fifteen minutes, they be provided with some way of expressing their joy and gratitude to you for furnishing them with a life-changing experience. They do this by signing up to receive a second iteration of your treatment in written form. Putting their name on your form completes all contractual obligations. There’s no requirement that you actually follow up and email them your poster. In fact, doing so only inconveniences your audience. Last time I came back from CNS I spent an entire morning hitting the ‘delete’ button. I could have been doing much more productive things, like brushing my teeth.

* * *

I’m slowly realizing that New York is an expensive place with terrible service. Take for instance this morning. I was standing on the corner outside the hotel when a nice man approached me and said he was an artist and that he could put a beautiful glossy sheen on my poster for just $80. So I gave him my poster and $80, and he said he’d be back in twenty minutes. Well it’s been 3 hours and I haven’t heard anything. When he comes back, I’m going to be very angry with him. Just wait till he sees what kind of customer evaluation I give him at the information desk.

* * *

Auditory perception, memory systems, emotion, and numerical processing. These are all important areas of research, and certainly worthy of inclusion in the poster sessions. But there’s no reason to be elitist. Cognitive neuroscience is a diverse field. I’ve been emailing the poster committee my suggestions for topics for several years, and I’ve yet to see any follow-through or receive a reply. What’s wrong, people? Too creative? Too novel? Don’t envy me just because I thought of having a symposium on brick-selective cortex and you didn’t. It’s not my fault you lack vision.

* * *

There’s a lot of talk at this conference about how the brain is this wonderfully clever device that lets us project ourselves effortlessly into the past and future, move forwards and backwards in time, etc. etc. It’s not entirely unlike that other device that smoothly whisks you from the seventeenth floor down to the atrium while you’re busy placing mental bets on the length of the coffee line. Between your brain and the elevator, there’s no dimension you can’t conquer! You’re a master of time and space! Then the doors open up and someone jams your shoulder into the wall as they rush by you. Looks like you’re a lowly grad student again, grasshopper.

* * *

After a long day spent milling around hundreds of posters made by hundreds of scientists, all as smart and creative as you, all working on equally interesting problems, it’s easy to get a little down on yourself. What’s the point, you might ask yourself. Why bother participating in science if the best any of us can ever hope for is to make a tiny, insignificant contribution to that great puzzle that is the human mind. And what’s so great about the human mind anyway, if it’s just the temporal analog of an elevator. You might as well be studying dirt. Dirt is less dynamic than the mind, but more tractorable.

* * *

If a neuroscientist gives a great keynote address at a conference and no one hears it because they’ve all skipped the morning session to go roam around lower manhattan, does she still get to put it on her vita?

The really striking thing about the program is the sheer amount of money it throws at a select number of students—a projected 8,000 in 2014, with the number of awards gradually increasing over the next six years. It funds graduate students in natural science and engineering disciplines at a level up to $60,000 annually for three years, with applications renewable up to four times. Why the dramatic increase in funding over such a protracted period? From the article:

Critics complain that allowing graduate students to secure major funding for up to 12 years of predoctoral work will encourage complacency and clog up universities with ‘lifers’. Others see it is a necessary step if the US wishes to remain competitive with emerging Asian countries. Dr. Ron Sekubus, chair of the department of public policy at George Washington University, notes that without the funding, American universities would be forced to admit an ever-increasing number of international students in order to buffer against the loss of American students to more lucrative fields such as medicine and law. When international students complete their degrees, they are almost invariably unable to find legal work in the US, leading them to return to their home countries. The long-run outcome of this perpetual ‘brain drain’, Sekubus suggests, is that the US will fall in scientific productivity relative to rapidly-developing countries such as India and China.

When I ask him whether the government couldn’t just solve this problem overnight by allowing more trained foreign scientists to stay in the US once they’ve completed their doctorate, Sekubus’ responds aggressively. “Immigration is not the solution,” he says. “Increasing funding for American citizens via merit-based and non merit-based programs is the solution.”

The reasoning seems pretty straightforward. But as always, the story isn’t as clear cut as the above quote suggests. Here’s another relevant bit from the article:

Tovaz Shikarti, a program officer at the NIH, points out that high levels of predoctoral funding in the sciences make sense within the current cultural context:

“When we sat down to look at it, we couldn’t really understand why the top 5% American graduate students are getting paid less than the bottom 1% of American faculty. American culture is built on the promise of potential; it’s a foregone conclusion that this generation of top students is going to do some pretty remarkable things in a few years. The Pell-Mell grant program is our way of equalizing the situation by honoring the American tradition. You could think of this as an NBA draft for scientists.”

Others don’t see it that way. John Jacobson, a professor of history at The College of Wooster, complains. “I certainly don’t object to students making a livable wage. I was a graduate student once too. But when a first year physics graduate student at Wisconsin makes more than I do as an Associate Professor of Historical Arts at Wooster, I don’t think that’s right. My wife and I are giving serious thought to respecializing in materials science just so we can get a piece of the pie. It’s almost like the government’s goal is to get rid of the humanities and social sciences altogether.”

One of the caveats to the program is that the awards are highly selective–more so than existing NSF and NIH fellowships. There’s a three-stage selection process. The first two are fairly standard: First, would-be Pell-Mell grantees send in an application similar to the one required for the current NSF predoctoral fellowships. In fact, applicants to the NSF program are automatically entered into the Pell-Mell competition if they fill in several new fields on the NSF forms. Second, successful first-stage applicants are subjected to a still more rigorous screening, including close scrutiny of applicants’ transcripts, letters of recommendation, and current institution.

But it’s the final stage that sets the Pell-Mell grants apart from other programs. Successful applicants must not only demonstrate their academic prowess, but must also pass muster in the eyes of a newly-created Human Excellence Review Board (HERB). One of the novel requirements implemented by HERB is that applicants must designate a non-academic hobby as their “special skill”. The goal of this requirement is to encourage applications from well-rounded students with broad interests, instead of automatons who spend all day in the lab living and breathing one narrow discipline.

“What you list as your special skill is flexible,” says Tovaz Shikarti. “There’s no strict criterion our applicants have to live up to.” He notes that when the NIH conducted a limited test run with graduate students at Darthmouth College and the University of Pennsylvania, it received applications from people with skills like lizard hunting, karaoke, and cheese making. “There was even one trapeze swinger,” he says.

If this all sounds a little bit cock-eyed, don’t worry, the government is on top of that too.

When prompted as to whether the Pell-Mell program might not produce bad press for the NIH and NSF at a time when American scientists are complaining about flatlining funding, administrators demur. “We had a serious discussion about that at HERB,” says Aashish Patel, a board member. “Some people wanted to kill the program, to stub it out. But it’s not like we’re sitting around smoking up when we come up with these ideas. They’re serious policy proposals.”

That’s the number of citations in PubMed containing the terms ‘fMRI’ and ‘language’ in the abstract or title, plotted by year of publication. Figures like this purport to show that interest in a topic is increasing dramatically. Just look at that increase! In 1996, there were only 13 hits; by 2005, there were 99! It’s as clear as daylight that interest in the neural bases of language is increasing!

Of course, the poorly-kept secret is that fMRI didn’t exist twenty years ago, and wasn’t really widely adopted until the last few years. So it’s natural to see an increase in publications that study language using neuroimaging methods. You’d expect a similar increase for almost every other area of research. The more pertinent question is whether interest in a particular topic has increased disproportionately relative to the general increase in the use of fMRI over the last few years. Instead of plotting absolute numbers, what we want is something like this:

In the above figure, the pink line represents the number of papers with the terms ‘fMRI’ and ‘language’ in the title (the blue line in the first figure has now turned pink–sorry about the color confusion!). But now the additional (blue) line shows the number of papers that have just the term ‘fMRI’ in the abstract. The increase in language papers starts to look suspect, since it’s clear the increase in fMRI papers on language is essentially paralleled by the increase in fMRI papers in general. Here’s an even better representation:

That’s the proportion of PubMed studies with the terms ‘fMRI’ and ‘language’ in the title or abstract over the last few years relative to the total number of studies with just the term “fMRI”. As you can see, it’s a very different picture. It’s a small sample size, but there’s not much reason to think people are any more interested in studying language in 2006 than in 1998—at least, relative to interest in other topics that can be studied with fMRI.

So what to make of claims that research interest is increasing in topics X, Y, and Z? Well, in a sense those claims are true, since the total number of neuroimaging publications continues to rise fairly dramatically. But in the sense that researchers probably care about more—namely, the “if I have a magnet and I want to do a study, what’s a hot topic right now?” sense—most research topics can’t be on the rise, by definition (just like most people can’t be of above average intelligence). Moreover, the number of academic publications in general has increased pretty dramatically over the last few years, so it’s not even clear from the above just how much of the increase in the number of fMRI papers on language is due to greater adoption of fMRI as opposed to a more global increase in scientific research output.

Now, the point of this post isn’t just to malign a ubiquitous research tactic. One can’t really fault people for wanting to think their own research is more interesting than other people’s. I’ll be the first to confess I’ve inserted some rather disingenuous comments about how oh-so-fascinating my results are and how much they (should) mean to other researchers in my papers. It’s hard to motivate a paper without doing that to some degree, or even to get motivated to do the research in the first place. What the second graph above does point up though, is that the question as to what topics are ‘hot’ is an empirical one—and fortunately, one that can be relatively easily (though imprecisely) tested.

To generate the above graphs, I used data from PubMed. One of the many nice things about PubMed is that it has an API that allows you to access the database programmatically (in contrast to Google Scholar, which is inaccessible via API due to agreements between Google and the major publishers to keep it that way). So, in the interest of doing some trendspotting, I wrote a small Visual Basic program to quantify the emergence (or lack thereof) of real ‘trends’ in research. I used the search string “fMRI [tiab]” as the control—i.e., all articles containing the string “fMRI” in the title or abstract. This is a conservative approach since the standard PubMed search also searches article contents, resulting in a difference of an order of magnitude in hits (7000 vs. 160000). But the more conservative approach is likely more accurate, since any study that includes the term in its title or abstract is much more likely to report original fMRI data than studies that just mention the terms in passing.

This reference number (broken down by year) was then compared with the results of a series of more specific searches. Basically, for a variety of topics, I added a single search term like “language” or “emotion” to the basic search. Again, the stipulation was that only titles and abstracts be searched. The ratio between the specific and the general term was then plotted for each year in order to highlight potential trends.

What do the results look like? Here are the ‘trends’ in neuroimaging for four major areas of research, broken down for the years 1996-2006:

What can we infer from the above figure? Well, just by eyeballing it, it looks like there’s a general trend toward relative increases in the number of papers on emotion, working memory, and attention, and no change for language. Statistical tests reveal that the three positive trends are significant (p < .05 for all three). So there’s at least some evidence that there are in fact trends in neuroimaging research (assuming there isn’t some alternative explanation, e.g., abstracts just getting longer and consequently mentioning more terms). The key point is that this kind of information can’t be gleaned just by looking at the first figure presented in this post. Absolute increases in publication count aren’t particularly informative. In contrast, when you use a control condition—though in this case, an admittedly crude one—you can feel a little more confident about the conclusions you’re able to draw. Naturally, this is a small sample size, and as I mentioned, the search is highly conservative (obviously, more than 46 fMRI articles on emotion were published in 2006!). But it’s likely that the results are a good representation of what’s out there, and that we can safely generalize to the many papers that use fMRI to study these topics but didn’t use the exact term in the abstract.

What about other ways of carving up the literature? Here’s the breakdown by sensory modality:

Doesn’t look like much is going on, and indeed none of the regression slopes are statistically significant. But at least this analysis is somewhat reassuring given the increases seen above for working memory, attention, and emotion: it’s clearly not as though all search terms are being mentioned more frequently in more recent fMRI abstracts.

Here’s one last figure (this could obviously go on for a very long time) plotting the trajectory of publication count in a few less-studied domains:

The trends for ‘social’, ‘reward’, and ‘decision making’ are significant here, but the trendline for pain isn’t. Social neuroscience research in particular appears to be emerging as a prominent domain of fMRI research, more than doubling its relative share of the literature between 2005 and 2006, though it’s still a relatively small field.

In evaluating the figures above, there are several caveats to keep in mind. One major limitation of this trendspotting approach is that it’s not well-suited to quantifying trends in more fine-grained areas of research, because there may only be a handful of studies per year, resulting in a pretty unreliable measure. Then again, claims that one small niche of research within the broader field of cognitive neuroscience is on the rise probably aren’t that interesting to begin with. If a particular topic was studied by 2 people in 2000 and 6 in 2005 (instead of a projection of, say, 4), you might want to wait a while before hopping on the bandwagon.

Another obvious limitation is that the procedure I used to generate these graphs was extremely simplistic. One can easily imagine more sophisticated approaches that control much more tightly for potential confounds (e.g., tier of journal, mean abstract length, etc.) and use better quantitative measures than the simple ratio I used above. That’s ok though; the point I want to make isn’t that this particular set of graphs provides a particularly accurate insight into the state of the field of neuromaging. Rather, the point is that scientific trends can be studied empirically just like anything else, and there’s a massive amount of data freely available for mining. Entire journals are devoted to tracking and discussing current research fads (see the ‘Trends in…’ series), but it’s unclear whether the editors at such outlets make their decisions on the basis of quantitative information. Conversely, from an author’s perspective, knowing what’s hot isn’t just a matter of curiosity—careful attention to trends could conceivably increase the rate of acceptance of one’s publications.

As a side note, if anyone wants to suggest possible searches for trends they’d like to see quantified, feel free to leave a comment below or to email me. I may release the VB program at some point, but it’s in no shape to see the light of day at the moment. Of course, you can always head over to PubMed and enter search terms manually.

Now that I’m in the position of having to grade students’ multiple choice exams and explain their mistakes to them during office hours, I often find myself wishing I had a concise explanation as to why they really shouldn’t feel bad about getting Question Number 26 wrong, and why it’s still a perfectly good question even if they felt the wording was ambiguous. There are plenty of guides about how to take multiple choice exams floating around on the web, but what I’m after is a Damage Control Guide explaining how to defuse tension associated with students’ perceptions that they got screwed over on the last test. So rather than wait around indefinitely, I thought I’d write one, in the hopes others might find it useful.

The overarching point students need to understand and accept about multiple choice exams is that they are almost always made up of mostly bad questions, and that this is in fact mostly a good thing. By ‘mostly’ bad I mean that almost any question on a multiple choice exam is going to be ambiguous to some degree. Wording that seems crystal clear to one student is going to seem horribly vague to another; a question to which one students thinks B is unambiguously the right answer may confuse and anger another student, who think B, C, and D are all perfectly acceptable answers based on what the textbook says. Ideally, of course, such ambiguity shouldn’t be so pervasive as to completely paralyze and perplex the majority of students taking a test. However, some measure of ambiguity and even outright error is unavoidable.

It also turns out not to be a very big deal. It can be demonstrated mathematically that even a multiple choice test made up of mostly bad questions can still provide a very good measure of student’s knowledge of the tested material, provided that (a) there’s at least a weak correlation between students’ scores on individual questions and their overall knowledge, and (b) there are enough questions on the exam.

In practice, both of these numbers can usually be surprisingly modest. The reliability of a measure (or multiple choice test) is most commonly estimated using Cronbach’s alpha, which, in one form, allows us to compute a reliability coefficient as a function of two quantities: the number of items (or questions) on the test, and the average correlation between items. The formula is as follows:

Where N is the number of items and r is the average inter-item correlation. Given this formula, it’s easy to estimate the reliability of a hypothetical test. For example, a test with 30 questions and an average inter-item correlation of only .2 (equivalent to an average of only 4% shared variance between items!) will have a reliability coefficient of .88. In general, anything over .85 or so is considered good, so even by a creating a test with only 30 questions and weakly inter-correlated items, you can see that an instructor can end up with a very reliable test. Given that grades are typically derived from more than one test, the reliability of students’ overall grades will generally increase further. Moreover, if you were to increase the number of items on a given test to 90, reliability jumps to .96, or near perfect.

Note that because an average inter-item correlation of .2 is pretty low, the above calculation essentially gives instructors a free pass to have several bad questions on each exam. The net effect of poorly wording a question is to reduce its ability to correlate with other questions, because whether or not a student gets a bad question right depends on chance rather than knowledge. So smarter students are no more likely to get a bad question right than are poor students. Just how many bad questions one can afford to have on a test depends on how inter-correlated the good questions are; but it’s clear to see that even on a test of 30 questions with an average inter-item correlation of .2, having 4 or 5 questions that are completely uncorrelated with the rest of the test would have relatively little impact on the overall reliability of the test. And since reliability increases as a function of number of items, any concern about the drop can easily be offset by adding another 10 or 20 items.

Of course, all of this may initially seem like mumbo-jumbo to an irate student who feels they were mortally wronged by ambiguous wording on one or two questions. But it’s useful to explain nonetheless, because students who understand the logic will not only complain less, making your life easier, but will also have a more pleasant college experience, since they won’t spend four or more years feeling persecuted by malevolent instructors.

Having said all of this, there are a couple of important caveats, and one shouldn’t just conclude that any reasonably well-thought out multiple choice test is acceptable for class use. First, bad exam questions (even when there are only a few) do present a genuine problem for a small minority of students, namely those whose performance is at ceiling. If you’re a student who would have performed perfectly on a test made up of clear, relevant, and unambiguously-worded questions, the inclusion of bad items can only hurt you, since you have nowhere to go but down. In contrast, students who score lower in the distribution, say, around 75%, have little to complain about, since it’s entirely possible for their score to increase due to the inclusion of bad questions. Students who score near the bottom would actually experience a beneficial effect, with noise generally increasing their scores. But since the distribution of scores is almost always top-heavy in academic settings (more people pass than fail!), the overall net effect of unreliability is to shift the distribution of scores slightly downwards. In most cases this isn’t a problem since most instructors implicitly account for this (e.g., by making some exams ‘easy’ in order to shift scores upwards), but it’s worth keeping in mind anyway. Even if the reliability of your test is very high, it may still make sense to throw out the worst questions in order to prevent a systematic slip in the distribution.

A second and more important concern is that establishing that a test is reliable doesn’t necessarily mean it’s a valid measure of students’ learning. A reliable test is simply one that measures the same thing consistently. Nothing about the reliability coefficient tells you what that thing is. There are lots of things you could measure consistently in student populations that have little or nothing to do with the course material you’re teaching. For example, if you like to write extremely tricky multiple choice questions that require students to perform rigorous exercises in logic (e.g., “the answer can’t be A, because only one of these answers is right, and A entails that B is true as well”), you may well end up with highly reliable tests. However, these tests may not be valid measures of students’ knowledge of, say, organic chemistry or developmental psychology, because in effect, by turning your exams into an exercise in logic, you’ve loaded the ability to reason abstractly into your questions. In other words, what determines whether students do well on your exams may turn out to be their general level of fluid intelligence, and not the degree to which they’ve studied and assimilated the material. So while an unreliable test is always a lousy test, a reliable test may still be a lousy test. The ability to easily calculate Cronbach’s alpha isn’t an excuse to stop worrying about what your exams are testing for. But it does let you establish that wording problems or ambiguity on some questions don’t have much of an impact on your overall ability to measure students’ performance.