trendspotting the fMRI literature

Select a few neuroimaging papers at random and you’re likely to come across a handful of statements in the introduction to the effect that the topic under study is of “increasing interest”. At conferences and research talks, you’ll sometimes see speakers invoke a familiar kind of figure that looks something like this:

That’s the number of citations in PubMed containing the terms ‘fMRI’ and ‘language’ in the abstract or title, plotted by year of publication. Figures like this purport to show that interest in a topic is increasing dramatically. Just look at that increase! In 1996, there were only 13 hits; by 2005, there were 99! It’s as clear as daylight that interest in the neural bases of language is increasing!

Of course, the poorly-kept secret is that fMRI didn’t exist twenty years ago, and wasn’t really widely adopted until the last few years. So it’s natural to see an increase in publications that study language using neuroimaging methods. You’d expect a similar increase for almost every other area of research. The more pertinent question is whether interest in a particular topic has increased disproportionately relative to the general increase in the use of fMRI over the last few years. Instead of plotting absolute numbers, what we want is something like this:

In the above figure, the pink line represents the number of papers with the terms ‘fMRI’ and ‘language’ in the title (the blue line in the first figure has now turned pink–sorry about the color confusion!). But now the additional (blue) line shows the number of papers that have just the term ‘fMRI’ in the abstract. The increase in language papers starts to look suspect, since it’s clear the increase in fMRI papers on language is essentially paralleled by the increase in fMRI papers in general. Here’s an even better representation:

That’s the proportion of PubMed studies with the terms ‘fMRI’ and ‘language’ in the title or abstract over the last few years relative to the total number of studies with just the term “fMRI”. As you can see, it’s a very different picture. It’s a small sample size, but there’s not much reason to think people are any more interested in studying language in 2006 than in 1998—at least, relative to interest in other topics that can be studied with fMRI.

So what to make of claims that research interest is increasing in topics X, Y, and Z? Well, in a sense those claims are true, since the total number of neuroimaging publications continues to rise fairly dramatically. But in the sense that researchers probably care about more—namely, the “if I have a magnet and I want to do a study, what’s a hot topic right now?” sense—most research topics can’t be on the rise, by definition (just like most people can’t be of above average intelligence). Moreover, the number of academic publications in general has increased pretty dramatically over the last few years, so it’s not even clear from the above just how much of the increase in the number of fMRI papers on language is due to greater adoption of fMRI as opposed to a more global increase in scientific research output.

Now, the point of this post isn’t just to malign a ubiquitous research tactic. One can’t really fault people for wanting to think their own research is more interesting than other people’s. I’ll be the first to confess I’ve inserted some rather disingenuous comments about how oh-so-fascinating my results are and how much they (should) mean to other researchers in my papers. It’s hard to motivate a paper without doing that to some degree, or even to get motivated to do the research in the first place. What the second graph above does point up though, is that the question as to what topics are ‘hot’ is an empirical one—and fortunately, one that can be relatively easily (though imprecisely) tested.

To generate the above graphs, I used data from PubMed. One of the many nice things about PubMed is that it has an API that allows you to access the database programmatically (in contrast to Google Scholar, which is inaccessible via API due to agreements between Google and the major publishers to keep it that way). So, in the interest of doing some trendspotting, I wrote a small Visual Basic program to quantify the emergence (or lack thereof) of real ‘trends’ in research. I used the search string “fMRI [tiab]” as the control—i.e., all articles containing the string “fMRI” in the title or abstract. This is a conservative approach since the standard PubMed search also searches article contents, resulting in a difference of an order of magnitude in hits (7000 vs. 160000). But the more conservative approach is likely more accurate, since any study that includes the term in its title or abstract is much more likely to report original fMRI data than studies that just mention the terms in passing.

This reference number (broken down by year) was then compared with the results of a series of more specific searches. Basically, for a variety of topics, I added a single search term like “language” or “emotion” to the basic search. Again, the stipulation was that only titles and abstracts be searched. The ratio between the specific and the general term was then plotted for each year in order to highlight potential trends.

What do the results look like? Here are the ‘trends’ in neuroimaging for four major areas of research, broken down for the years 1996-2006:

What can we infer from the above figure? Well, just by eyeballing it, it looks like there’s a general trend toward relative increases in the number of papers on emotion, working memory, and attention, and no change for language. Statistical tests reveal that the three positive trends are significant (p < .05 for all three). So there’s at least some evidence that there are in fact trends in neuroimaging research (assuming there isn’t some alternative explanation, e.g., abstracts just getting longer and consequently mentioning more terms). The key point is that this kind of information can’t be gleaned just by looking at the first figure presented in this post. Absolute increases in publication count aren’t particularly informative. In contrast, when you use a control condition—though in this case, an admittedly crude one—you can feel a little more confident about the conclusions you’re able to draw. Naturally, this is a small sample size, and as I mentioned, the search is highly conservative (obviously, more than 46 fMRI articles on emotion were published in 2006!). But it’s likely that the results are a good representation of what’s out there, and that we can safely generalize to the many papers that use fMRI to study these topics but didn’t use the exact term in the abstract.

What about other ways of carving up the literature? Here’s the breakdown by sensory modality:

Doesn’t look like much is going on, and indeed none of the regression slopes are statistically significant. But at least this analysis is somewhat reassuring given the increases seen above for working memory, attention, and emotion: it’s clearly not as though all search terms are being mentioned more frequently in more recent fMRI abstracts.

Here’s one last figure (this could obviously go on for a very long time) plotting the trajectory of publication count in a few less-studied domains:

The trends for ‘social’, ‘reward’, and ‘decision making’ are significant here, but the trendline for pain isn’t. Social neuroscience research in particular appears to be emerging as a prominent domain of fMRI research, more than doubling its relative share of the literature between 2005 and 2006, though it’s still a relatively small field.

In evaluating the figures above, there are several caveats to keep in mind. One major limitation of this trendspotting approach is that it’s not well-suited to quantifying trends in more fine-grained areas of research, because there may only be a handful of studies per year, resulting in a pretty unreliable measure. Then again, claims that one small niche of research within the broader field of cognitive neuroscience is on the rise probably aren’t that interesting to begin with. If a particular topic was studied by 2 people in 2000 and 6 in 2005 (instead of a projection of, say, 4), you might want to wait a while before hopping on the bandwagon.

Another obvious limitation is that the procedure I used to generate these graphs was extremely simplistic. One can easily imagine more sophisticated approaches that control much more tightly for potential confounds (e.g., tier of journal, mean abstract length, etc.) and use better quantitative measures than the simple ratio I used above. That’s ok though; the point I want to make isn’t that this particular set of graphs provides a particularly accurate insight into the state of the field of neuromaging. Rather, the point is that scientific trends can be studied empirically just like anything else, and there’s a massive amount of data freely available for mining. Entire journals are devoted to tracking and discussing current research fads (see the ‘Trends in…’ series), but it’s unclear whether the editors at such outlets make their decisions on the basis of quantitative information. Conversely, from an author’s perspective, knowing what’s hot isn’t just a matter of curiosity—careful attention to trends could conceivably increase the rate of acceptance of one’s publications.

As a side note, if anyone wants to suggest possible searches for trends they’d like to see quantified, feel free to leave a comment below or to email me. I may release the VB program at some point, but it’s in no shape to see the light of day at the moment. Of course, you can always head over to PubMed and enter search terms manually.