musings & news articles 09 Jul 2006 12:06 am
Sokal, in reverse
Update 08/01: Harry Collins left a comment below (be sure to read it!) noting some inaccuracies in this post, on the basis of which I’ve made some changes.
Remember the Sokal affair? Back in 1996, physicist Alan Sokal submitted a paper full of clever-sounding gibberish to Social Text, a leading postmodern journal. Shortly after the paper was accepted, Sokal revealed it was a hoax. He pointed out that even a cursory familiarity with physics should have tipped the editors off, since the paper was riddled with ludicrous (and humorous) statements that any qualified reviewer would have been able to pick out. The fact that no physicists were asked to review the paper (on the relationship between quantum gravity and hermeneutics!) was taken to show just how low standards have fallen in some humanities fields.
This week’s Nature has what at first glance appears to be a counter-point: a news article entitled “Sociologist fools physics judges.” Could it be? Have social scientists finally cracked the physical code? Should we celebrate the champions of a new quantum sociology?
Well, not quite. The sociologist in question, Harry Collins, hasn’t actually managed to get a physics article through peer review. Nor has he tried to. What he has done is provide sensible answers (in English, not math) to seven questions related to gravity waves. At least, sensible enough to render a panel of 9 physicists unable to distinguish his answers from those of a real physicists. Based on this result (and a paper to be published later this year in Studies in History and Philosophy of Science–preprint available here), Collins proposes the concept of ‘interactional expertise’: the idea that non-experts can acquire sufficient knowledge of a technical field to converse intelligently.
Personally, I think this idea is something of a stretch, at least with respect to quantitative disciplines like physics. Being able to understand the meaning of physical equations in general terms (and Collins certainly has the background: he’s been studying physicists for 30 years!) doesn’t mean you can pass for a physicist, unless you think being a physicist consists in writing prose answers to canned questions on a relatively circumscribed topic. The Nature article has a nice quote from Sokal in this regard:
Sokal says he is struck by Collins’s skills in physics, but notes that such understanding would not be enough for more ambitious sociology research that attempts to probe how cultural and scientific factors shape science. “If that’s your goal you need a knowledge of the field that is virtually, if not fully, at the level of researchers in the field,” says Sokal. “Unless you understand the science you can’t get into the theories.”
Now it’s probably a testament to Collins’ versatile intellect that he can answer questions on gravity waves at all; but as a demonstration of ‘interactional expertise’ it’s on roughly the same level as early chatbot attempts to pass the Turing Test. The fact that a very simple program like Eliza can fool some people into thinking they’re communicating with a real individual is amusing, but it hasn’t helped us create anything like an AI that can pass for a human under normal circumstances. Similarly, physicists don’t normally display their knowledge in a forum as constrained as Collins’ series of questions. They can use their acquired knowledge and formal skills to generate novel theorems and predictions in a way that a sociologist of physics presumably can’t. So unless one weakens the notion of interactional expertise to the point where it’s almost vacuous, it’s not clear how much interaction can really take place. You simply can’t do physics without being able to do math, and hanging out with physicists for 30 years probably isn’t a good substitute for calculus textbooks–though it may make for interesting dinner conversations.
on 31 Jul 2006 at 4:35 am 1.Harry Collins said …
The account given on this website of my experiment: `Sokal in Reverse.’ is incorrect in at least three ways.
First it is stated that the journal in which the original is to be published was not supplied by Nature. It was! As stated there, the journal is Studies in History and Philosophy of Science — the paper will be out in December.
Second, the experiment was not intended as a Sokal hoax in reverse, it was intended to demonstrate how linguistic skill in a technical area could be genuinely acquired. The whole point of a hoax is that the skill is _not_ acquired.
Third the level of skill bears not resemblance to that encoded in an ELIZA-like program.
Anyone who has any interest in what the experiment was really about rather than scoring cheap points can read the preprint, the website of which was also given in the Nature article. It is http://www.cf.ac.uk/socsi/expertise — click on `Experiments on Interactional Expertise.’
Harry Collins
on 01 Aug 2006 at 7:08 am 2.small and gray said …
Thanks for the comment, Dr. Collins. I’m certainly not interested in “scoring cheap points”; I could have sworn the link to the preprint wasn’t on the Nature site when I wrote the post (I read the article twice!), but I guess I wasn’t careful enough. I apologize for that, and I’ve changed the post to link to your preprint.
With respect to your second point, I appreciate that you didn’t intend the work as a Sokal hoax, but that’s a link many people are likely to draw–the Nature writer included. So it’s worth considering the similarities, if only to reject the parallel (which I do). Similarly, I wasn’t suggesting acquiring linguistic skill is exactly like writing an ELIZA-like program; the question is whether there’s a qualitative rather than quantitative difference. At any rate, I think think the latter points are open to debate. I’ve pointed to your comment at the top of the article, and I’d encourage people to read the preprint and draw their own conclusions. I appreciate your comments.
on 01 Aug 2006 at 8:45 am 3.Harry Collins said …
Thanks for posting my reply. I have had a few responses which entirely ignore what the original article was about and I am getting a bit `shirty’ about it. Crucially, a test where the judges know that one of the persons is not what they are pretending to be is entirely different to one where they judges do not know. In the later cases — which include hoaxes and confidence tricks — it has long been documented that `the mark’ does most of the work of `repairing’ the deficiencies in the performance and only a minimal performance is required. That is why ELIZA-like programs can work so well: they have no bearing on properly conducted Turing-Test like experiments. It is also peeving to be accused of such a conflation because I have written extensively on these differences — eg `Artificial Experts: Social Knowledge and Intelligent Machines’ (MIT Press 1990) or `Dr Golem: How to think about medicine’ (University of Chicago Press, 2005 — the chapter on bogus doctors). Cheers, Harry
on 30 Dec 2006 at 9:16 am 4.Joseph Hyde said …
I would respectfully disagree…
Examples of this, that is someone who is an expert in a field that they don’t have formal training in, that is a formal degree are all around us… The common name for such people are ‘Renaissance Men’. I am sure you can name a few, some even have Phd’s in other disiplines.
Thanks.