I've recently made the decision to say no to requests for peer reviews (outside of editorial boards on which I already participate, of course). While in the future, I may reconsider on my terms, I am finding at least one request a week for these and I think that it would be useful for to explain why this is so.
Obviously, I am well aware that peer-reviewed work is part of system that academics established in order to verify that work is up to scholarly standards and that it plays an important role in the tenure system. But the peer review system, as presently constituted, is broken.
First, there are far too many requests going around. Given the pervasiveness of global telecommunications, I get requests not just from the US, Ireland, and Lithuania, but from worldwide. Some of them are from entirely different systems and I am so far outside of the context that I have no framework to accurately respond with. If a dissertation is done badly by the standards that I would apply to it, is my evaluation appropriate if the person's work is head and shoulders above that of their own peers? How do I evaluate such work?
Moreover, academics asking for peer review are asking for free labor, time spent away from my work and my family. In an ideal world, this is communistic in which we all participate equally and do so for the mutual good of the system.
But my own position, outside of the tenured framework with have no sign that this will change any time in the near future, is the norm today. During the last decade, universities eager to engorge themselves with administrative staff have done so at the expense of tenure-track and tenured positions. If in the past, tenure-track was the rule, it is now the exception. The vast majority of my colleagues are not tenured or even tenure track. Most of us are not evaluated on the basis of peer-reviewed accomplishments, so asking us to peer-review work is to ask us to provide free labor for a system we are excluded from, and frankly that adds insult to injury.
Nor is this something that can simply be fixed. For every ten people who get a tenured position, I hear at least one unbelievable story of tenure denial. I couldn't think of a denial that I know of that has hit the press and therefore I can mention, apart from this famous one (ok, they have some grounds in denying him, no question), but if you are in academe and don't have friends who have been scarred in the process, then you are either a student or should still consider yourself "freshly minted" in the lingua franca.
Finally, Lyotard is right. We speak in incommensurable languages and, particularly in the messy realm of digital and network culture, we often have no way to evaluate each other's work. For example, the trumpeting of sources, in which a pastiche of names is strung together with nary an argument, is endemic in certain strains of sociology, but it would get a failing grade from a student in one of my courses. How do I evaluate it fairly? In another strand of geography, writers constantly refer to how their feelings about a place and the way that the ground feels under their feet. Normally I try to weed that out of my students like so much poison ivy in a yard. What am I to do with a review request regarding that sort of work?
I don't mean to say that these strains of academia should be snuffed out like old candles whose wick has burned down, although I suppose that I enthusiastically urge that on, but given that I have encountered this sort of work recently in reviews, was it appropriate to have a reviewer like myself on board? I'll leave it to your imagination as to how I might have responded to such requests, but obviously I have to balance being objective about the quality of the work with the fact that there is another person on the other side, no matter how ill-informed. In my own case, my attempt to find balance is informed by having been the victim of reviewers who I still think of as unqualified to be considered my peers. A long wihle back one of my articles was rejected for publication in one venue only to win an award in another. Another piece that I submitted as a talk to a conference was turned down because "the author's is derivative of research being done at the Center for Land Use Interpretation." This so-called peer was not sharp enough to fathom that the work was not derivative but was rather done in collaboration with the Center for Land Use Interpretation! Given that the submission was to be anonymous, I did not find it appropriate to list the Center any more than I would have listed the university I teach in. Another peer reviewed a project negatively and their decision stood even though the editor said it was quite clear that they had not actually read what I had submitted (believe me it perfectly was clear to me!). And so on.
Peer review is broken and I have no good ideas for fixing it, but more than griping my experiences with it or making excuses about why I am not responding to over-the-transom peer reviews, I want to put this complaint on the blog, and therefore, in public as a political statement, as a call to openly discuss the failings of this system.
Finally, I really miss Lingua Franca. If you are too young to have read it and you are in academe or planning to be, check out that link. Lots of grist for the mill there.