1. Elijah Wright on July 1, 2004 11:26 AM writes...

I tend to agree about the fragmentation issue - part of the reason for sending the original discussion via email was that I wasn't sure that blog posts would be easily trackable. And they're not. Reason #2 was that I wasn't feeling terribly confident about what I was writing - and that's not usually something that folks like to expose to the blogosphere. :)

2. Liz Lawley on July 1, 2004 11:39 AM writes...

To pick up where we left off, I would tend to disagree with your comment in the last message that comments and feedback on blogs don't constitute peer review. I think one of the most interesting things about blogs in an academic context is the emergent peer review that's possible. That's not to say I think it can or should completely replace blind, refereed reviewing. But there's a flawed assumption in most peer review processes in the selection of "peers" (too much like "peerage", I think). I've had very disappointing experiences with peer review for some conferences, and in contrast, the feedback I get from people in the blogosphere tends to be very valuable. All in all, I've found more "aggressive, critically conscious" feedback in this distributed environment, often from people who have very important and interesting things to say but who haven't been "deputized" by the academy to say them officially.

Additionally, this informal peer review can happen at the early stages--at exactly the point where we might not be feeling completely confident about what we've written, and are more likely to be accepting of thoughtful critique. Once the paper is "done," it's a lot harder to accept that it needs significant changes...at least it is for me.

3. Elijah Wright on July 1, 2004 11:51 AM writes...

I don't disagree with the notion of 'informal peer review' being useful; I do, however, think that it is useful in very different ways than formal, blind peer review. Circulating text among your blogging peers is quite parallel to circulating it for commends among colleagues or other f2f friends.

We've also had more than a few poor experiences with blog-related peer review, lately. There's a post on my weblog from late last week about some of the positively oddball (and just plain sad) things have have happened. That's one of the risks of the peer review system - that some people don't take it seriously, that they don't understand how to adequately interact with critics or how to provide good feedback to others. A degree of empathy is required that is very difficult for most of us to put forth.

Formal peer review, done correctly, is supposed to take care of methodological problems and make good papers better. A poor reviewer is one who weakens a work rather than making it stronger.

[I'm afraid this comment is going to grow a little bit too long, so I'm going to pop out of it and start a new one.]

4. Elijah Wright on July 1, 2004 12:01 PM writes...

Another tangent that it seems useful to take, with regard to 'informal peer review' - I think it is more than a little bit difficult to identify which people in the community are actually doing research in this area, and which people it would be productive to send material in order to have them eyeball it.

In general our group has been sharing material with whomever it seems appropriate to do so, but those contacts are largely rooted in our personal social networks and in the amount of trust that we have for these other reviewers.

It would be nice to see quite a bit of this negotiation automated, or at least see the generation of some sort of registry of folks who consider themselves primarily interested/involved in weblog work.

[I think the words "weblog research" are not quite appropriate here - there's a tremendous amount of overlap between folks who are interested in the semantic web, online systems like wikis and discussion boards, YASNs, and even things like instant messaging and distributed cognition. I'm sure that other people could add a dozen or so areas of interest to my off-the-cuff list - which is why it's an off-the-cuff list.]

5. Elijah Wright on July 1, 2004 12:12 PM writes...

Hey Liz, maybe we should write a formal paper about this. ;)

6. Nancy White on July 1, 2004 11:33 PM writes...

Having appreciated Liz's posting on blog research, I was happy to come across this and even more grateful for both of you, Elijah and Liz, to share your conversation so openly... even the things that you might not be so sure of. As a non academic interested in the issue, you did two things for me.

1. You helped me see a bit more where you are coming from as researchers in acdemia

2. You helped me find other ways to question my research as a practitioner, and suggested new questions for me to explore.

What you did, in fact, is something I have been craving - a way to bridge the divide of practitioners and academics who care deeply about some shared things, but don't always have the opportunity for intersection. Or sometimes the language to understand each other. And as an independent, I can't go to much more than one conference a year, so I miss out in that important intersection.

In sharing your thinking publically, you gifted me with an intersection. Thanks.

7. Lilia Efimova on July 2, 2004 4:58 AM writes...

Was delighted to see this discussion moving public :)

A few points to add:

1. For me "blogging" peer-review has a totally different nature than the usual one. It's not much about formal/informal, it is about granularity and (as Liz pointed out) research stage. For me blogging my research provides a "feedback per idea", not "per paper", challenging my assumptions and interpretations, allowing to reiterate before it is too late and to incorporate multiple perspectives. Then, once ideas are ripe for a paper, I need "per paper" review as well, which goes both "formal reviewers" and "informal peers" ways.

2. On the understanding of the phenomenon: I agree with Liz, it it not much about methods used, but about undertstanding the nature of the study object (posted an example some time back - http://blog.mathemagenic.com/2004/06/20.html#a1247). The problem with weblogs is that meta-blogging, reflection on what's going on is much faster than any published paper, so for me reading weblogs is in a sense "state-of-the-art literature analysis" (also: finding papers on weblogs is much easier when you read weblog researchers' blogs; otherwise one can end up without referring to "seminal" weblog research texts :)... From my perspective "weblog researchers should blog" means at least reading other weblogs, but of course writing is better for the dialogue in the community :)

8. Lilia Efimova on July 2, 2004 5:05 AM writes...

Decided to make two posts :)

3. Another issue to bring in; a piece from the paper under review:

"Further weblog research also requires an understanding of the degree of influence that weblog researchers have in the process of developing blogging practices. As many weblog researchers are active and influential bloggers themselves, academic discussion on weblogs is not restricted to traditional academic publications: it actively "leaks" to the blogosphere and shapes further development of tools and practices."

Wonder what do you think?

4. Just a practical suggestion: don't think that it would be easy to keep the conversation in this space only. I would suggest participants to ping 'weblog research' channel at topicExchange (http://topicexchange.com/t/weblog_research/). In this case the probability of finding things back is higher :)

9. Elijah Wright on July 2, 2004 9:25 AM writes...

In response to Lilia's "point 3":

I think that this paragraph hit something in a spot-on way - things that we talk about at conferences, or things that we 'lift' from our major disciplines, are the things that will eventually get mentioned in passing on someone's weblog.

Something that I've been missing, I think, is a little more acknowledgement of that process. And I'm not completely sure how I think it could be improved. Asking people to more thoroughly cite the sources of their thoughts doesn't seem to be the answer - many blogs are already nearly unmanageable pools of links. I suspect that this (scholar-bloggers 'leaking' their primary content out into the perceived-as-non-academic blogspace) is just a rhetorical situation that the community hasn't quite worked out how to deal with, yet. It certainly can help a good idea spread quickly - and that's often hugely beneficial to everyone.