[Corpora-List] why LREC2012 NOT blind-reviewed?

Reinhard Rapp reinhardrapp at gmx.de
Mon Oct 3 17:16:41 CEST 2011

Previous message: [Corpora-List] why LREC2012 NOT blind-reviewed?
Next message: [Corpora-List] why LREC2012 NOT blind-reviewed?
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Dear colleagues,

Note sure if this was mentioned before in the discussion: Concerning LREC, as many papers deal with language resources which are often made available on the web, it makes sense that reviewers can also look at these sites. This is not compatible with double blind reviewing.

Concerning bad reviews, I think reviewers usually have the best intentions (otherwise they would not volunteer), but life can be very harsh with us poor scientists, so completely unforeseen problems such as the following may arise:

1) The reviewer was too optimistic timewise and finally has to

act in a last minute fashion (90% probability).

2) The assigned paper simply does not match the reviewer's

competences or interests (50% probability).

3) The paper is so bad that it does not deserve a detailed review

(10% probability; some help for newcomers might nevertheless

be nice).

In my experience, bidding is very helpful to obtain better matches between papers and reviewers' interests and thus helps to solve problem 2). But it is usually based on abstracts which can be misleading.

So let me suggest to take this one step further:

Allow reviewers to return papers which they don't like to review after looking at them.

This could be realized as follows:

>From the pool of all papers reviewers can pick freely the ones that
look interesting to them (of course based on some pre-classification and up to a maximum number). These papers are then assigned to the reviewer for a certain time slot (a fraction of the total reviewing period). If a paper has enough assignments, it is locked, i.e. invisible for new reviewers. If a reviewer does not provide a review within the given time slot (alternatively: makes a binding commitment), the paper is unlocked, i.e. returned to the pool.

This method would hopefully solve problem 2) above. But it would also reduce problem 1): As early reviewers have a better choice, fast action is encouraged. Papers which are returned several times would finally (towards the end of the reviewing period) have to be dealt with in a last minute fashion. But the hope is that many of these papers would fall into category 3) anyway (rather than papers on less popular topics finding no reviewer).

This procedure would also remove from the programme committee (some of) the burden of making appropriate paper assignments, and would encourage or even force reviewers to provide their reviews at an early stage.

But needless to say that I am not sure whether this would work in practice, and whether the benefits would outweigh the extra effort.

Kind regards,

Reinhard

----- Original Message -----

From: Ted Pedersen

To: corpora at uib.no

Sent: Monday, October 03, 2011 1:39 PM

Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?

I think one thing we've learned here is that there are some fairly significant concerns and frustrations with reviewing.

For myself, I review too much, and am probably a little lazy in some cases as a result. I started to notice that a bit more recently, and so have started saying "No" with much more frequency, so hopefully I am doing my small part to correct something in the process. Reviewers who feel they don't have time to review really need to say "No" more often - the field will survive without us, and we aren't doing anyone any favors by submitting reviews that we don't really spend enough time on. LIkewise, I don't think saying "Yes" and then farming out reviews to graduate students is all that helpful, unless the senior person is willing to spend some time with the student on reviewing (until they are sufficiently experienced). I realize more folks doing this will make it harder to get reviews, but I think a smaller number of better reviews is in the end more helpful and healthy.

I would like to suggest that maybe we ought to ask people who submit papers to provide the answers to the following two questions (separate from their papers) in an effort to streamline the process.

1) What is the most important idea presented in this submission (in 50 words or less)

2) What other paper is most similar to this submission, and how does this paper improve upon or extend that? (in 100 words or less)

If I'm not sufficiently excited by the answers to both 1 and 2, then the paper can be rejected without further review. A good paper will of course make 1 and 2 fairly clear, but sometimes you have to dig a little, so I'd like to dispense with the kabuki dance and simply ask authors to answer these questions at the start, and then we decide as reviewers if we should read further.

The other part of the equation is that most published papers don't end up having much impact beyond advancing the author's career (Zipf's Law for Papers? A few papers cited a lot, most not cited much at all). This doesn't mean they shouldn't be published, and career advancement is a good thing generally both for the authors and our field, but it can also make for lots and lots of incremental papers that just aren't all that interesting and so they aren't cited much, and they tend to have a mind numbing effect on reviewers and is part of what I think makes reviewing such a chore sometimes. So, maybe if we make authors self-identify the incremental work versus the big new ideas then reviewers can have a better idea of what to expect. Incremental in the area you care most about can be fascinating stuff, so I don't think incremental is always a bad thing, but there also needs to be a balance between the incremental and the more novel. As a reviewer I feel like I spend huge amounts of time on incremental work, and it just gets a little dull to be honest...

Cordially,

Ted

On Fri, Sep 30, 2011 at 9:45 AM, Isabella Chiari <isabella.chiari at uniroma1.it> wrote:

Dear Corpora members,

I just noticed that the LREC2012 call specifies that submissions are NOT anonymous and there will not be blind-reviewing.

Does anyone know why? Which is the policy under this decision?

Best regards,

Isabella Chiari

Dipartimento di Scienze documentarie, linguistico-filologiche e geografiche

Università di Roma “La Sapienza”

pl.le Aldo Moro, 5, III Piano, Edificio ex Facoltà di Lettere e Filosofia, 00185 Roma, tel. +30 06 4991 3575

E.mail: isabella.chiari at uniroma1.it