This fantastic reviewing-ralated topic has been probably the most active one for some time. It looks like most of Corpora members put their 2 cents into the bag. The bag seems now quite heavy and shows somewhat big potential to turn it into something creative.
My question/proposition is,
How about working out a better reviewing system than the present double blind reviewing?
It could actually change something for the better, and since most of us took an active part in the discussion we probably have some ideas.
To give a good start let me say my bunch of ideas.
- Field assessment before reviewing. Something that has been implemented in easychair and several other systems to some extent. A reviewer selects her fields of expertise to give the conference organizers a hint on what papers he/she could review with the highest confidence. - Make blinding a choice. Some researchers want to be anonymous, and some want to use their previous research, but don't want to describe their previous works too extensively (saving paper space). - Discussion between authors and reviewers. For conferences there could be too short time span for double rebuttal, but for journals there could be allowed a longer open discussion. Its sometimes difficult to convince a reviewer only in one rebuttal session. - Using NLP technology to help reviewers and authors. Automatic summarization, credibility verification of infromation contained in a paper - for the reviewers, and for the other side (or both), opinion mining/sentiment analysis of all other reviews of the paper (comparing reviews) and of all reviews of one reviewer (is he/she an understanding, tolerant reviewer or an innocent bunny killer?)
Let me know about your ideas.
Michal
-------------------------- Od: Anil Singh <anil.phdcl at gmail.com> Kopia dla: corpora at uib.no Do: Gemma Boleda <gemma.boleda at upf.edu> Data: Thu, 6 Oct 2011 02:39:41 +0530 Temat: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
Not to forget the fact that getting a visa from countries like India to the US or to Europe is a project in itself. And it costs quite something. And if the visa is rejected after all that time, effort and expenditure... Rejection of visa is not rare at all: for the US (where a substantial proportion of the highest profile conferences are held) the rate is very high, especially for gradutate students.
On Thu, Oct 6, 2011 at 2:09 AM, Anil Singh <anil.phdcl at gmail.com> wrote:
I mostly agree on this point, except that this is quite a radical proposal and can be implemented only in the long term. We will need a lot of good quality journal to replace conference proceedings. Will be these be printed or purely online? Will those purely online considered to have the same credibility as those printed (why not?)?
The cost of attending a conference/workshop is, of course, a major hurdle for all researchers in developing countries. Nowadays even registration fee is so high that it is hard to afford it (say, in India) just for one author, even if no one travels abroad to attend the event and present the paper. And no solution for this is in sight (never mind the Emerging Economy, New Economic Powerhouses etc.). We are clearly being told explicity to not try to publish in conferences, but to try for journals (students by supervisors and administration, teachers/researchers by administrations and funding agencies). Easy to say, but how many journals are out there for the whole of CL/NLP (as compared to the number conferences and workshops)?
Still, shifting to journals from conferences (for publication) is something that has to happen in CL/NLP.
The academic evaluation forms (in India at least) give a much higher weightage to journal publications than to conference publications, which is a big disadvantage for those working in CL/NLP under the current situation.
Of course, as even Church (2005) had hinted, there is practical problem involved. Conference publications mean registrations and registrations mean $$. Where will the money for the journals come from? Who will sponsor them? If commercial publishers publish them, won't other factors come in which might affect their cridibility?
Just some doubts. But generally I support the idea.
On Wed, Oct 5, 2011 at 9:45 PM, Gemma Boleda <gemma.boleda at upf.edu> wrote:
Dear list members,
some of the concerns that have been raised in this discussion, such as reviewer load and "incrementality" in papers, could be addressed if the field moved to journal, rather than conference, publishing, and used conferences for dissemination of ideas (where only abstracts would be reviewed) and journals for actual publication. This would have the positive side-effect of making the citation indices of computational linguistics as a subfield go up, thus making it more visible in the "scientific market". For a 1.5-page long elaboration of these ideas, see http://www.lsi.upc.edu/~gboleda/pubs/gboleda_publishingCL.pdf
Best,
Gemma Boleda
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list Corpora at uib.no http://mailman.uib.no/listinfo/corpora