[e2e] double bland reviewing

[Manish Karir]

An alternate solution might be to limit each author to submitting no
more than say 5 papers per year.  Surely this will make people reconsider
both the maturity and the quality of their submissions a bit more
carefully before using one of their submission tokens.
This will in turn greatly reduce load on reviewers.  Even 1
day workshops accepting 10 papers get 200-300 submissions! Given 10 people on
the TPC this is a huge workload and quite possibly the cause for casual
reviews.

Realistically, is it possible for any one person to have accomplished
10-20 papers worth of work in a single year?!

-manish


On Tue, 11 May 2004, Kostas Pentikousis wrote:

> Dear David,
>
> On Thu, 29 Apr 2004, David P. Reed wrote:
>
> |Separate the two problems: conference integrity and constructive feedback.
> |
> |If an author needs honest and constructive feedback, that is best obtained
> |by seeking out reviewers separate from a "judging" process such as the
>
> Then, they should be called "judges", not reviewers, right?
> Although the outcome is binary (pass/reject), "judgements" should
> be justified. I do expect a review, i.e. a critical evaluation. A
> score and empty comments mean the "reviewer" is simply a grader.
> Is this how the state of the art advances?
>
> A reviewer for a flagship ComSoc conference back in 2001 made
> clear that ECN was studied thoroughly, and that nothing more was
> left to be investigated. I am happy to see that Partridge et al.
> believe otherwise. More recently, a reviewer in a measurement
> conference was clueless about a predominant traffic trace record
> format. I wonder how many reviewers spent 3-5 hours on a paper.
>
> |program committee.   The goal of the program committee and peer review
> |judgment of quality is not well aligned to the process of improving
> |unselected papers.   Selection comes first, and improving comes after.
>
> Alexander Hars points out that it may not be long till journals
> and conferences will have only one value-added feature: enhance
> and improve papers. Publishing is not a scarce resource anymore.
> When was the last time you visited the library to photocopy a
> journal article?
>
> |If you want to eliminate the effect of cheap shots and logrolling, make the
> |program committee process transparent and public - it's there (in my
> |experience) that the integrity of a conference is lost.
>
> That is, follow an open source model and bring meritocracy back in
> rule: Novel ideas or approaches at examining a problem; solid,
> reproducible results; sound discussion and comments; justified,
> verifiable conclusions; thoughtful, engaged reviews. In this brave
> new world it is correctness, utility, and availability that
> matter, not numbers of papers.
>
> Allman in his reviewer plea (http://www.icir.org/mallman/plea.txt)
> makes it clear: If [the reviewer] cannot understand the paper
> [s/he] will recommend rejecting it. Very typical of the human
> nature: destroy what you cannot comprehend. Is it possible that
> uninitiated, untrained reviewers are in the loop (and, no, I do
> not mean Mark ;)?
>
> Maybe it's time for an author's plea.
>
> Best regards,
>
> Kostas