Skip to Content

Procedure followed by the program committee for reviewing submissions

Members of the program committee:

Thomas C. Scott-Phillips
Mónica Tamariz
Erica A. Cartmill
James R. Hurford

We are putting this information here so that the procedure for determining which papers were accepted to the conference is transparent. 

As per the Call for Papers, all submissions stated whether they wished to be considered first for a talk or a poster. We received 156 ‘talk first’ and 42 ‘poster first’ submissions.

Submissions were first assessed for relevance to the conference aims. If there was unanimous agreement among the program committee that a submission was not relevant to the conference, then it was not sent to review.

All other submissions were sent to 3 or 4 reviewers. These reviewers graded each submission on a scale of 1?7 (1 = strong reject; 2 = reject; 3 = weak reject; 4 = neutral; 5 = weak accept; 6 = accept; 7 = strong accept). Reviewers were also asked about the relevance of the submission to the conference aims ((a) = relevant; (b) = relevance loose or not made clear; (c) = not relevant).

A maximum of 128 slots are available at the conference: 68 for talks and 60 for posters. Submissions were ordered according to the average rating given by the reviewers, and this created default cut?off points for acceptance to the conference, and for acceptance as talks. This created three groups of submissions: talk; poster; and those that could not be accepted.

Because we received more ‘talk first’ than ‘poster first’ submissions, then some talks were assigned as posters, but no posters were assigned as talks. Submissions of both types were marked as not accepted if they fell outside the highest 128 submissions.

Having established these initial groups, we then considered, for all submissions, whether they should be moved between groups either because there were outlying reviews that distorted the average (e.g. two very positive but one negative and unfair review), or because the reviewers considered the submission not relevant to the conference aims. In making these decisions, we took into account both the reviewers’ detailed comments and the reviewers’ own tendencies (some reviewers were more generous than others). A significant minority of submissions were moved between groups as a result of these discussions
Copyrightstory |