[Corpora-List] Reminder: CLEF 2020 Call for Labs Proposals - submission deadline July 5, 2019

Aurélie Névéol aneveol at gmail.com
Wed Jun 26 17:30:54 CEST 2019

Please, apologies for cross-posting


CLEF 2020

Thessaloniki, Greece - September 22-25, 2020



Call for Labs Proposals

CLEF is a leading annual international conference exploring topics in information access technologies. CLEF uniquely combines evaluation laboratories and workshops with research presentations, panels, poster and demo sessions. CLEF 2019 (http://clef2020.clef-initiative.eu/) will be held on 22-25 September 2020 at the CERTH - The Centre for Research & Technology Hellas, Thessaloniki, Greece.

Researchers and practitioners from all areas of information access and related communities are invited to submit proposals for evaluation labs for consideration for inclusion in CLEF 2020. Lab proposals will be reviewed by the lab selection committee. Organizers of selected proposals will be invited to include their lab in the CLEF 2020 lab programme, possibly subject to small suggested modifications to their proposal to better suit the CLEF lab workflow or timeline.


The CLEF Initiative (http://www.clef-initiative.eu/) is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual information in different modalities - including text and multimedia - with various levels of structure. CLEF promotes research and development by providing an infrastructure for:


independent evaluation of information access systems;


investigation of the use of unstructured, semi-structured,

highly-structured, and semantically enriched data in information access;


creation of reusable test collections for benchmarking;


exploration of new evaluation methodologies and innovative ways of using

experimental data;


discussion of results, comparison of approaches, exchange of ideas, and

transfer of knowledge.

CLEF 2020 will be the twenty first edition of the current CLEF conference launched as a continuation of the annual CLEF evaluation campaign series which ran with great success from 2000 to 2009 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks.

Scope of CLEF Labs

Proposals are accepted for two different types of labs (see 2019 Lab at http://clef2019.clef-initiative.eu/index.php?page=Pages/labs_info.html):


Evaluation Labs that are a "campaign-style" evaluation for specific

information access problems (during the twelve month period preceding the

conference), similar in nature to the traditional CLEF campaign "tracks".

Topics covered by campaign-style labs can be inspired by any information

access-related domain or task.


Labs that follow a more classical "workshop" pattern, exploring issues

of evaluation methodology, metrics, processes etc. in information access

and closely related fields, such as natural language processing, machine

translation, and human-computer interaction.

If the organisers of the proposal are new to CLEF or other shared task evaluation campaigns, we highly recommend that a lab workshop first be organised to discuss the format, the problem space, and the practicalities of the shared task. The CLEF 2020 programme will reserve about half of the conference schedule for lab sessions. The labs will present their overall results "overview presentations" during the plenary scientific paper sessions to allow non-participants to get a sense of where the research frontiers are moving. Organisers of each lab are expected to organise the separate sessions for their lab at the conference so that they contain ample time for general discussion and engagement by all participants - not just those presenting campaign results and papers. Organisers should plan time in their sessions for activities such as panels, demos, poster sessions etc. as appropriate. CLEF is always interested in receiving and facilitating innovative lab proposals. Potential task proposers who are unsure of the suitability of their task proposal or its format for inclusion at CLEF are encouraged to contact the CLEF 2020 Lab Organizing Committee Chairs to discuss its suitability or design at an early stage.

Proposal Submission

Lab proposals must provide sufficient information to judge the relevance, timeliness, scientific quality, benefits for the research community, and the competence of the proposers to coordinate the lab. Each lab proposal should identify one or more organizers as responsible for ensuring the timely execution of the lab. Proposals should be 3-4 pages long and should provide the following information:


Title of the proposed lab.


A brief description of the lab topic and goals, its relevance to CLEF

and the significance for the field.


A brief but clear statement of usage scenarios or domain to which the

activity is intended to contribute, including the evaluation setup and



Details on the lab organizer(s), including identifying the task chair(s)

responsible for ensuring the running of the task. This should include

details of any previous of involvement in organizing or participating in

evaluation tasks at CLEF or similar campaigns.


The planned format of the lab, i.e. campaign-style ("track") or workshop.


Is the lab a continuation of an activity from previous year(s) or a new



For activities continued from previous year(s): Statistics from

previous years (number of participants/runs for each task), a clear

statement on why another edition is needed, an explicit listing of the

changes proposed, and a discussion of lessons to be learned or insights to

be made.


For new activities: A statement on why a new evaluation campaign is

needed and how the community would benefit from the activity.


Details of the expected target audience, i.e. who do you expect to

participate in the task(s), and how do you propose to reach them.


Brief details of tasks to be carried out in the lab. The proposal should

clearly motivate the need for each of the proposed tasks and provide

evidence of its capability of attracting enough participation. The dataset

which will be adopted by the Lab needs to be described and motivated in the

perspective of the goals of the Labs; also indications on how the dataset

will be shared are useful. It is fine for a lab to have a single task, but

labs often contain multiple closely related tasks, needing a strong

motivation for more than 3 tasks, to avoid useless fragmentation.


Expected length of the lab session at the conference: half-day, one day,

two days. This should include very high level details of planned structure

of the session, e.g. participant presentations, invited speaker(s), panels,

etc., to justify the requested session length.


Arrangements for the organization of the lab campaign: who will be

responsible for activities within the task; how will data be acquired or

created, what tools or methods will be used, e.g., how will necessary

queries be created or relevance assessment carried out; any other

information which is relevant to the conduct of your lab.


If the lab proposes to set up a steering committee to oversee and advise

its activities, include names, addresses, and homepage links of people you

propose to be involved.

Lab proposals have to be submitted at the following address:


choosing the “CLEF 2020 Lab Proposals” track.

Reviewing Process

Each submitted proposal will be reviewed by the CLEF 2020 lab organizing committee. The decision will be sent by email to the responsible organizer by July 26, 2019. The final length of the lab session will be determined based on the overall organization of the conference and the number of participant submissions received by a lab.

Advertising Labs at CLEF 2019 and ECIR 2020

Organizers of accepted labs are expected to advertise their labs at both CLEF 2019 (9-12 September 2019, Lugano, Switzerland) and ECIR 2020 (14-17 April 2020, Lisbon, Portugal). So, at least one lab representative should attend these events.

Advertising at CLEF 2019 will consist of displaying a poster describing the new lab, running a break-out session to discuss the lab with prospective participants, and advertising/announcing it during the closing session.

Advertising at ECIR 2020 will consist of submitting a short lab description (1-2 pages) to be included in ECIR 2020 proceedings (15 October 2019), advertising the lab in a booster session during ECIR 2020, and displaying a poster about the lab.

Mentorship Program for Lab Proposals from newcomers

CLEF 2019 introduced a mentorship program to support the preparation of lab proposals for newcomers to CLEF. The program will be continued at CLEF 2020 and we encourage newcomers to refer to Friedberg et al. (2015) for initial guidance on preparing their proposal:

Friedberg I, Wass MN, Mooney SD, Radivojac P. Ten simple rules for a community computational challenge. PLoS Comput Biol. 2015 Apr 23;11(4):e1004150.

The CLEF newcomers mentoring program offers help, guidance, and feedback on the writing of your draft lab proposal by assigning a mentor to you, who help you in preparing and maturing the lab proposal for submission. If your lab proposal falls into the scope of an already existing CLEF lab, the mentor will help you to get in touch with those lab organizers and team up forces.

Lab proposals for mentorship have to be submitted at the following address:


choosing the “CLEF 2020 Lab Mentorship” track.

Important Dates


Requests for mentorship submission (only newcomers): 17 May 2019


Mentorship period: by 30 June 2019


Lab proposals submission (both newcomers and veterans): 5 July 2019


Notification of lab acceptance: 26 July 2019


Planning for lab breakouts at the next CLEF: late July 2019 - early

September 2019


Advertising labs at CLEF 2019: 9-12 September 2019, Lugano, Switzerland


Submission of short lab description for ECIR 2020: 15 October 2019


Labs registration opens: 5 November 2019


Advertising labs at ECIR 2020: 14-17 April 2020, Lisbon, Portugal

CLEF 2020 Lab Chairs


Carsten Eickhoff, Brown University, USA


Aurélie Névéol, LIMSI, CNRS, Université Paris Saclay, France

CLEF 2020 Lab Mentorship Chair


Lorraine Goeuriot, Université Grenoble Alpes, France

-- Aurélie Névéol, PhD

LIMSI-CNRS Bâtiment 507, bureau A111 Rue du Belvédère Université Paris-Sud 91403 ORSAY France

Tel: +33 (0)1 69 15 81 59 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 43664 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20190626/24c25719/attachment.txt>

More information about the Corpora mailing list