Projects

QA@CLEF: Multilingual Question Answering Track at CLEF

DESCRIPTION:

Started in 2000, the Cross Language Evaluation Forum (CLEF), supports global digital library applications by developing an infrastructure for the testing, tuning and evaluation of IR systems operating on European languages in both monolingual and cross-language contexts. The major goal is to encourage research and the development of truly multilingual, multimodal systems for European languages, through:

• the creation of evaluation infrastructure and the organization of regular evaluation campaigns for system testing;
• the building of a MLIA/CLIR research community;
• the construction of public available test-suites.

QA4MRE - Question Answering for Machine Reading Evaluation - is a new task of the Question Answering (QA) Track at CLEF 2011.
The goal is to evaluate Machine Reading abilities through Question Answering and Reading Comprehension Tests. The focus is on capturing knowledge from given text collections and using it to answer questions. The task consists of the reading of single documents, where correct answers require some inference and previously-acquired background knowledge. Like in Reading Comprehension Tests, questions will be in the form of multiple choice questions. The participating systems will be required to answer the questions by choosing in each case one answer from the alternatives proposed.
Coordination: UNED (Spain), ISI (USA), CELCT (Italy), University of Limerick (Ireland).
For more information see: http://celct.fbk.eu/QA4MRE/

CELCT’S ROLE:

CELCT has been coordinating the Multilingual Question Answering Track since 2005, which implies writing track guidelines; creating the necessary datasets (i.e. both training and test data); building the Gold Standard corpus for the evaluation of systems output and assessing results. The evaluation exercise culminates in a final workshop where results are discussed and future activities are planned.

Since 2004, CELCT has edited the CLEF proceedings published by Springer in LNCS (Lecture Notes in Computer Science) series.
Since 2005 the center also participated as organizers of the Workshops: CLEF 2005 Vienna (Austria),  CLEF 2006 Alicante (Spain), CLEF 2007 Budapest  (Hungary), CLEF 2008 Aarhus (Denmark), CLEF 2009 (Greece), CLEF 2010 (Italy).

CELCT has been also a sponsor of the annual conference together with the Network of Excellence for Digital Libraries (DELOS), Microsoft, the European Language Resources Association (ELRA) and Evaluation & Language resources Distribution Agency (ELDA).
Moreover, since  2008, CELCT, together with UNED, organize and coordinate all evaluation activities which implies:

• define the task
• provide data to test and tune systems
• define metrics to be used in results assessment

The aim is to produce a task that allows a comparative evaluation among systems and approaches.

LINKS: