QA4MRE 2012 evaluation ongoing
The main objective of this exercise is to develop a methodology for evaluating Machine Reading systems through Question Answering and Reading Comprehension Tests. Systems should be able to extract knowledge from large volumes of text and use this knowledge to answer questions.
The Machine Reading task addresses the problem of building a bridge between knowledge encoded as natural text and the formal reasoning systems that need such knowledge. In contrast to text mining (or text harvesting, sometimes also called macro-reading), where the system reads and combines evidence from hundreds or thousands of texts, MR is the task of obtaining an in-depth understanding of just one, or a small number, of texts. In fact, the task will focus on the reading of single documents, where correct answers require some inference and the consideration of previously acquired background knowledge.
Beside the Main Task, also two pilot tasks are offered this year at QA4MRE:
Processing Modality and Negation for Machine Reading, aimed at evaluating whether systems are able to understand extra-propositional aspects of meaning like modality and negation. Modality is a grammatical category that allows expressing aspects related to the attitude of the speaker towards his/her statements. Negation is a grammatical category that allows changing the truth value of a proposition.
Machine Reading of Biomedical Texts about Alzheimer, aimed at setting questions in the Biomedical domain with a special focus on one disease, namely Alzheimer. This pilot task will explore the ability of a system to answer questions using scientific language. Texts will be taken from Medline (Medical Literature Analysis and Retrieval System Online) abstracts.
The results of the evaluation will be presented at the CLEF 2012 Conference, Rome, September 17-20, 2012 [http://clef2012.org/index.php].
For more information on QA4MRE see http://celct.fbk.eu/QA4MRE/.