qa4mre

  • Description:

QA4MRE dataset was created for the CLEF 2011/2012/2013 shared tasks to promote research in question answering and reading comprehension. The dataset contains a supporting passage and a set of questions corresponding to the passage. Multiple options for answers are provided for each question, of which only one is correct. The training and test datasets are available for the main track. Additional gold standard documents are available for two pilot studies: one on alzheimers data, and the other on entrance exams data.

FeaturesDict({
    'answer_options': Sequence({
        'answer_id': Text(shape=(), dtype=string),
        'answer_str': Text(shape=(), dtype=string),
    }),
    'correct_answer_id': Text(shape=(), dtype=string),
    'correct_answer_str': Text(shape=(), dtype=string),
    'document_id': Text(shape=(), dtype=string),
    'document_str': Text(shape=(), dtype=string),
    'question_id': Text(shape=(), dtype=string),
    'question_str': Text(shape=(), dtype=string),
    'test_id': Text(shape=(), dtype=string),
    'topic_id': Text(shape=(), dtype=string),
    'topic_name': Text(shape=(), dtype=string),
})
  • Feature documentation:
Feature Class Shape Dtype Description
FeaturesDict
answer_options Sequence
answer_options/answer_id Text string
answer_options/answer_str Text string
correct_answer_id Text string
correct_answer_str Text string
document_id Text string
document_str Text string
question_id Text string
question_str Text string
test_id Text string
topic_id Text string
topic_name Text string 
@InProceedings{10.1007/978-3-642-40802-1_29,
author="Pe{\~{n} }as, Anselmo
and Hovy, Eduard
and Forner, Pamela
and Rodrigo, {\'A}lvaro
and Sutcliffe, Richard
and Morante, Roser",
editor="Forner, Pamela
and M{\"u}ller, Henning
and Paredes, Roberto
and Rosso, Paolo
and Stein, Benno",
title="QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation",
booktitle="Information Access Evaluation. Multilinguality, Multimodality, and Visualization",
year="2013",
publisher="Springer Berlin Heidelberg",
address="Berlin, Heidelberg",
pages="303--320",
abstract="This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011--2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.",
isbn="978-3-642-40802-1"
}

qa4mre/2011.main.DE (default config)

  • Config description: This configuration includes the main track for DE language in 2011 year.

  • Download size: 217.08 KiB

  • Dataset size: 1.69 MiB

  • Splits:

Split Examples
'train' 120

qa4mre/2011.main.EN

  • Config description: This configuration includes the main track for EN language in 2011 year.

  • Download size: 197.74 KiB

  • Dataset size: 1.52 MiB

  • Splits:

Split Examples
'train' 120

qa4mre/2011.main.ES

  • Config description: This configuration includes the main track for ES language in 2011 year.

  • Download size: 212.52 KiB

  • Dataset size: 1.64 MiB

  • Splits:

Split Examples
'train' 120

qa4mre/2011.main.IT

  • Config description: This configuration includes the main track for IT language in 2011 year.

  • Download size: 209.73 KiB

  • Dataset size: 1.61 MiB

  • Splits:

Split Examples
'train' 120

qa4mre/2011.main.RO

  • Config description: This configuration includes the main track for RO language in 2011 year.

  • Download size: 216.32 KiB

  • Dataset size: 1.68 MiB

  • Splits:

Split Examples
'train' 120

qa4mre/2012.main.AR

  • Config description: This configuration includes the main track for AR language in 2012 year.

  • Download size: 347.83 KiB

  • Dataset size: 2.62 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.BG

  • Config description: This configuration includes the main track for BG language in 2012 year.

  • Download size: 434.63 KiB

  • Dataset size: 3.33 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.DE

  • Config description: This configuration includes the main track for DE language in 2012 year.

  • Download size: 275.00 KiB

  • Dataset size: 2.02 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.EN

  • Config description: This configuration includes the main track for EN language in 2012 year.

  • Download size: 237.76 KiB

  • Dataset size: 1.71 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.ES

  • Config description: This configuration includes the main track for ES language in 2012 year.

  • Download size: 271.92 KiB

  • Dataset size: 1.99 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.IT

  • Config description: This configuration includes the main track for IT language in 2012 year.

  • Download size: 273.49 KiB

  • Dataset size: 2.01 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.main.RO

  • Config description: This configuration includes the main track for RO language in 2012 year.

  • Download size: 272.99 KiB

  • Dataset size: 2.01 MiB

  • Splits:

Split Examples
'train' 160

qa4mre/2012.alzheimers.EN

  • Config description: This configuration includes the alzheimers track for EN language in 2012 year.

  • Download size: 173.19 KiB

  • Dataset size: 1.57 MiB

  • Splits:

Split Examples
'train' 40

qa4mre/2013.main.AR

  • Config description: This configuration includes the main track for AR language in 2013 year.

  • Download size: 369.44 KiB

  • Dataset size: 4.04 MiB

  • Splits:

Split Examples
'train' 284

qa4mre/2013.main.BG

  • Config description: This configuration includes the main track for BG language in 2013 year.

  • Download size: 452.74 KiB

  • Dataset size: 5.21 MiB

  • Splits:

Split Examples
'train' 284

qa4mre/2013.main.EN

  • Config description: This configuration includes the main track for EN language in 2013 year.

  • Download size: 268.52 KiB

  • Dataset size: 2.81 MiB

  • Splits:

Split Examples
'train' 284

qa4mre/2013.main.ES

  • Config description: This configuration includes the main track for ES language in 2013 year.

  • Download size: 307.78 KiB

  • Dataset size: 3.35 MiB

  • Splits:

Split Examples
'train' 284

qa4mre/2013.main.RO

  • Config description: This configuration includes the main track for RO language in 2013 year.

  • Download size: 306.16 KiB

  • Dataset size: 3.26 MiB

  • Splits:

Split Examples
'train' 284

qa4mre/2013.alzheimers.EN

  • Config description: This configuration includes the alzheimers track for EN language in 2013 year.

  • Download size: 267.98 KiB

  • Dataset size: 2.50 MiB

  • Splits:

Split Examples
'train' 40

qa4mre/2013.entrance_exam.EN

  • Config description: This configuration includes the entrance_exam track for EN language in 2013 year.

  • Download size: 53.32 KiB

  • Dataset size: 186.01 KiB

  • Splits:

Split Examples
'train' 46