CLEF 2005
| Agenda
CLEF 2005 offered a series of evaluation tracks to test different aspects of information retrieval system development. The aim was to promote research into the design of user-friendly, multilingual, multimodal retrieval systems.
The
ad-hoc track tested system performance on a multilingual collection of
news documents. There
were two main tasks this year testing bilingual (L1->L2)
and monolingual non-English information retrieval systems.
Monolingual
and bilingual
retrieval tasks were offered
for Bulgarian, French, Hungarian
and Portuguese
collections.
In the bilingual task,
newcomers
only (i.e. groups that have not previously participated in a CLEF
cross-language task) could choose to search the English document collection using any topic
language. A
common set of topics (i.e. structured statements of information needs from
which queries are extracted) was prepared in
Amharic,
Bulgarian,
Chinese,
English, French, German,
Hungarian, Indonesian,
Italian, Portuguese,
Russian,
and
Spanish.
The multilingual
task was based on the CLEF 2003 multilingual-8 test collection
and aimed
at measuring progress over time in multilingual (L1->Ln) retrieval system
performance.
The track was coordinated jointly by ISTI-CNR,
U.Padua and
Dublin
City U.
This track studied retrieval in a domain-specific context using the GIRT-4 German/English social science database (as a pseudo-parallel corpus with identical documents) and the Russian Social Science Corpus (RSSC). Multilingual controlled vocabularies (German-English, English-German, German-Russian, English-Russian) were available. Monolingual and cross-language tasks were offered. Topics were provided in English, German and Russian. Participants could use the indexing terms inside the documents and/or the Social Science Thesaurus provided, not only as translation means, but also for tuning relevance decisions of their system. This track was coordinated by IZ Bonn.
The challenge was to build a system that will allow real people to find information that is written in languages that they have not mastered, and then measure how well representative users are able to use the system that has been built. This year the iCLEF track focused on the problems of cross-language question answering and image retrieval from a user-inclusive perspective. Participating groups adapted a shared user study design to test a hypothesis of their choice, comparing reference and contrastive systems. The track was coordinated by LSI-UNED and the QA/Image CLEF organizers, ITC-irst, CELCT Trento, and U. Sheffield. See the iCLEF website.
Following the positive outcome of the 2003 and 2004 QA@CLEF evaluation campaigns, in this evaluation exercise monolingual (non-English) and cross-language QA systems were tested. Combinations between nine or more source languages (Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese and Spanish) and eight target language collections (Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese and Spanish) were explored. The trackl included a main task, where factoid and definition questions were given as input, and a pilot tasks, that explored facets of multilingual QA on the WWW. The track was coordinated by ITC-Irst and CELCT, Trento. See the QA@CLEF website.
This track evaluated retrieval of images described by text captions based on queries in a different language; both text and image matching techniques were potentially exploitable. Four tasks were offered:
bilingual ad hoc retrieval (collection in English, queries in several languages)
interactive search
medical image retrieval (collection in English and French, queries are images)
The tasks offered different and challenging retrieval problems for cross-language image retrieval. The first task was also envisaged as an entry level task for newcomers to CLEF and to CLIR.
A number of test collections were available, including: St Andrews University historical photographic collection; the ImageCLEFmed collection made available by the University and University Hospitals Geneva in collaboration with Oregon Health and Science University (OHSU); the IRMA database of 10,000 medical images, copyright the IRMA group, Aachen University of Technology (RWTH), Germany (use is currently limited to the ImageCLEF competition). ImageCLEF was coordinated by Sheffield University, the University and U.Hospitals of Geneva, Oregon Health and Science U., Aachen RWTH and Victoria University. For more information see the ImageCLEF website.
Mono- and cross-language retrieval was assessed on the Malach collection of spontaneous conversational speech from the Shoah archives. The collection for 2005 was in English and consisted of approx. 750 hours in topically coherent segments plus ~5 keywords and 3 sentence summary. A thesaurus with ca 3000 core concepts, 30000 location-time pairs and is-a and part-whole relationships, an in-domain expansion collection, and a word lattice for part of the collection was also available on arrangement. 25 topics in 6 languages were prepared: Czech, English, French, German, Russian and Spanish. 40 existing topics could be used for training. There were 5-level relevance judgements. The track was coordinated by Dublin City U., Ireland, and U. Maryland, USA. See the CL-SR website for more information.
For multi/crosslingual retrieval the web is the natural and common setting. In the European context, many issues for which people turn to the web are essentially multilingual. These include law, economy, culture, education, leisure, travel. The WebCLEF document collection consists of webpages from European governmental sites for at least 10 languages/countries. The collection contained about 2M pages, with at least be 50K documents for each major language. The documents were in various formats: HTML, TXT, PDF, etc. Multilingual navigation tasks such as home page finding and named page finding were assessed, modeling users that access governmental information in the EU. The topic development and relevance assessment process was in the hands of participants. The track was coordinated by the University of Amsterdam. See the WebCLEF website.
The track provided a framework in which to evaluate GIR systems for search tasks involving both spatial and multilingual aspects. Given a statement describing a spatial user need (topic), the challenge was to find relevant documents from target collections of British English and German (or Spanish) news documents. 26 topics were prepared in several languages including English, Spanish, Italian and German. They were structured in the form <concept><spatial relation><region>, e.g. "find stories about disasters in Geneva". The track was run as a pilot experiment and was coordinated by UC Berkley and U. Sheffield. See the GeoCLEF website.