CLEF 2006 | Agenda

CLEF Agenda for 2006 

CLEF 2006 offered a series of evaluation tracks to test different aspects of information retrieval system development. The aim is to promote research into the design of user-friendly, multilingual, multimodal retrieval systems. Information on the test collections available for each track can be found in the instructions on How to Participate.

There were eight evaluation tracks in 2006

Mono-, Bi- and Multilingual Document Retrieval on News Collections (Ad-Hoc)

The ad-hoc track tested mono- and cross-language textual document retrieval. Similarly to 2005, the 2006 track offered mono- and bilingual tasks on target collections in French, Portuguese, Bulgarian and Hungarian (possibly also Polish). Topics (i.e. statements of information needs from which queries are derived) were prepared in a wide range of European languages. We also offered a bilingual task aimed at encouraging system testing with non-European languages against an English target collection. Topics were supplied in a variety of languages including Amharic, Oromo, Hindi, Telugu and Indonesian.

In addition, a new “robust” task was offered; this task emphasized the importance of stable performance over languages instead of high average performance in mono-, cross-language and multilingual IR. The robust task is essentially an ad-hoc task which makes use of test collections previously developed at CLEF. The evaluation methodology considered the geometric average as well as the mean average precision of all topics. Geometric average has proven to be a stable measure for robustness. Data collections were provided in six languages and a topic set of some 150 topics.In the long term, we are interested in topic difficulty and failure analysis for hard topics. The track was coordinated jointly by ISTI-CNR and U.Padua (Italy) and U.Hildesheim (Germany). For further details, see the Ad-Hoc website.

Mono- and Cross-Language Information Retrieval on Structured Scientific Data (Domain-Specific)

Domain-specific retrieval was studied using the GIRT-4 German/English social science and economics database (as a pseudo-parallel corpus with identical documents in two languages), and Russian sociology database data from the Russian Social Science Corpus (RSSC), and the ISSIC database. Multilingual controlled vocabularies (German/English, German-Russian, English-Russian) were available. Mono- and cross-language tasks were offered. Topics were available in English, German and Russian. Participants could use the indexing terms in the documents and/or the social science thesaurus provided, not only for translation but also to tune relevance decisions of their systems. The track was coordinated by IZ Bonn (Germany). See the Domain-Specific website for more information.

 

Interactive Cross-Language Information Retrieval (iCLEF)

For CLEF 2006, the interactive track joined forces with the image track to work on a new type of interactive image retrieval task to better capture the interplay between image and the multi-lingual reality of the internet for the public at large. The task was based on the popular image perusal community Flickr (www.flickr.com), a dynamic and rapidly changing database of images with textual comments, captions, and titles in many languages and annotated by image creators and viewers cooperatively in a self-organizing ontology of tags (a so-called “folksonomy”). The track was coordinated by UNED (Spain), U. Sheffield (UK) and SICS (Sweden). See the iCLEF website.

 

Multiple Language Question Answering
This track, which has received increasing interest at CLEF since 2003, evaluated both monolingual (non-English) and cross-language QA systems. Questions are posed in a source language and answers are searched for in a document collection of a target language. The languages were Bulgarian, Dutch, English, French, German, Italian, Portuguese and Spanish. All combinations between them will be explored. The main task evaluated open domain QA systems which find exact answers for factoid and definition questions; in addition, a pilot task evaluated cross-language QA systems in a real, user-oriented scenario. There was also a pilot task that assessed question answering using Wikipedia, the online encyclopedia, and an Answer Validation Exercise. The track was organised by several Institutions (one for each language) and coordinated by ITC-irst and CELCT, Trento (Italy). Information for participants was available at the QA@CLEF website.

Cross-Language Retrieval in Image Collections (ImageCLEF)

This track evaluated retrieval of images described by text captions based on queries in a different language; both text and image matching techniques were potentially exploitable. Five tasks were offered in 2006:

The tasks offered different and challenging retrieval problems  for cross-language image retrieval. The first task was also envisaged as an entry level task for newcomers to CLEF and to CLIR. Image analysis was not required for all tasks and a default visual image retrieval system was made available for participants as well as results from a basic text retrieval system. Four test collections was made available.  

The track coordinators are University of Sheffield (UK) and the University and U.Hospitals of Geneva (Switzerland), Oregon Health and Science University (USA), Victoria University, Melbourne (Australia), RWTH Aachen University (Germany), and Vienna University of Technology (Austria) collaborate in the task organisation. For more information see the ImageCLEF2006 flyer and the ImageCLEF website.

Cross-Language Speech Retrieval (CL-SR)

In 2005, the CL-SR track built a reusable test collection for searching spontaneous conversational English speech using queries in five languages (Czech, English, French, German and Spanish), speech recognition for spoken words, manually and automatically assigned controlled vocabulary descriptors for concepts, dates and locations, manually assigned person names, and hand-written segment summaries. The 2006 CL-SR track extended that collection to include additional English speech (about 900 hours), additional resources (word lattices and more accurate speech recognition), and a no-boundary evaluation condition.  A second test collection containing at least 500 hours of Czech speech was also created.  Multilingual topic sets with 25 topics were created for each language in 2006.  The track was coordinated by the University of Maryland (USA) and Dublin City University (Ireland). See  for details. See the CL-SR website for more information.

Multilingual Web Track (WebCLEF)

For multi/crosslingual retrieval the web is the natural and common setting. In the European context, many issues for which people turn to the web are essentially multilingual. These include law, economy, culture, education, leisure, travel. WebCLEF 2006 used an updated version of the EuroGOV collection, with web pages crawled from European governmental sites for over 20 languages/countries. The collection contains over 3M pages, with at least 50K documents for each major language. The documents were in various formats: HTML, TXT, PDF, etc. For WebCLEF 2006, multilingual navigation tasks such as home page finding and named page finding were assessed, modeling users that access governmental information in the EU. The track was coordinated by the University of Amsterdam (The Netherlands).  See the WebCLEF website.

Cross-Language Geographical Retrieval (GeoCLEF)

After its success as a pilot experiment in CLEF 2005, GeoCLEF became a regular track in CLEF 2006. The track provides a framework in which to evaluate GIR systems for search tasks involving both spatial and multilingual aspects. The aim is to compare methods of query translation, query expansion, translation of geographical references, use of text and spatial retrieval methods separately or combined, retrieval models and indexing methods. Given a multilingual statement describing a spatial user need (topic), the challenge is to find relevant documents from target collections in English and Portuguese (plus possibly German and/or Spanish) news documents. 25 topics were prepared in several languages including English, Spanish, Portuguese and German. They were structured in the form <concept><spatial relation><region>, e.g. "find stories about disasters in Geneva". Spatial analysis was not required to participate in this task but could be used to augment text-retrieval methods. The track is coordinated by UC Berkley (USA) and U. Sheffield (UK). See the  GeoCLEF 2006 flier and the GeoCLEF website.