There were 7 main evaluation tracks in 2008
This track tested mono- and cross-language text retrieval.
We offered a totally new main task for monolingual and cross-language search on library catalogue records. The task was organised in collaboration with The European Library (TEL) and searching was on collections
derived from the TEL archives in English, French, and German. We
also offered more traditional mono- and bilingual ad-hoc retrieval tasks on a very
exciting Persian newspaper corpus: the Hanmshari collection. The "robust" task was offered this year using a word sense disambiguated
collection of news documents in English and offering mono- and bilingual tasks. Hard topics from previous years were chosen to give the possibility of
developing advanced techniques to deal with them. The track was coordinated by ISTI-CNR (IT), U.Padua (IT),
U.Tehran (IR), U.Hildesheim (DE)
and U. Basque Country (ES).
See here for more information and also
http://ixa2.si.ehu.es/clirwsd. Mono- and cross-language domain specific retrieval on structured bibliographic data for the social sciences was studied. The following corpora were provided: GIRT-4 for German/ English, CSA Sociological Abstracts for English, ISISS for Russian. Multilingual controlled vocabularies (English, German, Russian) and bi-directional mappings between terminologies were available. Topics were offered in English, German and Russian. The track was coordinated by GESIS-IZ Bonn. See
http://www.gesis.org/en/research/information_technology/clef_ds.htm Interactive retrieval of images using the popular image perusal community Flickr (www.flickr.com) as a target database was studied. Flickr is a dynamic, rapidly changing database of images with textual comments provided by creators and viewers in a self-organizing ontology of tags (a so-called "folksonomy"). This labeling activity is naturally multilingual, reactive, and cooperative. In 2008, the focus was on measuring relevance, user confidence/satisfaction and user behaviour on a larger scale than previous years; to serve this purpose, a single multilingual interface to Flickr was used by all participants. The track was coordinated by UNED, Madrid, (ES). See http://nlp.uned.es/iCLEF for details. Both main tasks (QA) and exercises (AVE, QAST, QA-WSD) were proposed. The main task scenario was event-targeted QA on a heterogeneous document collection (news articles and Wikipedia). Many monolingual and cross-language sub-tasks were offered:
Basque, Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish were proposed as both query and target languages. The additional exercises were the following: The track was organized by several institutions (one for each source language) and jointly coordinated by CELCT, Trento (IT) and UNED, Madrid (ES). See http://clef-qa.itc.it/ and http://nlp.uned.es/clef-qa/ This track evaluated retrieval of images from multi-lingual collections; both text and visual retrieval techniques were exploitable. Four challenging tasks were offered: (i) multilingual ad-hoc retrieval (collection with mixed English/German/Spanish annotations, queries in many languages), (ii) medical image retrieval (queries in several languages, with visual, semantic and mixed topics), (iii) hierarchical medical image annotation (fully categorized collection), (iv) topic detection (non-annotated collection, several concepts need to be detected). Visual image retrieval was not required for all tasks and a default visual and textual retrieval system were made available for participants. The track coordinators were U. Sheffield (UK), U. of Geneva (CH), Oregon Health and Science U. (US), Victoria U. Melbourne (AU), RWTH Aachen (DE), Vienna U. of Technology (AT). See also: http://www.imageclef.org/
for a full description of all tasks. In the past three years this track had focused on evaluation of systems
providing multi- and cross-lingual access to web data. WebCLEF 2008 repeated
the task setup of the 2007 edition. In 2008, we evaluated a multilingual
information synthesis task, where, for a given topic, participating systems were
asked to extract important snippets from web pages (fetched from the live web
and provided by the task organizers). The systems had to focus on
extracting, summarizing, filtering and presenting information relevant to the
topic, rather than on large scale web search and retrieval per se. In the 2008
edition of the task, we focused on refining the assessment procedure and
evaluation measures. WebCLEF 2008 had lots of similarities with (topic-oriented)
multi-document summarization and with answering complex questions. An important
difference was that at WebCLEF, topics could come with extensive descriptions and
with many thousands of documents from which important facts had to be mined. In
addition, WebCLEF worked with web documents, that could be very noisy and redundant. For WebCLEF 2008 participants, the test collection, topics, assessments, and
source code of the best performing 2007 system were available, together with
evaluation scripts, so as to create a shared baseline. Tentative timeline for WebCLEF 2008: WebCLEF 2008 was organized by the University of Amsterdam and backed by the
following advisory board:
For more details, tools, resources, and signing up please visit
http://ilps.science.uva.nl/WebCLEF/WebCLEF2008/ Multilingual document retrieval with an emphasis on geographic search (GIR) was evaluated. The 2008 GeoCLEF track consisted of several parts: A modification of the existing GeoCLEF search task on newspaper collections. Some topics simulated the situation of a user who posed a query when looking at a map on the screen. For these topics, the system received the content part and a rectangular shape which defined the geographic context. The query parsing task: This sub-task was run by Microsoft Research Asia who supplied a logfile containing 800,000 queries. The participants were required to identify the geographic aspects of the query. Two pilot tasks offered new search challenges: For further information (including details on track coordinators),
see
http://www.uni-hildesheim.de/geoclef/ CLEF 2008 also offers two new tracks as pilot tasks The Vid2RSS feed task was a classification task performed on a video corpus containing episodes of a dual language television program. Task participants was provided with speech recognition transcripts, metadata and keyframes for the video data. The languages of the television program were Dutch and English. It is important to note that the languages occurred side by side in the program, both contributing to the spoken content of the program and were not translations of the same content. The task was to group videos into topic categories and generated an RSS-feed for each bcategory. The videos were classified (i.e. assigned to the topic categories) using the speech recognition transcripts for both languages spoken in the program. Keyframes and metadata supported the generation of the RSS-feeds, but were also used to support classification, if participants chose. See http://ilps.science.uva.nl/Vid2RSS/. Infile (information filtering evaluation) extended the last filtering track of TREC 2002 in the following ways:
Scientific Data Retrieval (Domain-Specific)
Interactive Cross-Language Retrieval (iCLEF)
Multiple Language Question Answering (QA@CLEF).
Cross-Language Image Retrieval (ImageCLEF)
CLEF Web Track (WebCLEF)
Julio Gonzalo, Universidad Nacional de Educacion a Distancia
Valentin Jijkoun, University of Amsterdam
Jimmy Lin, University of Maryland>
Constantin Orasan, University of Wolverhampton
Maarten de Rijke, University of AmsterdamCross-Language Geographical Information Retrieval (GeoCLEF)
Cross-Language Video Retrieval (VideoCLEF)
Multilingual Information Filtering (INFILE@CLEF)