There will be 7 main evaluation tracks in 2008
This track tests mono- and cross-language text retrieval.
We offer a totally new main task for monolingual and cross-language search on library catalogue records. The task is organised in collaboration with The European Library (TEL) and searching will be on collections
derived from the TEL archives in English, French, and German. We
also offer more traditional mono- and bilingual ad-hoc retrieval tasks on a very
exciting Persian newspaper corpus: the Hanmshari collection. The "robust" task will be offered this year using a word sense disambiguated
collection of news documents in English and offering mono- and bilingual tasks. Hard topics from previous years will be chosen to give the possibility of
developing advanced techniques to deal with them. The track is coordinated by ISTI-CNR (IT), U.Padua (IT),
U.Tehran (IR), U.Hildesheim (DE)
and U. Basque Country (ES).
See here for more information and also
http://ixa2.si.ehu.es/clirwsd. Mono- and cross-language domain specific retrieval on structured bibliographic data for the social sciences is studied. The following corpora are provided: GIRT-4 for German/ English, CSA Sociological Abstracts for English, ISISS for Russian. Multilingual controlled vocabularies (English, German, Russian) and bi-directional mappings between terminologies are available. Topics are offered in English, German and Russian. The track is coordinated by GESIS-IZ Bonn. See
http://www.gesis.org/en/research/information_technology/clef_ds.htm Interactive retrieval of images using the popular image perusal community Flickr (www.flickr.com) as a target database will again be studied. Flickr is a dynamic, rapidly changing database of images with textual comments provided by creators and viewers in a self-organizing ontology of tags (a so-called "folksonomy"). This labeling activity is naturally multilingual, reactive, and cooperative. In 2008, the focus will be on measuring relevance, user confidence/satisfaction and user behaviour on a larger scale than previous years; to serve this purpose, a single multilingual interface to Flickr will be user by all participants. The track is coordinated by UNED, Madrid, (ES). See http://nlp.uned.es/iCLEF for details. Both main tasks (QA) and exercises (AVE, QAST, QA-WSD) are proposed. The main task scenario is event-targeted QA on a heterogeneous document collection (news articles and Wikipedia). Many monolingual and cross-language sub-tasks are offered:
Basque, Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish are proposed as both query and target languages; other query languages may be added. The additional exercises are the following: The track is organized by several institutions (one for each source language) and jointly coordinated by CELCT, Trento (IT) and UNED, Madrid (ES). See http://clef-qa.itc.it/ and http://nlp.uned.es/clef-qa/ This track evaluates retrieval of images from multi-lingual collections; both text and visual retrieval techniques are exploitable. Four challenging tasks are offered: (i) multilingual ad-hoc retrieval (collection with mixed English/German/Spanish annotations, queries in many languages), (ii) medical image retrieval (queries in several languages, with visual, semantic and mixed topics), (iii) hierarchical medical image annotation (fully categorized collection), (iv) topic detection (non-annotated collection, several concepts need to be detected). Visual image retrieval is not required for all tasks and a default visual and textual retrieval system will be made available for participants. The track coordinators are U. Sheffield (UK), U. of Geneva (CH), Oregon Health and Science U. (US), Victoria U. Melbourne (AU), RWTH Aachen (DE), Vienna U. of Technology (AT). See also: http://www.imageclef.org/
for a full description of all tasks. In the past three years this track has focused on evaluation of systems
providing multi- and cross-lingual access to web data. WebCLEF 2008 will repeat
the task setup of the 2007 edition. In 2008, we evaluate a multilingual
information synthesis task, where, for a given topic, participating systems are
asked to extract important snippets from web pages (fetched from the live web
and provided by the task organizers). The systems will have to focus on
extracting, summarizing, filtering and presenting information relevant to the
topic, rather than on large scale web search and retrieval per se. In the 2008
edition of the task, we will focus on refining the assessment procedure and
evaluation measures. WebCLEF 2008 has lots of similarities with (topic-oriented)
multi-document summarization and with answering complex questions. An important
difference is that at WebCLEF, topics may come with extensive descriptions and
with many thousands of documents from which important facts have to be mined. In
addition, WebCLEF works with web documents, that may be very noisy and redundant. For WebCLEF 2008 participants, the test collection, topics, assessments, and
source code of the best performing 2007 system are available, together with
evaluation scripts, so as to create a shared baseline. Tentative timeline for WebCLEF 2008: WebCLEF 2008 is organized by the University of Amsterdam and backed by the
following advisory board:
For more details, tools, resources, and signing up please visit
http://ilps.science.uva.nl/WebCLEF/WebCLEF2008/ Multilingual document retrieval with an emphasis on geographic search (GIR) is evaluated. The 2008 GeoCLEF track will consist of several parts: A modification of the existing GeoCLEF search task on newspaper collections. Some topics will simulate the situation of a user who poses a query when looking at a map on the screen. For these topics, the system will receive the content part and a rectangular shape which defines the geographic context. The query parsing task: This sub-task will again be run by Microsoft Research Asia who will be supplying a logfile containing 800,000 queries. The participants are required to identify the geographic aspects of the query. Two pilot tasks offer new search challenges: For further information (including details on track coordinators),
see
http://www.uni-hildesheim.de/geoclef/ CLEF 2008 also offers two new tracks as pilot tasks The Vid2RSS feed task is a classification task performed on a video corpus containing episodes of a dual language television program. Task participants will be provided with speech recognition transcripts, metadata and keyframes for the video data. The languages of the television program will be Dutch and English. It is important to note that the languages occur side by side in the program, both contributing to the spoken content of the program and are not translations of the same content. The task is to group videos into topic categories and generate an RSS-feed for each bcategory. The videos are classified (i.e. assigned to the topic categories) using the speech recognition transcripts for both languages spoken in the program. Keyframes and metadata can support the generation of the RSS-feeds, but can also can be used to support classification, if participants chose. See http://ilps.science.uva.nl/Vid2RSS/. Infile (information filtering evaluation) extends the last filtering track of TREC 2002 in the following ways:
Scientific Data Retrieval (Domain-Specific)
Interactive Cross-Language Retrieval (iCLEF)
Multiple Language Question Answering (QA@CLEF).
Cross-Language Image Retrieval (ImageCLEF)
CLEF Web Track (WebCLEF)
Julio Gonzalo, Universidad Nacional de Educacion a Distancia
Valentin Jijkoun, University of Amsterdam
Jimmy Lin, University of Maryland>
Constantin Orasan, University of Wolverhampton
Maarten de Rijke, University of AmsterdamCross-Language Geographical Information Retrieval (GeoCLEF)
Cross-Language Video Retrieval (VideoCLEF)
Multilingual Information Filtering (INFILE@CLEF)