CLEF 2004 |
Agenda
CLEF 2004 offered a series of evaluation tracks to test different aspects of information retrieval system development. The aim was to promote research into the design of user-friendly, multilingual, multimodal retrieval systems.
These tracks (also called the ad hoc tracks) tested system performance on a multilingual collection of news documents. The document collection for CLEF2004 Ad-Hoc contained English, Finnish, French, Portuguese and Russian documents. A common set of topics (i.e. structured statements of information needs from which queries are extracted) was prepared in Amharic, Bulgarian, Dutch, English, Finnish, French, German, Italian, Portuguese, Spanish, Swedish, Russian, Japanese and Chinese. These tasks were coordinated by CNR-ISTI, ELRA/ELDA, Eurospider, IZ-Bonn, Linguateca, U.Tampere.
Using a selected topic language, the goal for systems was to retrieve relevant documents for all languages in the collection, listing the results in a single, ranked list.
The 2004 bilingual track on news collections accepted runs for the following source -> target language pairs:
Newcomers only (i.e. groups that had not previously participated in a CLEF cross-language task) could choose to search the English document collection using any topic language.
The CLEF experience has demonstrated the importance of monolingual system performance for multiple languages as a first step towards cross-language work. CLEF 2004 offered monolingual free-text retrieval tasks for Finnish, French, Portuguese and Russian (plus a task for German monolingual retrieval on structured data, see below).
The rationale for this track is to study retrieval in a domain-specific context using the GIRT-4 German/English social science database. GIRT-4 data is offered as pseudo-parallel German and English corpora. Multilingual controlled vocabularies (German-English, German-Russian) are available. Monolingual and cross-language tasks were offered in CLEF 2004. Topics were available in English, German and Russian.
The coordinator of this track
was Michael Kluck, IZ-Bonn (kluck@bonn.iz-soz.de)
In 2004, the
interactive CLEF track studied the problem of
Cross-Language Question Answering from a user-inclusive perspective. Depending
on the perspective, the challenges were twofold: from
the point of view of QA as a machine task (QA systems), interaction with the
user may help a QA engine to retrieve better answers; from the point of view of
QA as a user task, a search assistant may help the user in locating the answer
faster and more easily. The track was
coordinated by LSI-UNED and U.Maryland. See the
iCLEF
website for more
information.
Mono- and Cross-Language QA systems were tested. Target languages involved were Dutch, French, German, Italian, Portuguese, Spanish, English. Questions were mostly factoid, but the test sets also included also definition queries and questions that do not have a known answer in the target corpora. The track was coordinated by ITC-irst, LSI-UNED, DFKI, ELRA/ELDA, NIST, U.Amsterdam, U.Limerick. For further details, see the QA@CLEF website.
This track evaluated retrieval of images described by text captions based on queries in a different language; both text and image matching techniques were potentially exploitable. The track offered three tasks: (1) a bilingual ad hoc retrieval task, (2) an interactive search task (tentative), and (3) a medical image retrieval task. The tasks offered different and challenging retrieval problems for cross-language image retrieval. The first task is also envisaged as an entry level task for newcomers to CLEF and to CLIR. Two test collections were available St Andrews University historical photographic collection; University Hospitals Geneva medical images. ImageCLEF was coordinated by U.Sheffield and University Hospitals Geneva. See the ImageCLEF website.
This track aimed at the evaluation of CLIR systems on noisy automatic transcripts of spoken documents (from the TREC SDR collections), and addressed the following problems: bilingual-SDR from Dutch, French, German, Italian, and Spanish; retrieval with/without known story boundaries; use of multiple automatic transcriptions. It was coordinated by ITC-irst, and DCU. See the CL-SDR website.