The main task offers monolingual and cross-language search on library catalog records in English, French, and German, organised in collaboration with The European Library (TEL). The second task focuses more on linguistic issues, offering retrieval on test collections in languages that pose processing challenges. . The third task is the robust task which aims at assessing whether word sense disambiguated (WSD) data does impact on IR system performance.
The task is to search and retrieve relevant items from collections of library catalog cards. Our aim is to identify the most effective retrieval technologies for searching this type of data.
This data is very different from the news corpora previously used in the CLEF ad hoc track, consisting of bibliographic data (document surrogates). Whereas in the traditional ad hoc task, the user searches for a document containing information of interest, here the user will be searching to identify which publications are of potential interest – according to the information provided by the catalog card. The question the user is asking is “Is the publication described by the bibliographic record relevant to my information need?”
The collections have been provided by The European Library (www.theeuropeanlibrary.org) and the task is organised in collaboration with TEL. Three target collections are provided:
We have tagged the 3 collections (BL, BNF, ONB) as English, French and German because in each case this is the main/official language of the collection. However, all three collections are to some extent multilingual and contain documents (catalog records) in many additional languages. Thus the title and maybe (if existing) an abstract or description can be in a different language to that understood as the language of the collection. The subject heading information is normally in the main language of the collection.
Remember that this is structured data but the records tend to be very sparse. many records contain only title, author and subject heading information; other records provide more details.
About 66% of the documents in the English and German
collection have textual subject headings, in the French collection 37%.
Dewey Classification (DDC) is: not available in the
French collection; negligible (<0.3%) in the German collection; but occurs in
about half of the English documents (456,408 docs to be exact).
50 topics have been prepared for each of the 3 main collection languages (DE; EN; FR). Topics can be prepared in other languages on demand. We expect to have topics available also in Chinese, Greek and Polish.
Topics have 2 fields: Title field – 2-4 key terms; Description field: a sentence specifying the information item of interest
2 main tasks are offered: monolingual and bilingual - subdivided into subtasks to reflect the multilinguality of the data:
Monolingual; Monolingual+ and Bilingual; Bilingual+
The + tasks are tasks where the participating group also attempts to use additional tools to cater for the multilinguality of the collections. Groups must state whether their runs are to be considered as “+”
In both tasks: Monolingual and Bilingual, the aim is to retrieve documents relevant to the query - and your results are judged in this respect.
By monolingual we mean that the query is in the same language as the official language of the collection.
By bilingual we mean that the query is in a different language to the official language of the collection.
For example, in an EN -> FR run, relevant documents (bibliographic records) could be any document in the BNF collection (which we call the French collection) in whatever language they are written. The same is true for a monolingual FR -> FR run - relevant documents from the BNF collection could actually also be in English or German, not just French.
Documents referring to all types of works (e.g. books, articles, collections of images, videos, etc.) are judged for relevance unless the query specifically indicates otherwise.
In CLEF2009 the task we simulate is that of a user who has a working knowledge of English, French and German and who wants to discover the existence of relevant documents that can be useful for him/her in one of our three target collections (either monolingually - the query is in the official language of the collection; or bilingually, the query is in a different language).
We will judge for relevance only those documents that are written totally or partially in one of these languages, e.g. a catalog record written, for example, entirely in Hungarian will be counted as not relevant for our hypothetical user. However, if any part of the record is written in one of the 3 languages (e.g. a catalog record with perhaps the title and a brief description in Hungarian, but with subject descriptors in French, German or English), or if the record contains terms which match against the query terms (e.g. a catalog record in Hungarian contains named entities which can be matched against named entities in the query) it will be judged for relevance as it could be potentially useful for the user.
Our assessors have no additional knowledge of the documents referred to by the catalog records (or surrogates) contained in the collection. They judge for relevance on the information contained in the records made available to the systems.
The + runs are those runs where the participating system has used additional tools to cater for the multilinguality of these collections (e.g. language identification tools, additional multilingual dictionaries, etc.) It will be interesting to see whether systems that use additional tools have better performance.
We were somewhat disappointed last year because only a few groups really attempted to address the specificity of this data; most groups just submitted runs using their favourite (CL)IR approach.
For this reason, we highly recommend that participants in this task do try to implement specific strategies to cater for the specificity of the TEL collections: structured, sparse and potentially multilingual data. We would like to see submissions from groups which include a base-line run, plus additional runs in which different strategies have been attempted
The aim of this task is to investigate the best approaches for retrieval from library catalogs, where the information is frequently very sparse and, as we have found, is often stored in unexpected languages. This is in fact very much a real world task and provide useful input for the European Digital Library (now known as Europeana).
Contact: Carol Peters, ISTI-CNR (carol.peters@isti.cnr.it) or Nicola Ferro, U. Padua (ferro@dei.unipd.it)
2. Persian@CLEF
This task is run in collaboration with the Database Research Group of the University of Tehran. It will use the Hamshahri corpus of 1996-2002 newspapers. A very complete description can be found on the Hamshahri website. Monolingual and bilingual (EN - > FA) tasks will be offered. Last year's topics are available as training topics. The objective is to query the target collection using topics in the same language (monolingual run) or topics in English (bilingual run) and to submit the results in a list ranked in decreasing order of relevance. Contact Abolfazl AleAhmad (a.aleahmad@ece.ut.ac.ir) or Hadi Amiri (h.amiri@ece.ut.ac.ir), DBRG, University of Tehran.