|
CLEF
2001: WORKSHOP OF THE CROSS-LANGUAGE EVALUATION FORUM
3 - 4 September,
Darmstadt, Germany
(immediately preceding
ECDL 2001)
The Workshop was held over two days. On the
first day, the results of the CLEF 2001 evaluation campaign were reported and analysed.
The second day focussed on the current needs of cross-language systems and how
evaluation campaigns in the future can best be designed to stimulate progress. Proposals
for future activities were presented that include other media, not just text, and other
types of evaluation, not only system performance. The organization of the CLEF 2002
campaign was also discussed.
Attendance on the first day was strictly limited to participants in the CLEF 2001
campaign, whereas on the second day anyone interested in the activities of the
Cross-Language Evaluation Forum was welcome to attend.
Programme
Day 1 - CLEF 2001
Evaluation Campaign
(Evaluation Campaign Participants only)
| 9.00 - 9.15 |
Welcome:
Carol
Peters - IEI-CNR, Pisa, Italy |
| 9.15 - 12.30 |
Multilingual
Information Retrieval |
|
Chair: Martin Braschler - Eurospider, Switzerland
The morning session will present and analyse the results of the multilingual task. Talks
will be given by groups who participated in this track, and the final part will be
dedicated to a general discussion of main issues that emerge during the morning.
Presentations will include: |
|
Overview of Results: Martin
Braschler, Eurospider, Switzerland |
|
Multilingual Information
Retrieval Using English and Chinese Queries: Aitao Chen, University of
California, Berkeley, USA |
|
Report on CLEF-2001 Experiments:
Jacques Savoy, Université de Neuchâtel, Switzerland |
|
JHU/APL Experiments at
CLEF-2001: Translations Resources and Score Normalization:
Paul McNamee, John
Hopkins University, USA |
|
Comparing Translation Resources:
Wessel Kraaij, TNO-TPD, The Netherlands
|
|
Working with Russian
Queries for the GIRT, Bilingual and Multilingual CLEF Tasks: Fredric Gey, Hailing
Jang and Natalia Perelman, University of California, Berkeley, USA |
|
Open
Discussion |
| 12.30 - 14.00 |
Lunch |
| 14.00 - 16.00 |
From Monolingual
to Bilingual Retrieval |
|
Chair: Djoerd Hiemstra - University of Twente, The Netherlands
The presentations and discussion in this session will focus in particular on
language-dependent system development issues. |
|
University of Amsterdam at
CLEF 2001: Christof Monz and Maarten de Rijke, University of Amsterdam, The Netherlands |
|
Vector-based Semantic
Analysis and Morphological Analysis for Cross-Lingual Information Retrieval: Jussi
Karlgren, Magnus Sahlgren, SICS, Sweden |
|
Exeter at CLEF 2001:
Experiments with Machine Translation for Bilingual Retrieval: Gareth J. F. Jones
and Adenike M. Lam-Adesina, University of Exeter, UK |
|
Using Statistical
Translation Models for Bilingual IR: Jan-Yun Nie, University of Montréal, Canada |
|
UTACLIR @ CLEF 2001: Turid
Hedlund, Heikki Keskustalo, Ari Pirkola, Eija Airio and Kalervo Järvelin, University of
Tampere, Finland |
| 16.00 - 16.30 |
Break |
| 16.30 - 18.00 |
Poster Session |
|
|
| Day 2
- |
Looking
to the Future: CLEF 2002 and beyond
(Everyone is welcome) |
| 9.00 - 9.30 |
Introduction to CLEF:
Carol Peters - IEI-CNR, Pisa, Italy
The introduction is intended for newcomers to CLEF plus anyone interested in finding out
more about the CLEF objectives and activities. CLEF veterans are not expected to attend
this introductory session. |
| 9.00 - 10.30 |
New tasks for CLEF |
|
Chair:
Peter Schäuble - Eurospider, Switzerland
So far the CLEF campaigns have been focussed on the performance of text retrieval systems.
This session will report on the results of an experiment for interactive system evaluation
and proposals for other types of evaluation tasks will be presented. |
|
The Interactive CLEF
Track: Julio Gonzalo - UNED, Madrid, Spain, and Doug Oard, University of Maryland, USA
|
|
iCLEF at Maryland:
Comparing Term-for-Term Gloss and MT:
Jianqiang Wang and Doug Oard, University of Maryland, USA
|
|
iCLEF at Sheffield:
Zoë Bathie and Mark Sanderson, University of Sheffield, UK (presented by Daniela Petrelli)
|
|
Looking Ahead: iCLEF 2002:
Julio Gonzalo - UNED, Madrid, Spain, and Doug Oard, University of Maryland, USA
|
|
Cross-Language Spoken Document Evaluation Proposal: Marcello Federico - ITC-IRST,
Trento, Italy
Any other proposals and general discussion |
| 10.30 - 11.00 |
Break |
| 11.00 - 12.30 |
International
Evaluations for IR |
|
Chair:
Christa Womser-Hacker - University of Hildesheim, Germany
The activities of two other international initiatives for cross-language system evaluation
will be reported and the issues involved in building test collections for IR will be
presented. |
|
Report from TREC-9:
Donna Harman - National Institute for Standards and Technology (NIST), USA
CLIR System Evaluation at NTCIR:
Noriko Kando - National Institute of Informatics
(NII), Japan
Philosophy of IR System Evaluation:
Ellen Voorhees - National Institute for
Standards and Technology (NIST), USA |
| 12.30 - 14.00 |
Lunch |
| 14.00 - 16.00 |
Organizing CLEF 2002 |
|
Chair:
Michael
Kluck - IZ Sozialwissenschaften, Bonn, Germany
This session consisted of a discussion of the various aspects involved in the setting
up of the CLEF infrastructure and the organization of the evaluation campaigns. Plans for
CLEF 2002 was presented. Issues discussed included resource-sharing,
inclusion of new tasks (see above), coverage of new languages.
|
|
Issues involved in multilingual topic generation:
Christa Womser-Hacker - University of Hildesheim, Germany |
|
Resource Sharing for CLEF: Jussi Karlgren,
SICS, Sweden |
|
Relevance Assessment in Practice:
Djoerd Hiemstra, University of Twente, The Netherlands |
|
Plans for 2002: Carol Peters, IEI-CNR, Italy |
|