Publications

Proceedings

Evaluating Systems for Multilingual and MultiModal Information Access
9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denamrk, September 17-19, 2008, Revised Selected Papers
Series: Lecture Notes in Computer Science , Vol. 5706
Peters, C.; Deselaers, Th.; Ferro, N.; Gonzalo, J.; Jones, G.J.F.; Kurimo, M.; Mandl, Th.; Peñas, A.; Petras, V. (Eds.)
2009, XXIV, 1002 p. With online files/update., Softcover
ISBN: 978-3-642-04446-5

Advances in Multilingual and MultiModal Information Retrieval
8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, Budapest, Hungary, September 19-21, 2007, Revised Selected Papers
Series: Lecture Notes in Computer Science , Vol. 5152
Peters, C.; Jijkoun, V.; Mandl, Th.; Müller, H.; Oard, D.W.; Peñas, A.; Petras, V.; Santos, D. (Eds.)
2008, XXI, 922 p. With online files/update., Softcover
ISBN: 978-3-540-85759-4

Evaluation of Multilingual and Multi-modal Information Retrieval
7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain, September 20-22, 2006, Revised Selected Papers
Series: Lecture Notes in Computer Science, Vol. 4
730

Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (Eds.)
2007, XXIV, 998 p. With online files/update., Softcover
ISBN: 978-3-540-74998-1

Accessing Multilingual Information Repositories
6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria, 21-23 September, 2005, Revised Selected Papers
Series: Lecture Notes in Computer Science, Vol. 4022
Peters, C.; Gey, F.; Gonzalo, J.; Mueller, H.; Jones, G.; Kluck, M.; Magnini, B.; de Rijke, M. (Eds.)
2006, XIX, 1013 p. Also available online., Softcover

Multilingual Information Access for Text, Speech and Images
Fifth Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers
Series: Lecture Notes in Computer Science, Vol. 3491
Peters, C.; Clough, P.; Gonzalo, J.; Jones, G.J.F.; Kluck, M.; Magnini, B. (Eds.)
2005, XVI, 845 p.

Comparative Evaluation of Multilingual Information Access Systems
Fourth
Workshop of the Cross-Language Evaluation Forum, CLEF 2003, Trondheim, Norway, August 2003.Revised papers.
Lecture Notes in Computer Science 3237, Springer 2004.
Carol Peters, Julio Gonzalo, Martin Braschler, Michael Kluck (Eds.). Springer 2004

Advances in Cross-Language Information Retrieval
Third Workshop of the Cross-Language Evaluation Forum, CLEF 2002, Rome, Italy, September, 2002.Revised papers. ISSN 0302-9743, ISBN 3-540-40830-4 
Lecture Notes in Computer Science 2785, Springer 2003, 828p.
Carol Peters, Martin Braschler, Julio Gonzalo, Michael Kluck (Eds.). Springer 2003. 

Evaluation of Cross-Language Information Retrieval Systems
Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001, Darmstadt, Germany, September 3-4, 2001.Revised papers. ISSN 0302-9743, ISBN 3-540-44042-9 
Lecture Notes in Computer Science 2406, Springer 2002, 601p.
Carol Peters, Martin Braschler, Julio Gonzalo, Michael Kluck (Eds.). Springer 2002. 

Cross-Language Information Retrieval and Evaluation
Workshop of Cross-Language Evaluation Forum, CLEF 2000, Lisbon, Portugal, September 21-22, 2000, Revised Papers. Lecture Notes in Computer Science 2069, Springer 2001, 387p.
C. Peters (Ed.)

Working Notes

       

        Results of the CLEF 2007 Cross-Language System Evaluation Campaign

      Working Notes for the CLEF 2007 Workshop
          21-23 September, Budapest, Hungary

          A. Nardi and C. Peters  (Ed.)

        Results of the CLEF 2006 Cross-Language System Evaluation Campaign

      Working Notes for the CLEF 2006 Workshop
          21-23 September, Alicante, Spain

            A. Nardi, C. Peters and J.L. Vicedo (Ed.)

        Results of the CLEF 2005 Cross-Language System Evaluation Campaign

      Working Notes for the CLEF 2005 Workshop
          21-23 September, Vienna, Austria

          C. Peters (Ed.)

       

        Results of the CLEF 2004 Cross-Language System Evaluation Campaign

        Working Notes for the CLEF 2004 Workshop
          15-17 September, Bath, UK

          C. Peters and F. Borri (Eds.)

        Results of the CLEF 2003 Cross-Language System Evaluation Campaign

        Working Notes for the CLEF 2003 Workshop
          21-22 August, Trondheim, Norway

          C. Peters (Ed.)

Results of the CLEF 2002 Cross-Language System Evaluation Campaign

Working Notes for the CLEF 2002 Workshop
19-20 September, Rome, Italy

C. Peters (Ed.)

Results of the CLEF 2001 Cross-Language System Evaluation Campaign

Working Notes for the CLEF 2001 Workshop
3-4 September, Darmstadt, Germany

C. Peters (Ed.)

Results of the CLEF 2000 Cross-Language System Evaluation Campaign

Working Notes for the CLEF 2000 Workshop
 21-22 September, Lisbon, Portugal

C. Peters (Ed.)

Project Presentations

Project Overview

Presented at EU-NSF Digital Library All Projects Meeting
25-26 March 2002, Rome, Italy

Results of CLEF 2003

Presented at TREC-12, November 2003, Gaithersburg, USA

Journals

Special Issue of Information Retrieval dedicated to CLEF

Volume 7, Nos.1/2, January/April 2004

Papers

Braschler, M., Peters, C. (2002). The CLEF Campaigns: Evaluation of Cross-Language Information Retrieval Systems, UPGRADE, pages 78-81 (UPGRADE: The European Online Magazine for the IT Professional, Vol. III, Issue no. 3, CEPIS).

See: http://www.upgrade-cepis.org/issues/2002/3/upgrade-vIII-3.html 

Also available in Spanish

See: http://www.ati.es/novatica/2002/157/nv157-mono-2.pdf

Gonzalo, J. (2002). Scenarios for Interactive CLIR systems, Cross-Language Information Retrieval: A Research Roadmap, Workshop at SIGIR-2002, Tampere Finland August 15, 2002

Kluck, M., Womser-Hacker, C. (2002).  Inside the Evaluation Process of the Cross-Language Evaluation Forum (CLEF): Issues of Multilingual Topic Creation and Multilingual Relevance Assessment. In: M. G. Rodríguez, C. P. S. Araujo (Eds.): Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, Las Palmas de Gran Canaria 29-31 May 2002, Paris: ELRA  

Preprint: http://www.educat.hu-berlin.de/~kluck/lrec-paper-final.doc or  http://www.educat.hu-berlin.de/~kluck/lrec-paper-final.pdf Additional Poster at: http://www.educat.hu-berlin.de/~kluck/lrec2002-poster.pdf

Kluck, M. (2002). Das Cross-Language Evaluation Forum (CLEF) - Evaluationsumgebung und Forschungskontext für mehrsprachiges Information Retrieval (mit einer Skizze der Ergebnisse von CLEF 2002). In: Rainer Hammwöhner, Christian Wolff, Christa Womser.-Hacker (Eds.): Information und Mobilität. Optimierung und Vermeidung von Mobilität durch Information. Proceedings des 8. Internationalen Symposiums für Informationswissenschaft (ISI 2002). Konstanz: UVK 2002, 225-237. Preprint : http://www.educat.hu-berlin.de/~kluck/clef-isi2002-2.rtf . Slides of the talk with additional information at: http://www.educat.hu-berlin.de/~kluck/isi2002-clef2002.ppt

Peters, C., Braschler, M. (2002). The Importance of Evaluation for Cross-language System Development: the CLEF Experience. In: M. G. Rodríguez, C. P. S. Araujo (Eds.): Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, Las Palmas de Gran Canaria 29-31 May 2002, Paris: ELRA

Peters, C. (2002). Evaluating Cross-Language Systems the CLEF Way. Cultivate Interactive 6

See: http://www.cultivate-int.org/issue6/clef/

Peters, C. (2002). The Contribution of Evaluation, Cross-Language Information Retrieval: A Research Roadmap, Workshop at SIGIR-2002, Tampere Finland August 15, 2002

Peters, C., Braschler, M. (2001). "Cross-Language System Evaluation: the CLEF Campaigns", Journal of the American Society for Information Science and Technology, 52(12):1067-1072, 2001.  

Bibliography

Publications by CLEF participants that cite CLEF results

 

Project Deliverables available to the Public

 

Due Date

WP1

User Needs D1.1.1 Report on User Needs

31/01/02 

WP2

Evaluation Infrastructure

D2.2.1

User-Licence for CLEF Participants

31/12/01

D2.2.2

User-Licence for after-CLEF Use

31/03/04

WP3

Test Data

D3.2.1

Test Collection Report for Campaign 1

31/10/02

D3.2.2

Test Collection Report for Campaign 2

31/10/03

WP4

Evaluation Campaigns

D4.2.1

Results Interpretation and Report (Campaign 1)

31/12/02

D4.2.2

Results Interpretation and Report (Campaign 2)

31/12/03

WP5

Exploitation and Dissemination

D5.1.1

Workshop Working Notes (Evaluation Campaign 1)

30/11/02

D5.1.2

Workshop Working Notes (Evaluation Campaign 2)

30/11/03

D5.2.2

Test-suite

31/03/04