Agreement Concerning Dissemination of CLEF 2001 Results

The annual Cross-Language Evaluation Forum (CLEF) is an event in which organisations with an interest in cross-language information retrieval take part in a coordinated series of experiments using the same experimental data and queries. The results of these individual experiments are then presented at a workshop where tentative comparisons may be made. In order to preserve the desired, pre- competitive nature of these conferences, the organisers have developed a set of guidelines constraining the dissemination and publication of CLEF evaluation results. These guidelines are meant to preclude the publication of incomplete or inaccurate information that could damage the reputation of the conference or its participants and could discourage participation in future conferences. The guidelines apply to all CLEF participants, regardless of the type of organisation or institution involved. 

A signed agreement is required of each organisation participating in the CLEF evaluations. Any organisation that is found to have violated the terms or spirit of the agreement may be denied participation in future CLEFs. 

GUIDELINES: 

1. SCIENTIFIC OR TECHNICAL PUBLICATIONS: Scientific or technical publications, including newsletters from universities or research laboratories, should adhere to community standards for fairness and objectivity and should accurately and clearly state the limitations of the testing conditions and other factors which might influence scores. The experimental nature of the tasks, data and evaluation procedures should also be stated. The full conference proceedings should always be referenced. 

2. ADVERTISEMENTS: No advertisements using the CLEF evaluation results can be placed in magazines, journals, newspapers, or other publications. 

3. PRESS RELEASES: Press releases about CLEF results to organisations with national/international coverage are also prohibited. 

4. MARKETING LITERATURE, LOCAL NEWSLETTERS: Although it is recognised that extensive evaluation discussions are not appropriate in this type of literature, it is expected that any claims made on the basis of evaluation results are accurate, that the evaluation measures used to substantiate these claims are stated, and that a reference is made to the full conference proceedings. Where promotional material is subject to prepublication revision by the media, the author should make every effort to see that the revision does not cause a violation of the guidelines. 

5. CROSS-SYSTEM COMPARISONS: Cross-system comparisons may not be made with other named teams for individual tests, and may only be made when they are supported by accepted methods of statistical significance testing. Comparisons must be accompanied by the results of those tests and should reference the publication of those tests in the conference proceedings or related literature. Informal, qualitative comparisons with recognised baselines or benchmarks, and with general levels or trends in performance, must be clearly stated to be such and thus open to statistical reassessment. 

6. PUBLICATION WITH CLEF REFERENCES: A copy of any publication that quotes or contains references to the CLEF evaluation results must be provided to the CLEF coordinator. 

I, _____________________________________________________

(name), acting on behalf of 
____________________________________________ (organisation), acknowledge my understanding of the above guidelines and accept this agreement. As a condition of our participation in the CLEF evaluation, my organisation and its employees and representatives agree to abide by these guidelines. I am qualified to make this commitment. 

Signature: _______________________________________________

Date:        _______________