Overview
Istella22 enables comparison of feature-based Learning to Rank methods and neural approaches based on pre-trained language models. It provides query and document text together with industrial query-document feature vectors.
- 8.4 million web documents
- 220 query-document features
- 2,198 textual test queries
- Relevance judgments on a five-level scale
How to cite this dataset
If you use this dataset in your research, please acknowledge Istella SpA and cite the following publication:
Domenico Dato, Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto. 2022. The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 10.1145/3477495.3531740.
BibTeX
@inproceedings{istella22,
author = {Domenico Dato and Sean MacAvaney and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
title = {The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation},
booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2022},
doi = {10.1145/3477495.3531740},
url = {https://doi.org/10.1145/3477495.3531740},
pages = {3099--3107},
}
Download
The dataset is available solely for non-commercial use. Read and accept the license before downloading the files. Read the license.
Optional supplement: document identifiers
Added in May 2023. The file test-svm-docIds.gz (7.67 MB) contains document identifiers for the query-document pairs in test.svm, aligned row by row. Use it with the Istella22 dataset to link test examples to corpus documents.