Call for participation SISAP 2025 Indexing Challenge

March 4th, 2025 Daniela Lopez de Luise
The SISAP Indexing Challenge 2025 invites researchers and practitioners to participate in exciting tasks to advance the state of the art in similarity search and indexing. The challenge provides a platform for presenting innovative solutions and pushing the boundaries of efficiency and effectiveness in large-scale similarity search indexes. This year, we are opening two challenging tasks.
Datasets can be found at https://huggingface.co/datasets/sadit/SISAP2025/tree/main; you can clone the full repository or download each file.
This task challenges participants to develop memory-efficient indexing solutions with reranking capabilities. Each solution will be run in a Linux container with limited memory and storage resources.
  • Container specifications: 8 virtual CPUs, 16 GB of RAM, the dataset will be mounted read-only into the container.
  • Wall clock time for the entire task: 12 hours.
  • Minimum average recall to be considered in the final ranking: 0.7.
  • Dataset: PUBMED23 (23 million vectors (384 dimensions) with out-of-distribution queries).
  • The goal is to evaluate k=30 nearest neighbors for a large set of query objects, as follows:
    • The final score of each team is measured as the best throughput evaluated on up to 16 different search hyperparameters.
    • Teams are provided with a public set of 11,000 query objects for development purposes.
    • A private set of 10,000 new queries will be used for the final evaluation.
In this task, participants are asked to develop memory-efficient indexing solutions that will be used to compute an approximation of the k-nearest neighbor graph for k=15. Each solution will be run in a Linux container with limited memory and storage resources.
  • Container specifications: 8 virtual CPUs, 16 GB of RAM, the dataset will be mounted read-only into the container.
  • Wall clock time for the entire task: 12 hours.
  • Minimum average recall to be considered in the final ranking: 0.8.
  • Dataset: GOOAQ (3 million vectors (384 dimensions) ).
  • The goal is to compute the k-nearest neighbor graph (without self-references), i.e., find the k-nearest neighbors using all objects in the dataset as queries.
    • We will measure graph’s quality as the recall against a provided gold standard and the full computation time (i.e., including preprocessing, indexing, and search, and postprocessing)
    • We provide a development dataset; the evaluation phase will use an undisclosed dataset of similar size computed with the same neural model.
For data description, hardware specifications, registration and participation instructions, please, refer to https://sisap-challenges.github.io/2025/index.html
All participants will be considered for paper submissions. We aim to accommodate all accepted papers within the conference program. Papers should be short, focusing on the presentation and poster.
We look forward to your participation and innovative solutions in the SISAP Indexing Challenge 2025! Let's push the frontiers of similarity search and indexing together.
Any transformation of the dataset to load, index, and solve nearest neighbor queries is allowed. Transformations include but are not limited to, packing into different data types, dimensional reduction, locality-sensitive hashing, product quantization, or transforming into binary sketches. Reproducibility and open science are primary goals of the challenge, so we accept only public GitHub repositories with working GitHub Actions as submissions. Indexing algorithms may be already published or original contributions.
You can find more detailed information, data access, and registration at the SISAP Indexing Challenge website https://sisap-challenges.github.io/2025/
  • June 6th. Submission of solution implementations deadline.
  • June 13th. Short paper descriptions deadline.
  • July 1st. Final ranking announcement.
  • July 11th. Paper notification.
  • July 31st. Participant (short paper) camera ready.

Posted in Congress
«
»
You can leave a response, or trackback from your own site.

Leave a Reply

Design by 2b Consult