BMVC 2026

CountVideoBench

A benchmark to evaluate prediction biases on the object counting task by video-language models.

Marc Chapus · Carlos Crispim-Junior · Véronique Eglin · Atilla Baskurt

INSA Lyon, Université Lumière Lyon 2, CNRS, Université Claude Bernard Lyon 1, LIRIS, UMR5205, Villeurbanne, France

Video coming soon
5 yellow cubes
Video coming soon
20 red cubes
Video coming soon
50 blue spheres

CountVideoBench is a diagnostic benchmark for object counting in video-language models. Counting questions in existing video benchmarks concentrate on a few small values, so average accuracy hides how models behave on less frequent counts. CountVideoBench contains 2,600 synthetic videos spread evenly over 26 counts between 5 and 50, each showing a single class of static objects under a moving camera. Across sixteen models, none exceeds 27.6% Exact Match. Accuracy does not simply decrease as counts grow: models concentrate their answers on a few preferred values, with accuracy peaks at those values and collapses at neighbouring counts.

Dataset

Videos2,600, with 100 per count
Counts26 between 5 and 50, in groups of consecutive values 5–12 · 18–22 · 28–32 · 38–42 · 48–50
ObjectsOne class per video: cube or sphere, in five colours
SceneStatic objects, moving camera
FormatMP4 · 1920×1080 · 30 fps · 5 s
Ground truthExact, from the generator
LicenseCC BY 4.0 · 139 GB

Quick start

CountVideoBench is hosted on the Hugging Face Hub. Load it in Python with the datasets library, or download the video files with the Hugging Face command-line tool hf. One install provides both, and the dataset is public: no account is needed.

Install
pip install -U datasets
Python · datasets
from datasets import load_dataset, Video

# Downloads the full dataset (139 GB) on first call
ds = load_dataset("countvideobench/CountVideoBench", "single_class", split="test")
# Keep file paths instead of decoded videos, to sample frames your own way
ds = ds.cast_column("video", Video(decode=False))

sample = ds[0]
print(sample["question"], sample["count"], sample["video"]["path"])
Command line · hf
# All 2,600 videos
hf download countvideobench/CountVideoBench --repo-type dataset --local-dir CountVideoBench

# Or one count only, e.g. the 100 videos with 20 objects
hf download countvideobench/CountVideoBench --repo-type dataset --local-dir CountVideoBench \
  --include "single_class/metadata.csv" "single_class/20_entities/*"

Use the question field as the prompt to compare with the paper.

Comparison with existing benchmarks

In each of the nine video benchmarks below, at least 72% of counting questions have an answer of 10 or less. CountVideoBench gives every count between 5 and 50 the same weight.

Distribution of ground-truth counts in each benchmark, each row scaled to its own maximum. The shaded area covers counts 0 to 10; counts above 50 are grouped in the last bar.

Results

Sixteen video-language models from nine families, evaluated zero-shot on ten uniformly sampled frames per video.

0%100%

Exact Match by ground-truth count, over all 2,600 videos.

ModelEMAcc@10%MAERMSECoverage|S|PP

EM, Acc@10% and Coverage in percent; MAE and RMSE in objects, over answered videos. |S|: number of distinct values predicted; PP: prediction perplexity. Bold marks the best value per column. † answers only 1.3% of videos and is not ranked on MAE and RMSE.

On a 104-video subset, 20 human participants reach 90.8% Exact Match, against 26.9% for the best models on the same videos.

To add your model to the table, send its predictions to marc.chapus@liris.cnrs.fr.

Citation

@inproceedings{chapus2026countvideobench,
  author    = {Chapus, Marc and Crispim-Junior, Carlos and Eglin, V{\'e}ronique and Baskurt, Atilla},
  title     = {{CountVideoBench}: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models},
  booktitle = {37th British Machine Vision Conference 2026, {BMVC} 2026, Lancaster, UK, November 23-26, 2026},
  publisher = {BMVA},
  year      = {2026}
}