Beyond a caption’s benchmark score

01 / MOTIVATION

The question

What do standard caption metrics tell us about model outputs when humans and models are evaluated under the same reference conditions?

02 / METHOD

The approach

The study evaluates 1,000 LLM-generated captions against 5,000 human references from MS COCO, using BLEU, METEOR, and CIDEr.

Matched leave-one-out comparisons put model and human captions under identical four-reference conditions. A review of 50 captions examines omissions, additional detail, and reference mismatches.

03 / EVIDENCE

Findings & scope

Human captions showed higher agreement on BLEU-4 and CIDEr, while METEOR slightly favored model captions. Model outputs averaged 2.27 times the length of human captions.

Reference agreement is not equivalent to visual correctness. Differences in detail and length matter when interpreting metric scores.