SubmittedReference Agreement Is Not Acoustic Correctness: Evaluating Large Audio Language Model Captions on Clotho
ERC-LM Workshop at ICASSP 2027
Evaluation · Submitted · ERC-LM @ ICASSP 2027
Does the caption match the sound?
Testing whether reference-based metrics agree with listening when ranking large audio language model captions on Clotho.
4,180captions from four LALMs
14reference-based and audio-aware metrics
47 / 50GPT-Audio-1.5 captions rated accurate by listening
The question
Audio captions are usually scored by how closely they match human-written references. Does a high reference score mean a caption describes what is actually audible, and do these metrics rank large audio language models the same way a listener would?
Approach
GPT-Audio-1.5, Gemini 3.8 Flash, Qwen2.5-Omni-7B, and Audio Flamingo 3 caption all 1,045 Clotho v2.1 evaluation recordings under one shared prompt, producing 4,180 captions.
Captions are scored with 12 reference-based metrics and 2 audio-aware metrics (CLAP similarity and MACE), against a matched human leave-one-out baseline. Model differences are tested with Holm-corrected permutation tests.
A blind review of 200 captions first compares each caption with its references and then judges it by listening to the recording.
Findings and scope
All four models scored below the human baseline on every reference-based metric, with Audio Flamingo 3 scoring highest.
Listening told a different story. GPT-Audio-1.5, ranked third or fourth by reference-based metrics, was rated accurate on 47 of 50 recordings, and 56 of 64 captions with weak reference agreement had no major acoustic error. Reference agreement is not acoustic correctness.
The manuscript is under review; results describe these four models, one prompt, and the Clotho v2.1 evaluation split.