A benchmark for voice-agent decisions

DECIDE

Decision Evaluation in Calls:
Intent, Dialogue and Emotion

From hearing a conversation to deciding what happens next.

Harsh Pare · Naman Jain · Sachin Kumar Singh · Jayakrishna Sahit
Siddharth Batra · Sanket Bhat · Surya Vamsee Ayyagari

tazo/labsCall data from Bolna

Evaluate a model Code Paper Coming soon Listen to examples

4,867: decision questions
1,081: audio clips
32: decision tasks
3–25s: clip duration

01 / Overview

What should a voice agent do next?

A caller says “hello?” Is it a greeting, a request to repeat, or an interruption? The words alone may leave the decision unresolved. Voice agents need to combine what was said with how it sounded and what happened earlier in the call.

DECIDE evaluates these local decisions using short clips from real production conversations supplied by Bolna. Each question pairs an audio clip with permitted context and a set of candidate decisions. The task is to choose the decision supported at the end of the clip.

DECIDE overview: clips from Bolna calls, permitted earlier context and candidate decisions lead to one decision at the clip boundary, across six decision families. — **Figure 1.** From real calls to local decisions. Each question combines a fixed audio boundary, permitted context and candidate options. The waveforms are schematic; real clips are available in the six examples below. Open figure ↗

The benchmark covers 570 source calls and 34 agents, predominantly in college admissions, with English and Hindi speech. A clip can support multiple questions. Reference decisions were manually annotated by a team of four.

02 / Tasks

Decisions across the voice-agent stack.

Question counts by family: business and conversation 1,403; graph routing 743; tool decisions 128; turn taking 376; voice properties 1,886; language switching 331. Total: 4,867. — **Figure 2.** Question distribution across the six decision families in DECIDE v1.0. A single audio clip can support more than one question. Open figure ↗

Task descriptions and accessible chart data

Six decision families and representative questions
Decision family	What the model decides	Questions
Business & conversation	Lead interest, availability, callback requests, course preferences and caller responses.	1,403
Graph routing	Stay, advance, handle an interruption or close; choose the next graph node.	743
Tool decisions	Which external tool is appropriate, and whether its required information is available.	128
Turn taking	Whether the agent or caller should speak next, or whether the call should end.	376
Voice properties	Caller emotion, perceptually distinct speakers and audible disturbance.	1,886
Language switching	Whether the spoken language changes within the clip.	331

03 / Audio examples

Listen. Consider the context. Decide.

Six examples give a feel for the questions in DECIDE. These previews use shortened context; the evaluation toolkit supplies the versioned benchmark prompts.

Loading audio examples…

Only these six preview clips are public. The full recordings, transcripts, contexts and reference labels are held privately.

04 / Evaluation

Bring your own decision model.

Evaluate a direct-audio or STT + text model through a compatible API. Our worker reads the private benchmark and returns aggregate scores; participants do not download the dataset.

01
Submit your APIAudio or STT + text. Public-example validation.
02
Run the benchmarkPrivate worker. Fixed questions and references.
03
Get aggregate scoresAccuracy by task, family and clip duration.

Start with the public preview

git clone https://github.com/tazolabs/DECIDE.git
cd DECIDE
pip install -r requirements.txt
python evaluation.py validate --preview

Evaluation instructions

05 / Citation

Reference DECIDE.

The paper and arXiv link will be added here when available.

Provisional citation

@misc{decide2026,
  title = {DECIDE: Decision Evaluation in Calls:
           Intent, Dialogue and Emotion},
  author = {Pare, Harsh and Jain, Naman and
            Singh, Sachin Kumar and Sahit, Jayakrishna and
            Batra, Siddharth and Bhat, Sanket and
            Ayyagari, Surya Vamsee},
  year = {2026},
  howpublished = {\url{https://github.com/tazolabs/DECIDE}}
}