Two-speaker interviews: what speech recognition does and does not do

Whisper speaker diarization is the one thing people expect from a two-person interview and the one thing the model does not do. Feed the recording to a tool built on OpenAI’s open-source Whisper model and you get the words and the timings back in one undivided block, with nothing to mark which voice said what. Separating audio by speaker is a different task, handled by different software. That leaves you two sensible options: add the labels yourself using the timestamps, which takes a few minutes, or pay for a service that does it.

The model returns text and times, and nothing else

The Whisper paper, “Robust Speech Recognition via Large-Scale Weak Supervision”, spells out the exact shape of what comes back.

Audio goes in as fixed windows: the authors break audio files into 30-second segments paired with the subset of the transcript that occurs within that time segment. Text comes back wrapped in time markers, the start time token predicted before each caption’s text and the end time token after. Those times are rounded, “quantizing all times to the nearest 20 milliseconds”.

Now count the fields. A segment has a start time, an end time and some words. There is no fourth field, so there is nowhere to put a speaker. Nothing in that format ties a caption boundary to a change of voice either, which is why a segment can run straight through a turn, one person’s last four words sharing a line with the other person’s first six.

That is why your interview came back as a wall of text. The tool didn’t fail. It handed you exactly what this kind of model produces.

Whisper speaker diarization is a separate component the model leaves out

The paper names diarization once, in the section that explains what the model was designed to do. Predicting which words were spoken, it says, is a core part of the speech recognition problem but not the only part. A fully featured system can involve further components such as voice activity detection, speaker diarization and inverse text normalization, and those components are “often handled separately, resulting in a relatively complex system around the core speech recognition model”.

That is the authors drawing a line around their own scope, with diarization outside it.

One of the three components in that sentence did get pulled in, which sharpens the contrast. Voice activity detection is listed in the openai/whisper README among the tasks the models perform: “multilingual speech recognition, speech translation, spoken language identification, and voice activity detection”. The paper matches, saying that where a segment holds no speech the model is trained to predict a no-speech token. Speaker diarization is not in the README’s task list, and the README doesn’t mention speaker labels anywhere.

So there is no model size to upgrade to and no setting to switch on. Anything that prints “Speaker 1” is a second piece of software that grouped the audio by voice, then stitched its answer onto the transcript’s timings.

What you want, what the transcript gives you, what to do

What you want	What the transcript gives you	What to do
Speaker 1 and Speaker 2 on each line	Plain text, no identity attached	Type the two labels in yourself, working from the gaps
A clean break at every change of turn	Segments that end where the model’s caption ends	Split at the sentence boundary and label each half
The time each turn started	A start and an end time on every segment	Export SRT or VTT, not TXT
Real names on the labels	Nothing, the model never hears a name as an identity	Say both names at the top, then label by initials
Talk time per person	Nothing until the lines are labelled	Label first, then count, or buy diarization
Overlapping speech pulled apart	One line of text, nothing to show two people spoke at once	Listen back to those spots, record separate tracks next time

Labelling a two-speaker transcript yourself takes a few minutes

Stop treating it as a listening job. You are hunting for turn changes, and the timestamps tell you where to look.

Export SRT or VTT, not TXT. Every cue then carries a start and an end time. Our post on SRT, VTT and plain text covers the difference.
Open the transcript beside the audio. Any player with a visible time counter works, because you will be jumping to specific seconds.
Mark who speaks first. Listen to the opening ten seconds, then label cue one.
Find the gaps. Compare each cue’s end time with the next cue’s start time. The larger gaps are your candidate turn changes, because the pause before an answer is usually longer than a pause inside one.
Spot the questions. A cue ending in a question mark is usually the interviewer, and the cue after it the other person. That one pattern resolves most of an interview.
Check only the candidates. Jump to each candidate time, listen for two seconds, confirm or reject. You never play the recording end to end.
Fix the split lines last. Where a turn changed mid-cue, add a line break and give each half its own label.

Keep the labels short and identical every time, so find and replace works later. “Q:” and “A:” or two sets of initials are enough. Once they are in, the other fixes worth making are in our guide to transcript mistakes to fix before you publish.

How you record decides how much of that work you have to do

Every minute spent on the recording saves several on the transcript.

Record each person to their own file when your tool offers it. Some meeting and recording tools can save a separate audio file per participant. Check your recording settings before the call, not after. Two files means two transcripts, each already belonging to one person, which you interleave by start time.
Use two microphones and two tracks in a room. If your recorder writes two mono tracks, export them separately and transcribe each one. The other voice still bleeds into both, so label from whichever track is louder.
Leave a beat at each turn change. A deliberate half-second before answering gives you a visible gap between cues, and it keeps the two voices from overlapping.
Say the names at the top. Ten seconds of “I’m Alain, I’m here with Sam” makes the voices identifiable weeks later, to you or whoever edits the file.
Stop people talking over each other. Overlap is the one case labelling cannot repair, because the model writes a single stream of text for that window regardless.

Paying for a service that does diarization is sometimes the right call

Doing it by hand is fine for a 30-minute interview with two people. It stops being fine at some point, and that point is worth naming before you start.

Pay for a service that does diarization when you have more than two speakers, when hours of material arrive every week, when people interrupt each other constantly, or when speaker-attributed text is the thing you deliver to someone else. At that volume the hand method stops being cheap, and paying for speaker labels is the sensible answer.

Do it by hand when the volume is low, or when you would rather not hand an interview to a third party at all. FreeTranscribe covers that second case: it runs OpenAI’s open-source Whisper model on your own graphics card through WebGPU, so the audio never leaves the browser, with no account and no length cap. The limits matter here. It needs desktop Chrome or Edge with a working WebGPU adapter, it is English only for now, it runs the base English model, and it took about 1.5x real time on a desktop with a graphics card in our test, closer to the length of the recording on a thin laptop. It does not label speakers.

If you want the timestamps to work from, drop your interview on the home page and export it as SRT.

Frequently asked questions

Can I prompt the model into writing Speaker 1 and Speaker 2? Don’t count on it. The paper describes the output as caption text with start and end time tokens around it, and neither the paper nor the README describes any speaker-label output. Anything appearing in that shape is the model imitating a transcript format from its training data, not a judgement about who spoke.

Does a larger model do diarization? The README lists one set of tasks for the models it ships, from tiny through large and turbo, and diarization is not among them. Size changes how well the words come out, not what the model does.

The two voices sound completely different, so why can’t it tell them apart? Because telling them apart is not what it was asked to do. Nothing in the output carries any property of the voice, so a deep voice and a high one land in the transcript looking identical.

Should I export TXT or SRT for this job? SRT or VTT. TXT throws the timings away, and the timings are what make hand labelling quick. Strip them out at the end.

Sources, checked 15 September 2026

https://arxiv.org/html/2212.04356 - Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision”: section 2.1 (30-second segments), section 2.3 (diarization named as a separate component, the no-speech token, timestamp tokens interleaved with caption tokens, times quantised to 20 ms)
https://github.com/openai/whisper - openai/whisper README: the task list the models perform (multilingual speech recognition, speech translation, spoken language identification, voice activity detection), the model size table from tiny to turbo, no mention of diarization or speaker labels