A video with poor audio

• In 1991, a bystander filmed Rodney King, a Black man, being beaten by several white Los Angeles police officers
• All four officers were charged and acquitted. At a later civil rights trial, two of them were sentenced to 30 months in prison. That conviction rested mainly on the pictures. The audio, with a police helicopter overhead, was considered too poor to be admitted

Two questions for a phonetician

Angelika Braun later analysed the soundtrack (Braun, 1994).

• Did the officers shout racial slurs, as King said? After filtering out as much of the helicopter noise as possible, she heard none
• Was a taser fired? King’s facial injuries came from falling, and the question was whether he fell because of a taser or was hit in the face with batons. She compared known samples of taser noise with the recording. Its spectrogram showed regular vertical striations typical of a taser, so a taser had probably made him fall

Today

1. Audio forensics and the chain of evidence
2. Authentication: has the recording been changed?
3. Enhancement: making speech easier to understand
4. Speech content determination: transcripts and questioned utterances

Covert recordings

Much casework involves covert recordings of people who have been bugged. The microphone may be hidden under clothing or behind an obstacle, people may talk quietly, and engines or music may cover the speech.

Who does what

• Audio forensics covers checking that a recording is authentic, enhancing it, working out what was said and who said it, and helping to reconstruct events
• Authentication and enhancement are technical, and usually done by audio engineers. Working out what was said needs linguistic and phonetic training
• Every step is documented in a chain of evidence: who received, copied, accessed or changed the material, and when. This protects the evidence from interference while it is with the police, lawyers or experts

Has the recording been changed?

• Authentication establishes whether a recording has been tampered with (edited), or created artificially, which is called spoofing
• Someone might remove incriminating words, or insert material. Authentication can also compare versions of a recording where nobody intended to deceive. The work is slow, and a skilled forger may leave no trace that an expert can find under time pressure

Critical listening

The expert listens many times, in a quiet room with good equipment, while looking at the waveform and spectrogram, and notes anomalies:

• sudden changes in background noise or in loudness
• clicks and other abrupt sounds
• odd syntax or breaks in the flow of conversation
• sudden voice breaks, which may point to a synthetic voice

Tape and digital files

• On analogue tape, the signal is a pattern on the magnetic surface, which special equipment can make visible. In the Yorkshire Ripper hoax case (session 3), this showed that the speech had been recorded over existing music
• Digital files have no physical form, but the electric network frequency (ENF, session 3) can show whether a recording has been edited
• For both, experts check level changes, such as voices suddenly getting louder, and the noise floor: the background sound of a recording, which normally stays fairly constant

A change in the noise floor

Simulated. The traffic noise drops abruptly for half a second and comes back just as abruptly, a sign that something may have been inserted or removed there.

Splices and metadata

• A splice copies a stretch of signal to another place. Done carelessly, it leaves a jump in the waveform that sounds like a very short click
• Digital files carry metadata: sampling rate, number of channels, recording device, date. Editing can leave traces there
• For example, a video said to have been filmed on an iPhone, which saves .mp4 or .mov files, turns out to be a Windows-format file
• Experts may also give an overall judgement on provenance: which equipment was used, in how many sessions, and on which dates

Spoofing

• Voices can now be synthesised with AI and used, for example, to fool banks that let customers use voice banking
• Journalists used an AI-generated voice to access information held by the Australian Taxation Office
• Some spoofed speech can still be detected acoustically: unnatural breathiness or creak, formants without clear definition, unnatural glottal striations, and a “robotic” quality (Kirchhübel & Brown, 2022)
• As synthetic voices improve, such cases are likely to become more common

Studio and forensic recordings

Adapted from Zjalic (2020), as in Leemann et al. (2024).

The golden rule

• Enhancement aims to improve intelligibility, the proportion of words a listener can identify, even when the overall quality of the signal suffers
• A disturbing noise stays in if removing it would make speech harder to understand. Every step is documented so that it can be repeated, and enhanced files are usually not used for acoustic measurements, because processing can change the properties of speech sounds

The CSI effect

• Television makes enhancement look like magic. Public expectations are distorted, which is sometimes called the “CSI effect”
• What the microphone never captured cannot be recovered, for example because the speaker was too far away. Speech completely masked by a passing truck, or several people talking at once, are close to impossible to resolve

Repair techniques

• De-clicking replaces a click by interpolating between the signal just before and after it. It can leave audible bumps, and a click that covers a whole speech sound cannot be undone
• When a signal was too loud for the microphone, the waveform is cut off at the top: clipping. De-clipping rebuilds the missing part from the waveform on either side
• Spectral repair removes a small area of the spectrogram, such as a dog’s bark, and fills it with material resynthesised from the surroundings

Removing noise

• Reference cancellation subtracts a known sound from the signal. A fire alarm can be profiled in a stretch without speech and then removed. If a suspect turns on the TV to mask a conversation, the TV sound can be partly cancelled, provided the original broadcast is available
• Filtering weakens certain frequencies. A low-pass filter can dampen hiss above about 7 kHz while letting most speech information through. Equalisation adjusts the level of frequency regions, for example to restore weak high frequencies
• Aliasing, from digitising at too low a sampling rate, adds mirror-image tones that sound a little like birdsong. Filtering removes them, but whatever was in those frequencies is lost

Loudness and reverberation

• Amplification can be applied by hand to quiet passages, or automatically with dynamic range compression, which raises quiet sounds and lowers loud ones. Background noise is amplified too
• Reverberation smears the energy of each sound over the following sounds, as in a dining hall or an indoor pool. De-reverberation needs the room’s acoustic response and the positions of speaker and microphone, and is very hard to do well

Order matters

• Enhancement is a sequence, for example a filter, then de-hum, then de-reverberation, and each step works on the output of the previous one. A different order can give a different result, so experts try several filters and orders on copies, and keep the one that gives the most intelligible speech
• Quality and intelligibility can pull apart: a recording full of crackle and hiss may stay intelligible because fricatives are still audible, and filtering out the hiss would remove the fricatives too

Hiss and fricatives

Simulated spectra. The filter removes the hiss above 7 kHz, and with it the upper part of [s]. The weaker [f] lies close to the level of the hiss across the whole range, so any attempt to remove the hiss also weakens [f].

Mishearing is normal

• Even in quiet, people confuse thirty and thirteen, F and S, I can do it and I can’t do it
• Telephones, mumbling, fast speech and background noise make this worse, and so do accent differences. Australian Kate [kʰɐ̟ɪt] can sound like kite to a British or American listener

Two kinds of task

Speech content determination, also called disputed or questioned utterance analysis, is the task of describing as well as possible what was said.

• A forensic transcript of a whole recording, mostly done by careful listening, helps a court follow the recording
• When lawyers dispute one word or phrase, a detailed acoustic analysis of that stretch may help

Transcripts are not neutral

• Listening is subjective. What a transcriber hears can depend on what they know about the case and on the instructions they were given, and code-switching, unfamiliar accents, unclear articulation and poor recordings mean that different parties may hear different things
• The police may be convinced a suspect is guilty, and produce a transcript that differs from an expert’s

Safeguards

• The analyst first works without knowing the purpose of the case. Ideally an assistant prepares the material and talks to the client, and passes on no context
• Information can be revealed later to see whether it changes what the analyst hears. This matters most for names and places, which are especially prone to priming
• One analyst can transcribe the material several times, or several analysts can work independently and compare. Sometimes a court decides that the content is too uncertain to be admitted

Discussion

A detective has listened to a covert recording dozens of times and knows the suspects’ voices well. A phonetician has heard it twice and knows nothing about the case.

Whose transcript would you rather give a jury, and what could each of them get wrong?

Techniques

• The linguistic context helps. In I was talking last night to a σ, the unclear syllable σ is almost certainly a common noun. Vowels, intonation and rhythm survive noise better than consonants, so the number of syllables is often clear
• Clearer instances of the same word elsewhere in the recording help with the unclear ones. Because of this cross-referencing, transcripts are built up out of order, going back and forth
• Consultants help with particular languages and dialects, specialist vocabulary, or speech disorders

What goes into the transcript?

• Real speech has false starts, repetitions, pauses, overlaps and slang. How much should a transcript for jurors show?
• Uncertain passages are usually marked, for example in brackets, and alternatives are given side by side. Overlap is shown only where it matters
• Words and grammar are kept as spoken (he weren’t), but spelling is standard: a dropped /h/ or [f] for /θ/ is not usually shown
• Recordings in several languages raise questions of translation as well

Format of such a transcript

Invented example in the format of Foulkes et al. (2019). English in bold; M: a male speaker who could not be identified; brackets: lower confidence; …: speech that could not be transcribed.

One word in dispute

• Sometimes one phrase, word or even one sound matters in a case. There may be no agreed version, or two competing ones
• The phonetician analyses the questioned stretch acoustically, ideally comparing it with clear examples of the candidate words from the same recording, and with what is known about the speaker’s accent

Priming

• Context changes what we hear. A transcript, or a suggestion from one side, can make listeners lock in to one interpretation: this is priming
• Misheard song lyrics are a harmless example: once someone has told you the nonsense version, it is hard to hear anything else. It matters most when jurors are given a transcript of a poor recording

Before we go on: activity 6A

In the next case, the recording itself is the point. If you want to try it, listen before reading any further.

• The full emergency call is on Helen Fraser’s website (forensictranscription.net.au), and the questioned section is at about 30–35 seconds. The speaker is very distressed, and the recording is upsetting to listen to
• Write down what you hear, if anything. Listen several times: does your version change?

The Bain emergency call

• New Zealand, 1994: David Bain called emergency services after finding his family shot dead at home. He was convicted of the murders and sentenced to life imprisonment
• In 2007 the Privy Council quashed the convictions. For the retrial, held in 2009, a detective claimed to hear a confession in the call: “I shot the prick”. No one had claimed this at the first trial
• Experts for both sides agreed that no definite transcript was possible. Some had written “I can’t breathe” in drafts, all with caution; none heard anything like a confession

Keeping it from the jury

• Both legal teams agreed that the phrase should not appear in any transcript given to the jury, because of the risk of priming
• The defence also asked for that section to be cut from the recording played in court. This was refused at first, then granted on appeal. The retrial jury found Bain not guilty. It is not clear what part the recording played

The priming experiment

Fraser, Stevenson & Marks (2011) played the call to listeners.

• Without any suggestion, almost no one heard “I shot the prick”
• When it was suggested, 30% heard it
• After they were told that the recording was too unclear to support it, half of those still heard it

“Dom” or “Tim”?

• Sweden, 2008: in a police interview with a witness in a murder case, one word was disputed. Was it dom (‘they’, [dɔm]) or the name Tim ([tʰɪm])?
• The witness would not make further recordings, but the interview contained many clear tokens of both words. The experts measured voice onset time and vowel formants in every token, three times each, and compared the disputed token with the known ones (Morrison, Lindh & Curran, 2014)

Voice onset time

Simulated, following the pattern in Morrison et al. (2014). Tim has a long positive VOT, dom a negative one (prevoicing). The disputed token falls among the dom tokens, and so did its vowel formants: the witness most likely said dom.

Discussion

• Should juries ever be given a transcript of a recording that experts cannot transcribe with confidence?
• In the Murdaugh case, the court let witnesses say what they heard. What should have happened instead?

Activities

For the practical sessions (book activities 6A–6C):

• 6A: the Bain call, before and after reading about the case
• 6B: transcribe a sentence mixed with train noise at signal-to-noise ratios from −10 dB to +5 dB, and without noise. Where does it become intelligible?
• 6C: look for evidence of tampering in a recording in Praat, and describe what you find with exact time points

Summary

Slides and code

These slides

mcanzi.de/ws27_fss/06_authentication_enhancement.html

Code and all sessions

github.com/mcanzi/ws27_fss