NeurIPS 2026

Speech Tokenizers are Vulnerable

Transferable Semantic Attack and Robust Tokenizer

Zhisheng Zhang1,†, Yifan Mi2,†, Yixuan Zhou1, Haiyun Li1, Shengbo Cai1,
Dongfu Song1, Ziang Xu1, Jie Hao2,‡, Zhiyong Wu1,‡

1 Shenzhen International Graduate School, Tsinghua University
2 Beijing University of Posts and Telecommunications

† Equal contribution   ‡ Corresponding authors

Abstract

Speech tokenizers encode speech into representations used by automatic speech recognition (ASR) systems and large audio-language models (LALMs). Small waveform perturbations can disrupt this semantic encoding process and severely degrade downstream performance. We introduce T-SemAttack, a transferable semantic attack based on an ensemble of semantic encoders. It jointly perturbs their representation spaces while using psychoacoustic constraints to preserve perceptual quality.

Our analysis connects attack transferability to layer-wise representation drift and cross-model gradient alignment. Building on these findings, we introduce ROSETok, a robust speech tokenizer trained with noise augmentation and semantic distillation, followed by attack-aware fine-tuning. Experiments span multiple datasets, nine open-source and black-box ASR systems, and four LALMs. Explore the audio examples below and download the ROSETok-Base and ROSETok-AAF checkpoints.

Contents

T-SemAttack Workflow

T-SemAttack workflow

Figure 1. The architecture and pipeline of T-SemAttack and ROSETok.

Original and Attacked Speech

Original samples and adversarial counterparts produced by different attack/defense methods. The original transcripts are shown below.

Sample 1: "You see how doubly, how intimately, opposed the ideas are; yet how difficult to explain without apparent contradiction."
Sample 2: "At sight of this castle or cottage in the air, Rachel lighted up."
Sample 3: "Fairview was twelve miles away, but by ten o'clock they drew up at the county jail."
Method Sample 1 Sample 2 Sample 3
Original
Gaussian Noise
E2E-VGuard
AudioShield-0.1
AudioShield-0.3
T-SemAttack (Ours)

T-SemAttack Performance on ASR Systems

Two samples from the Common Voice corpus are presented. For each sample, we transcribe the audio with four ASR systems: HuBERT, Whisper, VibeVoice, and GPT-4o-transcribe.

Sample 1

Reference: "There's no danger, the boy said, when they had moved on past the encampment."
Method Audio HuBERT Whisper VibeVoice GPT-4o-transcribe
Original There's no danger, the boy said, when they had moved on past the encampment. There's no danger, the boy said, when they had moved on past the encampment. There's no danger, the boy said, when they had moved on past the encampment. There's no danger, the boy said, when they had moved on past the encampment.
Gaussian Noise there is no danger the boys said when they had moved on past the encampment there is no danger the boys said when they had moved on past the encampment there is no danger the boy said when they had moved on past the encampment there is no danger the voice said when they had moved on past the encampment
E2E-VGuard there is no danger the boy said when they had moved on past the encampment there is no danger the boy said when they had moved on past the encampment there is no danger the boy said when they had moved on past the encampment there is no danger troy said when they had moved on past the encampment
AudioShield-0.1 there is no danger the boy abserved when they are move our past the encampment there is no danger the board said when they had moved on past the encampment there is no danger the boys were out when they had moved on past the encampment there is no danger the voice read when they had moved on past the encampment
AudioShield-0.3 i was now the darn she and the poindser onvar whil we pad tair care unli i was not a giant shallot at the point where it started when we got it to i was not a director i was a producer i am the arti will probably pass that I was not a dire. the 1st decade and i was at yeah i will do it
T-SemAttack (Ours) radmission on the great enguramp witharodate larapoifonly nigenantonsl cargti have stayed witt and in my bill the 1st time i got injured i put the lights there i put the playmate at the little dog i have seen in a can of rum there is no danger but what is that like i had a little dog i stayed in camp and i am there is no danger but what is that if we may have no dog i have seen can

Sample 2

Reference: "100s must have seen it and taken it for a falling star."
Method Audio HuBERT Whisper VibeVoice GPT-4o-transcribe
Original 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star
Gaussian Noise 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star 100s must have seen it and taken it for a falling star
E2E-VGuard hinjech must have seen it and taken it for a falling star the pringles must have seen it and taken it for a falling star i am sure you must have seen it the thing taking that poor falling star 100s must have seen it and taken it for a falling star
AudioShield-0.1 100s must a seen et an taken at for wholl our star 100s must have seen it and taken it for a falling star 100 plus a student and taking it for a whole last star 100s must have seen it and take about 4 fall and star
AudioShield-0.3 100s motes of tan oand tide rect for iallavyl star 100s most associated activity around the world 100 most listened to in a 2 year at 4th of all annual scholar 100s of years to go.
T-SemAttack (Ours) crameho ten not be gemgaci poliviry eithenif ielenaiby any toonany i am going to tell her to leave and i am going to take me down to the forest and i am going to kill her 欢迎收听本期节目我是陈1 i remember it must have seen it i am shaking that poor pauline star

T-SemAttack Performance on LALMs

Two samples are evaluated against three Large Audio-Language Models. The transcribed responses for each LALM are reported below.

Sample 1

Reference: "several people are walking out in the spotlight."
Method Audio Qwen2-Audio Qwen3-Omni Gemini
Original several people are walking out in the spotlight several people are walking out in the spotlight several people are walking out in the spotlight
Gaussian Noise several people are walking now in the spotlight several people are walking now in the spotlight several people are walking out into the spotlight
E2E-VGuard so that young people are working now in this factory several people are walking now in this spot several people are out now in this parking lot
AudioShield-0.1 some people stay up all night some of them been home or walked my hall in the sport night so i am super hungry but i do not want to walk all the way to westfield mall
AudioShield-0.3 the audio states they would walk through open doors i walk now i must walk now but it is full of people walking up and down this block
T-SemAttack (Ours) several people are working now on this problem so there are people i walk now from this spot the little boat on water now from this boardwalk

Sample 2

Reference: "salmon is one of the most popular fish and very delicious though usually not sustainable."
Method Audio Qwen2-Audio Qwen3-Audio Gemini
Original salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable
Gaussian Noise salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable
E2E-VGuard salmon is one of the most popular fishes and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable salmon are one of the most popular fish and very delicious though usually not sustainable
AudioShield-0.1 salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable salmon is one of the most popular fish and very delicious though usually not sustainable
AudioShield-0.3 some of the most famous doctors were graduates from these schools though usually not students themselves some of this one of the most popular and huge regulations obviously not sustainable now unemployment is up labor force is down and this is clearly not sustainable
T-SemAttack (Ours) the unemployment rate among young people is alarmingly high and youth unemployment has been steadily increasing for years salmon is one of the most popular consumed fish in the united states though it is usually not sustainable some employment conditions apply and fees are charged if you fail to meet some eligibility requirements such as staying employed for at least 12 months

Reconstruction Performance of ROSETok

Reconstruction quality of seven speech tokenizers under varying attack budgets (clean, 4/255, 8/255, 16/255). One English sample and one Chinese sample are randomly selected from SeedTTS.

English Sample

Tokenizer clean 4/255 8/255 16/255
SNAC
XY-Tokenizer
X-Codec2
DAC
MOSS
Stable Token
ROSETok (Ours)

Chinese Sample

Tokenizer clean 4/255 8/255 16/255
SNAC
XY-Tokenizer
X-Codec2
DAC
MOSS
Stable Token
ROSETok (Ours)

Case Study: TTS for Protection

Speech generated by CosyVoice and Step-Audio-TTS using clean references and references protected by T-SemAttack. The synthesized voice is hard to clone when T-SemAttack is applied.

Model Reference Sample 1 Sample 2 Sample 3
CosyVoice clean
T-SemAttack
Step-Audio-TTS clean
T-SemAttack