Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning

Gunwoo Lee1, Yoori Oh2, Yoseob Han1

1Soongsil University  2Seoul National University

Overall Architecture

Abstract

Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance. To address this, we propose Watch Your Speech (WYS), a video-to-speech synthesis framework that incorporates textual conditioning as an explicit linguistic cue to mitigate visual ambiguity. Our framework features an attention-based embedding fusion module that synergistically integrates textual context with video sequences, coupled with a conditional flow matching objective for high-fidelity speech generation. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that WYS achieves superior performance, establishing new state-of-the-art results in audio-visual synchronization (LSE-C/D) while maintaining highly competitive textual accuracy (WER). Subjective evaluations further confirm that our model generates speech with near-human naturalness, validating the effectiveness of textual conditioning in content-controlled video-to-speech synthesis.

Comparison results on LRS3 dataset

Silent videoGround TruthVCA-GAN [1]SVTS [2]DiffV2S [3]IntelligibleL2S [4]LipVoicer [5]V2SFlow-A [6]V2SFlow-V [6]ProposedText
I love building robots
We can't see it most of the time
I mean how do they form
I tried everything
There were no back doors
But it's a global phenomenon

Comparison results on LRS2 dataset

Silent videoGround TruthVCA-GAN [1]DiffV2S [3]IntelligibleL2S [4]LipVoicer [5]V2SFlow-A [6]V2SFlow-V [6]ProposedText
It would have been Completely alien to the rest of london
So to do ballroom
And we were right
Some burn brightly for a year or two
That is a first
Something like a garden compost

[1] VCA-GAN: Lip to Speech Synthesis with Visual Context Attentional GAN
[2] SVTS: SVTS: Scalable Video-to-Speech Synthesis
[3] DiffV2S: DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding
[4] IntelligibleL2S: Intelligible Lip-to-Speech Synthesis with Speech Units
[5] LipVoicer: LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
[6] V2SFlow: V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow