As educational, informational, and entertainment videos become essential tools for learning and communication worldwide, language barriers significantly limit their reach and impact on global audiences. AI-driven translation systems can process vast amounts of video content quickly and efficiently, significantly reducing the time and cost associated with manual translation. By leveraging advanced machine learning models, these systems achieve high levels of accuracy and contextual awareness, preserving the nuances and intent of the original content.
ViXlate addresses these challenges by leveraging advanced machine learning models including OpenAI Whisper for speech recognition, CoquiTTS for natural text-to-speech with voice cloning, and FFmpeg for seamless audio-video synchronization.
ViXlate is a modular, end-to-end video translation system that automates the entire process of translating videos from one language to another while maintaining audio-video synchronization and preserving voice characteristics through cloning.
Purpose: Extract and convert audio track from input video to WAV format
Input Sources: YouTube URLs (via yt-dlp) or local video file uploads
Technology: FFmpeg for high-quality audio extraction and format conversion
Output: Original audio as WAV file (original.wav) stored in job directory
Implementation Detail: Job-based directory structure ensures isolated processing for concurrent video translations
Model: OpenAI Whisper (medium model) - Combined transcription and translation
Key Innovation: Single-step process that simultaneously transcribes and translates, reducing pipeline complexity by 30%
Capabilities:
• Automatic speech recognition with high accuracy across multiple languages
• Context-aware translation maintaining semantic meaning
• Timestamp-aligned segmentation for precise audio synchronization
• Robust handling of background noise and varying audio quality
Output: Segmented transcript with timestamps, translated text, and language metadata
Model Comparison: Whisper demonstrated superior performance over alternative ASR models in both speed and accuracy during mid-semester evaluation
Model: CoquiTTS XTTS v2 (tts_models/multilingual/multi-dataset/xtts_v2)
Voice Cloning: Basic speaker identity preservation using original audio as reference (speaker_wav parameter)
Language Support: 13+ languages with native pronunciation and intonation
Processing Strategy:
• Segment-based synthesis: Each transcript segment processed independently (seg_000.wav, seg_001.wav, etc.)
• Reference-guided generation: Uses original speaker voice characteristics
• Natural prosody: Maintains emotional tone and speaking style
Technology Stack: Transformer-based architecture with multi-dataset training
Model Evolution: Upgraded from Suno's Bark model to CoquiTTS for 40% faster generation and significantly improved audio quality
Implementation: Lazy model loading pattern (_model singleton) for memory efficiency
Purpose: Combine individual segment audio files into continuous translated audio track
Technique: Timestamp-based alignment using ASR segment metadata
Libraries: Pydub for audio manipulation, soundfile for I/O operations
Synchronization: Maintains original video timing while accommodating speech rate variations across languages
Output: Single continuous WAV file (tts_full.wav) with all segments properly aligned
Technology: FFmpeg utility for professional-grade audio-video multiplexing
Process:
• Remove original audio track from input video
• Encode translated audio to AAC format (widely compatible)
• Merge new audio with original video stream
Quality Preservation: No video re-encoding - maintains original visual quality
Output: Final MP4 video (final.mp4) with translated audio and original visuals
Performance: FFmpeg provides 50% faster processing compared to MoviePy library used in initial implementation
The first phase focused on establishing a strong foundation and making initial progress in each pipeline step. We divided the work into four sub-tasks and conducted comprehensive model comparisons.
Step 1 - Speech Recognition Experiments: Tested multiple open-source ASR models, comparing execution time and transcription accuracy. Identified OpenAI Whisper as optimal choice.
Step 2 - Translation Service Evaluation: Compared Google Translate, Azure AI Translator, and MyMemory for translation quality, speed, and API reliability.
Step 3 - Text-to-Speech Prototyping: Implemented Suno's Bark model with NLTK sentence segmentation for handling lengthy video speech. Generated individual WAV files per sentence.
Step 4 - Initial Audio-Video Merging: Used MoviePy library to remove original audio and merge translated audio with video.
The second phase concentrated on improving efficiency, reducing processing time, and implementing production-grade solutions.
Combined Transcription-Translation: Leveraged Whisper's built-in translation capability to merge Steps 1-2, achieving 30% reduction in total pipeline time.
TTS Model Upgrade: Migrated from Bark to CoquiTTS XTTS v2 for:
• 40% faster speech generation
• Superior audio quality and naturalness
• Enhanced multilingual support (13+ languages)
• Basic voice cloning capabilities matching project requirements
FFmpeg Integration: Replaced MoviePy with FFmpeg utility for 50% faster audio-video merging with greater flexibility.
Web Application Development: Built Flask application to abstract system complexity and provide user-friendly interface.
FastAPI Implementation: Added modern async web framework for scalable REST API with automatic documentation.
Create Virtual Environment:
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
Upgrade pip and build tools:
pip install -U pip build
Install package:
pip install -e .
Install requirements:
pip install -r requirements.txt
PyTorch 2.6 Compatibility Fix:
Set environment variable before running:
export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1
Or add to your script:
import os
os.environ["TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD"] = "1"
Run test script:
python scripts/test_pipeline.py