← Back to Home

ViXlate

Video Translation using Deep Learning

Details

Project Type: Course Project
Course: CS 59000 - Application of Deep Learning
Duration: Fall 2024 Semester
Status: Completed
Focus Area: Multilingual Video Translation & Audio Synchronization
Collaborators: Al Shafayet Haque Silvy, Chirag Bellara, Rohit Ramkumar, Sai Suraj

Technologies & Models

Speech Recognition: OpenAI Whisper (medium model)
Translation Services: Whisper integrated translation, Google Translate, Azure AI Translator, MyMemory
Text-to-Speech: CoquiTTS XTTS v2 with voice cloning
Audio-Video Processing: FFmpeg, MoviePy, Pydub, soundfile
Web Framework: FastAPI, Flask, Uvicorn
Deep Learning: Transformers, TensorFlow, Keras, PyTorch 2.6

Problem Statement & Motivation

As educational, informational, and entertainment videos become essential tools for learning and communication worldwide, language barriers significantly limit their reach and impact on global audiences. AI-driven translation systems can process vast amounts of video content quickly and efficiently, significantly reducing the time and cost associated with manual translation. By leveraging advanced machine learning models, these systems achieve high levels of accuracy and contextual awareness, preserving the nuances and intent of the original content.

Key Challenges Addressed:

  • Language Accessibility: Educational and informational videos are often available in only one language, limiting their reach to non-native speakers
  • Cost & Time: Manual translation and voiceover work is expensive and time-consuming, making content localization prohibitive for many creators
  • Quality & Scale: Existing automated systems either produce poor-quality translations or cannot handle diverse content types and languages
  • Voice Preservation: Maintaining speaker identity and natural-sounding speech across translations requires sophisticated voice cloning capabilities
  • Synchronization: Merging translated audio with video while maintaining lip-sync and timing requires sophisticated technical solutions
  • Inclusivity: Non-English speakers have limited access to knowledge, entertainment, and educational content

ViXlate addresses these challenges by leveraging advanced machine learning models including OpenAI Whisper for speech recognition, CoquiTTS for natural text-to-speech with voice cloning, and FFmpeg for seamless audio-video synchronization.

Our Solution: Modular Pipeline Architecture

ViXlate is a modular, end-to-end video translation system that automates the entire process of translating videos from one language to another while maintaining audio-video synchronization and preserving voice characteristics through cloning.

Core Pipeline Workflow

1. Audio Extraction (demux_to_wav)

Purpose: Extract and convert audio track from input video to WAV format

Input Sources: YouTube URLs (via yt-dlp) or local video file uploads

Technology: FFmpeg for high-quality audio extraction and format conversion

Output: Original audio as WAV file (original.wav) stored in job directory

Implementation Detail: Job-based directory structure ensures isolated processing for concurrent video translations

2. Speech Recognition & Translation (ASR Service)

Model: OpenAI Whisper (medium model) - Combined transcription and translation

Key Innovation: Single-step process that simultaneously transcribes and translates, reducing pipeline complexity by 30%

Capabilities:

• Automatic speech recognition with high accuracy across multiple languages

• Context-aware translation maintaining semantic meaning

• Timestamp-aligned segmentation for precise audio synchronization

• Robust handling of background noise and varying audio quality

Output: Segmented transcript with timestamps, translated text, and language metadata

Model Comparison: Whisper demonstrated superior performance over alternative ASR models in both speed and accuracy during mid-semester evaluation

3. Text-to-Speech with Voice Cloning (TTS Service)

Model: CoquiTTS XTTS v2 (tts_models/multilingual/multi-dataset/xtts_v2)

Voice Cloning: Basic speaker identity preservation using original audio as reference (speaker_wav parameter)

Language Support: 13+ languages with native pronunciation and intonation

Processing Strategy:

• Segment-based synthesis: Each transcript segment processed independently (seg_000.wav, seg_001.wav, etc.)

• Reference-guided generation: Uses original speaker voice characteristics

• Natural prosody: Maintains emotional tone and speaking style

Technology Stack: Transformer-based architecture with multi-dataset training

Model Evolution: Upgraded from Suno's Bark model to CoquiTTS for 40% faster generation and significantly improved audio quality

Implementation: Lazy model loading pattern (_model singleton) for memory efficiency

4. Audio Alignment & Stitching (Aligner Service)

Purpose: Combine individual segment audio files into continuous translated audio track

Technique: Timestamp-based alignment using ASR segment metadata

Libraries: Pydub for audio manipulation, soundfile for I/O operations

Synchronization: Maintains original video timing while accommodating speech rate variations across languages

Output: Single continuous WAV file (tts_full.wav) with all segments properly aligned

5. Audio-Video Merging (mux_aac)

Technology: FFmpeg utility for professional-grade audio-video multiplexing

Process:

• Remove original audio track from input video

• Encode translated audio to AAC format (widely compatible)

• Merge new audio with original video stream

Quality Preservation: No video re-encoding - maintains original visual quality

Output: Final MP4 video (final.mp4) with translated audio and original visuals

Performance: FFmpeg provides 50% faster processing compared to MoviePy library used in initial implementation

System Architecture Highlights:

  • Modular Design: Each component (transcription, translation, TTS, merging) can be independently optimized and tested
  • Job-Based Processing: Isolated directory structure (storage_root/job_id) enables concurrent processing
  • Configuration Management: Environment-based settings (config.py) using Pydantic for type safety
  • Error Handling: Graceful failure recovery with detailed error reporting and job state tracking
  • Multi-Source Translation: Flexible fallback between Whisper, Google Translate, Azure AI, and MyMemory
  • Web Application: FastAPI/Flask dual implementation for REST API and user interface

Development Journey & Iterative Improvements

Mid-Semester Review: Foundation & Model Selection

The first phase focused on establishing a strong foundation and making initial progress in each pipeline step. We divided the work into four sub-tasks and conducted comprehensive model comparisons.

Phase 1 Achievements:

Step 1 - Speech Recognition Experiments: Tested multiple open-source ASR models, comparing execution time and transcription accuracy. Identified OpenAI Whisper as optimal choice.

Step 2 - Translation Service Evaluation: Compared Google Translate, Azure AI Translator, and MyMemory for translation quality, speed, and API reliability.

Step 3 - Text-to-Speech Prototyping: Implemented Suno's Bark model with NLTK sentence segmentation for handling lengthy video speech. Generated individual WAV files per sentence.

Step 4 - Initial Audio-Video Merging: Used MoviePy library to remove original audio and merge translated audio with video.

End-Semester Review: Optimization & Production Readiness

The second phase concentrated on improving efficiency, reducing processing time, and implementing production-grade solutions.

Phase 2 Enhancements:

Combined Transcription-Translation: Leveraged Whisper's built-in translation capability to merge Steps 1-2, achieving 30% reduction in total pipeline time.

TTS Model Upgrade: Migrated from Bark to CoquiTTS XTTS v2 for:

• 40% faster speech generation

• Superior audio quality and naturalness

• Enhanced multilingual support (13+ languages)

• Basic voice cloning capabilities matching project requirements

FFmpeg Integration: Replaced MoviePy with FFmpeg utility for 50% faster audio-video merging with greater flexibility.

Web Application Development: Built Flask application to abstract system complexity and provide user-friendly interface.

FastAPI Implementation: Added modern async web framework for scalable REST API with automatic documentation.

Technical Optimizations:

  • Segment-Based Processing: Parallel TTS generation for individual segments reduces memory footprint
  • Lazy Model Loading: Singleton pattern prevents multiple model instantiations
  • PyTorch 2.6 Compatibility: Resolved weights loading issues via TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD environment variable
  • Structured Logging: Implemented structlog for detailed pipeline monitoring and debugging
  • Type Safety: Pydantic models for configuration validation and API contracts

Performance Metrics & Results

Model Performance Comparison:

30%
Time Reduction (Combined Transcription-Translation)
40%
Faster TTS (CoquiTTS vs Bark)
50%
Faster Merging (FFmpeg vs MoviePy)
13+
Languages Supported

Key Findings:

  • Speech Recognition: OpenAI Whisper medium model demonstrated superior accuracy compared to alternative ASR models while maintaining reasonable processing speed
  • Translation Quality: Whisper's integrated translation provides context-aware results superior to standalone translation APIs for most use cases
  • Audio Quality: CoquiTTS produces significantly more natural-sounding speech than Bark with better prosody and emotional tone preservation
  • Voice Cloning: Basic voice identity preservation achieved using reference audio, though perfect voice matching remains challenging
  • Processing Efficiency: Segment-based approach enables parallel processing and better memory management for long videos
  • Synchronization Accuracy: Timestamp-based alignment maintains video-audio sync within acceptable tolerances

System Requirements:

  • Python: Version 3.10 or higher required for all dependencies
  • GPU: Recommended for faster TTS generation (CUDA support)
  • Memory: Minimum 8GB RAM, 16GB recommended for longer videos
  • Storage: Sufficient space for temporary audio files during processing
  • FFmpeg: System installation required for audio-video operations

Impact & Applications

Educational Impact

  • Enables global distribution of educational videos without language barriers
  • Provides students worldwide access to high-quality educational content in their native language
  • Supports learning and knowledge sharing across language communities
  • Reduces educational inequality caused by language barriers
  • Enables rapid localization of online courses and MOOCs

Content Creator & Media Applications

  • Content Localization: YouTube creators can easily translate videos to reach global audiences
  • Documentary Translation: Streaming platforms can automatically localize documentaries and educational series
  • Corporate Training: Companies can efficiently localize training videos for international offices
  • News & Broadcasting: News organizations can quickly translate breaking news for global distribution
  • Entertainment: TV shows and movies can be rapidly adapted for international markets
  • Marketing Content: Businesses can create multilingual promotional videos cost-effectively

Accessibility & Inclusivity

  • Promotes inclusivity by making content accessible to non-English speakers
  • Eliminates reliance on manual translation, reducing time and cost barriers
  • Enables smaller content creators to reach global audiences without expensive translation services
  • Supports cultural exchange and cross-linguistic communication
  • Preserves speaker voice characteristics through cloning, maintaining authenticity

Economic Benefits

  • Cost Reduction: Automated process significantly cheaper than hiring professional translators and voice actors
  • Time Savings: Videos can be translated in hours instead of weeks
  • Scalability: Can process unlimited videos without proportional cost increase
  • Market Expansion: Reduces barriers to international content distribution
  • Resource Efficiency: Minimal human intervention required, freeing resources for creative work

Technical Innovations

  • Pipeline Integration: Seamless combination of multiple deep learning models (Whisper, CoquiTTS) into unified workflow
  • Voice Preservation: Basic voice cloning capabilities maintain speaker identity across translations
  • Multi-Service Architecture: Flexible fallback options ensure robustness and reliability
  • Modular Design: Each component independently optimizable for continuous improvement
  • Job-Based Processing: Isolated execution enables concurrent video processing
  • Configuration Management: Environment-based settings provide deployment flexibility

Complete Technology Stack

Deep Learning & ML

  • Transformers 4.35.2 (Hugging Face) - Neural network architectures and model loading
  • OpenAI Whisper - Speech recognition and integrated translation capabilities
  • CoquiTTS 0.22.0 - Text-to-speech with voice cloning (XTTS v2 model)
  • TensorFlow - Backend for neural network operations
  • Keras - High-level neural network API
  • PyTorch 2.6 - Deep learning framework for model execution

Web Framework & Backend

  • FastAPI 0.114.2 - Modern async web framework for REST API
  • Flask - Lightweight web framework for user interface
  • Uvicorn 0.30.6 - ASGI server for FastAPI deployment
  • Pydantic 2.9.2 - Data validation and settings management
  • python-multipart 0.0.9 - Multipart form data handling for file uploads

Media Processing

  • FFmpeg - Audio-video manipulation, format conversion, and multiplexing
  • yt-dlp 2025.07.21 - YouTube video downloading and metadata extraction
  • Pydub 0.25.1 - Audio processing and manipulation library
  • soundfile 0.12.1 - Audio I/O operations for WAV file handling
  • MoviePy - Initial audio-video merging (later replaced by FFmpeg)

Translation Services

  • OpenAI Whisper - Primary translation via integrated ASR
  • Google Translate API - Fallback translation service
  • Azure AI Translator - Enterprise-grade translation option
  • MyMemory Translation API - Open-source translation alternative

Utilities & Infrastructure

  • structlog 24.4.0 - Structured logging for pipeline monitoring
  • NLTK - Natural language processing for sentence splitting
  • Redis - Optional job queue and caching (redis_url configuration)

Development Tools

  • Python 3.10+ - Core programming language
  • pip - Package management
  • venv - Virtual environment isolation
  • Git - Version control
  • GitHub - Repository hosting and collaboration

Setup & Deployment Instructions

Installation Steps:

1. Environment Setup

Create Virtual Environment:

python -m venv .venv

source .venv/bin/activate # On Windows: .venv\Scripts\activate

2. Install Dependencies

Upgrade pip and build tools:

pip install -U pip build

Install package:

pip install -e .

Install requirements:

pip install -r requirements.txt

3. Configure Environment

PyTorch 2.6 Compatibility Fix:

Set environment variable before running:

export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1

Or add to your script:

import os

os.environ["TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD"] = "1"

4. Test Pipeline

Run test script:

python scripts/test_pipeline.py

Configuration Options:

  • VIXLATE_STORAGE_ROOT: Directory for temporary job files (default: ./data)
  • VIXLATE_WHISPER_MODEL: Whisper model size - tiny/base/small/medium/large (default: medium)
  • VIXLATE_TTS_MODEL: CoquiTTS model path (default: tts_models/multilingual/multi-dataset/xtts_v2)
  • VIXLATE_REDIS_URL: Redis connection URL for job queue (default: redis://localhost:6379/0)

System Requirements:

  • FFmpeg: Must be installed and available in system PATH
  • GPU: NVIDIA GPU with CUDA support recommended for faster processing
  • Memory: Minimum 8GB RAM, 16GB recommended for production
  • Storage: Sufficient disk space for temporary audio files during processing