Adding In-Browser Auto Captions to a Privacy-First Tool
Convert to Shorts, a web application designed to convert horizontal videos to the 9:16 YouTube Shorts format without server uploads, has introduced an auto-captioning feature. This addition directly addresses user requests and aims to improve the engagement of Shorts, which are often watched without sound. The core challenge was implementing this feature while maintaining the tool's privacy-first, serverless architecture.
The solution leverages OpenAI's Whisper AI model, specifically a smaller, optimized version, and runs it directly within the user's browser. This approach ensures that no video or audio data ever leaves the user's device, preserving the application's commitment to privacy. The integration was made possible through a combination of technologies, including Transformers.js, the Web Audio API, and ffmpeg.wasm.
Technical Implementation: Whisper AI in the Browser
The integration relies on Transformers.js (@xenova/transformers), a JavaScript port of Hugging Face's popular Transformers library. This library enables the execution of ONNX models directly in the browser using WebAssembly. For the automatic speech recognition (ASR) task, the team selected the Whisper tiny model. Despite its relatively small size (around 75MB), this model offers surprisingly accurate transcription for clear audio input. The choice of the 'tiny' model is a critical balancing act: it provides a good compromise between accuracy and performance within the resource constraints of a browser environment.
To extract the audio from the video file, the Web Audio API is utilized. This API allows for the manipulation and processing of audio streams within the browser. It's used here to access the video's audio track, resample it to the required sample rate for Whisper, and prepare it for transcription. This entire process occurs client-side, meaning the user's video file is loaded, its audio extracted, processed, and then transcribed by Whisper AI, all without ever being sent to a remote server.
The workflow begins when a user uploads a video. The application first processes the video using ffmpeg.wasm to extract the audio stream. This audio is then passed to the Whisper AI model, also running via Transformers.js in the browser. The model analyzes the audio and generates a text transcript. This transcript is then formatted into a standard caption file, such as a WebVTT (.vtt) file, which can be directly used by YouTube Shorts or other video platforms. The user can then download the converted video along with the generated caption file.
Advantages of the Client-Side Approach
Running Whisper AI directly in the browser offers several significant advantages. Firstly, it provides a robust privacy guarantee. Users can be confident that their video content is not being uploaded or processed on external servers, which is a major concern for many users, especially in professional or sensitive contexts. Secondly, it eliminates server-side costs associated with ASR processing. For a free tool, this is crucial for sustainability. By offloading the computational load to the user's device, the operational expenses are significantly reduced.
However, this approach is not without its challenges. The performance of the transcription is directly dependent on the user's device capabilities. Older or less powerful machines may experience slower processing times. Additionally, the accuracy of the 'tiny' Whisper model, while good, is not as high as larger, server-based models. It performs best with clear audio and minimal background noise. Future improvements might involve offering options for users to select different Whisper model sizes if their browser and device can handle them, or perhaps integrating with cloud-based ASR services as an optional, opt-in feature for those who prioritize speed and accuracy over absolute privacy.
The integration of Whisper AI into Convert to Shorts demonstrates a powerful pattern for bringing advanced AI capabilities to the web without compromising user privacy or incurring significant infrastructure costs. This approach is applicable to a wide range of applications that require audio or natural language processing, from video editing tools to accessibility features and real-time communication platforms.