AI Audiobook Videos with Cartesia: Build a Fully Automated Pipeline with Gemini + Python
Creating an AI audiobook video usually sounds simple: generate a voiceover, create a few images, put everything together, and export the video.
Then reality arrives, carrying timestamps and 47 audio files.
In this tutorial, I walk through the complete workflow I used to turn a raw story into an expressively narrated, illustrated audiobook video with automated visual timing and video assembly using Cartesia, Google Gemini, Python, and FFmpeg.
The goal is to eliminate as much manual editing as possible, from preparing the narration script to assembling the final video.
Watch the Full Tutorial
Video duration: ~16 minutes 04 seconds
The video walks through the complete workflow from the original story to the finished audiobook video.
What You Will Learn
In this tutorial, you will learn how to build a workflow that can:
Convert a raw story into an expressive performance script
Add pauses, emotion, speed, and volume instructions using XML tags
Generate natural-sounding narration using Cartesia TTS
Use Gemini to analyze the story and generate visual prompts
Generate timestamped prompts for individual scenes
Organize AI-generated images automatically
Assemble images and narration into a synchronized video
Use Python + FFmpeg to automate the final video-generation process
The important part is that these steps can be connected into a repeatable pipeline rather than manually editing every video from scratch.
The Complete AI Audiobook Pipeline
The workflow can be summarized as:
Raw Story → Expressive Script → AI Narration → Visual Prompts → AI Images → Timestamp File → Automated Video
Each stage handles a specific part of the production process.
1. Start with a Raw Story
The process begins with an ordinary text story.
The raw text is readable by humans, but it is not necessarily suitable for expressive AI narration. A conventional text-to-speech system may read the entire story with relatively uniform pacing and emotion.
That can make even a good story sound surprisingly lifeless.
The first step is therefore to transform the raw story into a script designed specifically for expressive speech synthesis.
Convert the Story into an Expressive XML Script
Instead of sending plain text directly to the TTS system, the story can be structured using XML-style tags to provide additional performance information.
These instructions can be used to control aspects such as:
Pauses
Speaking speed
Volume
Emotional delivery
Changes in emphasis
For example, instead of treating a sentence as nothing more than a string of words, the script can contain instructions describing how those words should be performed.
This gives the narration considerably more control than a plain-text input.
DOWNLOAD XML PERFORMANCE-SCRIPT PROMPT
This template can be used with Gemini to transform a raw story into a structured performance script.
Generate the Expressive Narration with Cartesia
Once the performance script has been prepared, it can be imported into Cartesia's text-to-speech system.
In the tutorial, I demonstrate the settings used for the narration, including the selected voice and model configuration.
Tool used
Cartesia:https://cartesia.ai
The tutorial also demonstrates how to adjust parameters such as:
Voice selection
Speed
Emotion
Volume
Language
Model
Sample rate
The voice used in the demonstration is Cavatha.
🎙️ Cartesia Voice Used in the Tutorial
Voice: CavathaVoice ID: Kavith_Voice_ID: 01d7796d-ac10-4ea3-8df0-3cc04f2d25ff
Turn the Story into Visual Scenes with Gemini
Once the narration is generated, the next problem appears:
What should actually be shown on screen while the narration is playing?
Instead of manually deciding where every image should appear, Gemini can be used to analyze the story and generate a sequence of visual prompts.
The objective is to break the narrative into meaningful visual beats and associate those scenes with specific points in the narration.
This allows the workflow to produce a timestamped list of image prompts.
📥 Download the Gemini Visual-Prompt Template
This template is designed to help generate structured image prompts that correspond to different sections of the narration.
Generate the AI Images
The visual prompts generated by Gemini can then be passed to your preferred AI image-generation system.
The important part of this workflow is not the specific image generator. The prompts are designed to maintain continuity between the different scenes while giving each section of the story an appropriate visual.
The resulting images should then be named consistently according to the timestamp or scene structure produced by the workflow.
For example:
001.png
002.png
003.png
004.pngThis makes it much easier for the automation script to determine which image belongs to which section of the narration.
Create the Timestamp File
The narration and images need to be synchronized.
To accomplish this, the workflow uses a timestamp file containing the timing information for the visual scenes.
The timestamp data essentially tells the video-generation script:
Which image should appear, and when?
This removes the need to manually drag every image onto a video timeline.
Humanity spent decades inventing video editors and then immediately decided that someone should spend three hours dragging 80 still images onto a timeline. We can do better.
Assemble Everything Automatically with Python + FFmpeg
The final stage is handled by a Python script together with FFmpeg.
The script takes the required assets, including:
Narration audio
Generated images
Timestamp information
and combines them into a finished video.
The system can automatically determine how the images should be displayed in relation to the narration.
📥 Download the Python + FFmpeg Video Assembly Script
The script is intended to automate the repetitive parts of video assembly rather than requiring manual editing in software such as Premiere Pro or DaVinci Resolve.
How the "Fit Audio" Synchronization Works
One of the key parts of the workflow is making the visual sequence fit the duration of the narration.
Instead of treating the images as independent video clips with arbitrary durations, the script uses the timing information to control when each image appears.
The result is a video in which the visual sequence follows the narration automatically.
This is particularly useful when creating multiple audiobook videos because the same workflow can be reused with different stories.
Comments