I tried dubbing my natural voice with text-to-speech using Python (pyannote.audio x whisper x VOICEVOX x OpenCV x SpeechRecognition x wave x FFMPEG)

Hello, this is Rcat.
This time, this is an explanatory version of the work introduced in this article.
Specifically, it is a tool thatautomatically converts videos recorded with your natural voice into narrated text-to-speech.Python is amazing; it has so many libraries, and you can do anything depending on how you combine them.
Let's take a look at what's inside.

Introduction

Terms of Service

Please check the terms of service in advance when using information or works.

https://note.com/rcat999/n/nb6a601a36ef5

About comments

Please check the guidelines in the terms of service before commenting.

Mechanism

First, I will explainwhat elements are included in this work.It is also okay to look only at the parts you are interested in.
The specific processing steps are as follows.

Overall processing steps

The items written in parentheses at the end are the libraries and software used.

1. Detach audio from video (FFMPEG)
   Obtain the natural voice audio data required for dubbing.

2. Audio preprocessing (FFMPEG)
   Perform noise reduction to improve transcription accuracy.
   There is a significant difference with or without it.
   I introduce how much of a difference there is here. Please use it as a reference.
   https://note.com/rcat999/n/n83bdd0d151ff

3. Speaker diarization of audio (pyannote.audio)
   Obtain the pronunciation intervals for each word. Information on which words should be spoken in how many seconds.

4. Audio data splitting (wave)
   Decompose the audio data for each pronunciation interval.

5. Transcription (whisper or speech_recognition)
   Transcribe each interval using the split audio data.
   Use whisper if transcribing on your own computer.
   Use speech_recognition if transcribing online.

6. Proofreading by LLM (Dify & Gemini)
   Text-to-speech software will automatically read text that way if you put a dictionary in it, but entering it manually doesn't feel very modern, does it?
   Let's use the power of generative AI for this.

7. Text-to-speech (VOICEVOX)
   Read the transcribed text using VOICEVOX.

   1. Speed adjustment
      Since the pronunciation speed differs depending on the character, it is generated while adjusting the speed so that it fits within the pronunciation interval.
      By doing this, we ensure consistency with the natural voice.

8. Concatenation (wave)
   Concatenate the files created by the text-to-speech into a single file.
   Specifically, an empty wav file is generated and overwritten according to the pronunciation intervals.

9. Merge (FFMPEG)
   Once re-attached to the original video data, the dubbing is complete.

   1. Adding subtitles (OpenCV & Pillow)
      Since we have gone to the trouble of transcribing, we will use OpenCV to write subtitles onto the video. Since it does not support Japanese, we will supplement that part by using Pillow.

Introduction to the libraries and software used

• Software

  • FFMPEG
    A tool that can process video and audio files.
    In this project, I am using it to separate audio from video, perform noise reduction, and convert file formats.

  • VOICEVOX
    Text-to-speech software.
    It includes over 30 characters in a single piece of software, allowing for reading aloud with a variety of voices.
    Since it is essentially a web server, it is very helpful that I can call the API from Python. I'm not sure if it's due to recent updates, but it feels like the generation speed has become much faster.
    Official site: https://voicevox.hiroshiba.jp/

• Libraries

  • pyannote.audio
    This is a library for speaker diarization.
    Speaker diarization appears to be a technique for analyzing who spoke when in an input audio file without performing transcription.

  • whisper
    This is an AI for speech recognition created by OpenAI.
    We input the live commentary audio here to convert it into text. A powerful GPU is required.

  • SpeechRecognition
    This is also a speech recognition library.
    It seems to support various services, but for now, I am using Google as it seems simple. Since it can convert online, a feature is that transcription can be done even on low-performance computers.

  • wave
    This is a standard library used for reading and writing wav sound files.
    In this project, it is used to split the audio into segments based on the information obtained from speaker diarization, and finally to merge them into a single file.

  • OpenCV
    This is the well-known video editing library.
    We use this to read frames from the video and write subtitles onto them.

  • Pillow
    This is also a well-known image editing library.
    We use it this time to embed Japanese text into images.

Regarding the entire source code

Naturally, I will not explain the entire source code of over 1000 lines.
Here, I will only explain the key points.

The entire source code is distributed in this article, so if you want to know the details of the relationships between functions and classes, please download it.

Step 1: Separate audio from video

The start of this project is obtaining audio data.
First, we assume we are processing a video containing natural voice recorded using the quickest method: "recording while live streaming."
Therefore, we first need to separate the audio from the video.

Source code

This is the source code for separating the video.
There were some comments for notes included, so please pretend you didn't see those.

That said, what it does is simple, and it is done in 5 lines starting from line 620.
First, I am trying to launch an external program using the subprocess library. The program being launched is ffmpeg.

It looks like this in a normal command line

'ffmpeg -i 入力データ -q:a 1 -map a -f wav - -map 0:v ビデオの出力先'

This content means separating the input video into audio and a silent video.
Also, as a feature,the audio output destination is set to standard output.
By doing this, you canhandle data with byte IOwithout having to go through files every time.
By the way, with this method,the data will have 8 bytes that are strange, so be careful about that. Click here for details.

Step 2 Preprocessing

Source code

Next is the noise reduction process for preprocessing.
Pass the byte IO data from the previous process to remove noise.

That said, what I'm doing is the same as last time, using FFMPEG to apply noise reduction.
Please check the following article for noise reduction commands, types, and effects.

The feature here is that the file input has also been set to standard input.
In the first step, it was still a video file, so I was reading the file, but since it has already been imported into byte IO, I am starting it by using that as standard input. Of course, the output is also byte IO.

Data error correction

Next, I willcorrect the problem that only occurs when using standard outputlike this time.
Specifically, it is a phenomenon where 0xFF is written to the area where the file size is written in the data. I thought it would be fine because it can be played as is, but it resulted in an error when I tried to perform additional processing using the wave library.
I introduce this in detail in this article.

Step 3 Perform speaker analysis

Once the audio data processing is complete, the next step is to detect the segments where speech is occurring.
You can transcribe the file as is, but just reading it out will cause the timing to be off from the video.
Therefore, I obtain information in advance about how many seconds each utterance starts and how long it lasts.
The whisper library outputs this kind of information at the same time as transcription, but since it may only return it in 1-second units, I decided to perform the analysis separately.

Also, as a plan, I am thinking that if I use the information of speaker diarization, that is, who is speaking, it will be possible to read out with multiple characters.

Source code

First is the first half.
This part is almost the same as the source code written in the official documentation.
I wanted to pass data using byte IO, so I also included reading in advance.

The Hook at the bottom is used to display progress. If it's short, it's fine, but if it's long, you won't know when it will end, so I think it's better to have it.
Also, regarding this library, you can provide information in advance about how many people are speaking.
In this case, the processing becomes overwhelmingly faster, so it is better to specify it if possible. Especially since this live commentary is just one person talking, you can just enter 1.

Next is the second half.
This is the stage where the analysis results are received and organized.

To use it later, I am temporarily reorganizing the data into a dictionary.
The information obtained is: who spoke, when they started speaking, when they finished speaking, and how many seconds they spoke.
Also, regarding who spoke, for some reason it comes out as a string, so I am using regular expressions to convert it to numbers only.

Other than that, I don't really understand, but for some reason there are times when it is judged as speaking for a 0.1-second interval, and of course, it cannot be transcribed or read out, so I have ignored such intervals.

Step 4 Divide audio according to segments

Now that the analysis is done, the next step is to divide the data in advance to ensure consistency between the analysis and the transcription.
After division, it will look like the following.

Source code

Parts not directly related to the current process have been blacked out. These are necessary for the overall operation of the tool, so please download it if you are interested.

The audio processing begins at line 539.
What is being opened here is the original audio file containing only the natural voice.
Next, it is divided using a For loop, but what is being looped through is the segment information obtained previously.
Within that, the process of calculating frames from the segments and extracting and saving from which frame to which frame is repeated. By the way, if you haven't performed the data error correction done above, an error will occur here.

Step 5: Transcribing with speech recognition

Now that the individual audio data is ready, it is finally time to transcribe the text.

This is the main function for transcription. Once again, irrelevant parts are hidden for clarity.
Since I have made it possible to select between two libraries this time, it starts with a branch.

Source code main

Source code for whisper

For whisper, which performs transcription on your own computer, it looks like this. Honestly, it's just comments.
I made it into a class mainly for notes and to manage GPU usage. Also, since the model can be kept loaded in the instance, it makes subsequent work easier.

First, the AI is loaded during instantiation.
At this time, you can specify the model size, so please decide based on the accuracy and the performance of your own computer. The types of models and required performance are written in the comments.

And then, text generation.
Here, I have made it a function that returns text when a wav file path is passed. By the way, there are many comments written, but these are the return values that whisper sends back.
This is the reason why I extracted the "text" key in the main function.

Source code for speech_recognition

This is the source for when transcribing online.
This is also just the source that often comes up if you search for it. The only difference is that it uses byte IO, perhaps?

A major feature is the process of adding a 1-second silent segment at the end before sending it for recognition.
I don't know if this is due to the speech_recognition library or a behavior specific to the recognize_google method, but for some reason, the end cannot be recognized as a specification.
Therefore, if you input a file cut exactly to the spoken audio segment, naturally the last few characters will not be transcribed.
Since that is problematic, I add a 1-second silent segment before sending it.

Also, the format when returning data is matched to that of whisper.
Well, since it only reads text, it's just making it a dictionary with a text key.

Step 6: Proofreading

Not implemented

Step 7: Reading aloud

Once the transcription is done, it is finally time for reading aloud.

Source code main

This is the main function for performing speech synthesis. It is named Step 3. Irrelevant descriptions are also hidden here for the time being.

First, we start by calculating the speed at which the character speaks. This is because the speaking speed varies significantly depending on the character.
Based on this information, we estimate the time required for reading based on the character count and set a speed multiplier that fits within the pronunciation interval.
Next is the creation of the data for speech synthesis.
First, we create the data for speech synthesis using a dictionary. This includes information such as the text to be read, which character to use, the speed multiplier, and the target audio length.

Finally, we execute the speech synthesis function to create the audio data.

By the way, since the VOICEVOX module itself is separated, it is imported separately.

Source code VOICEVOX

The speech synthesis function called from the main function is as follows.
Since this function assumes batch generation, it is necessary to input the speech synthesis data in dictionary format.
As written in the comments at the top, the main function was creating data to be input in this format.

By the way, for the sake of batch generation here, I am using a method where the audio file is saved and its path is returned.
There is a variable called 'report', which contains data including information such as the audio file generated by VOICEVOX.

By the way, the contents of the report look like this.
It includes where it was saved, the length of the audio, the content, and the difference from the target.

Source code VOICEVOX speech synthesis request related

Next, let's look at how the generation is specifically done.
The function used for generation is the following.
There are two steps to speech synthesis using VOICEVOX.
The first is text analysis and creation of data for speech synthesis. You might want to check the official API reference for this part, but first, we analyze the text to be read and convert it into symbols for speech synthesis.

Specifically, it is a mechanism where the necessary information is returned by sending the text you want to read to the specified URL via POST.

You might not understand it well, so let's look at the actual response.
Since this module can also be executed manually, executing it yields the following results (if the comments on the print lines are removed).

import voicevox as V
v = V.VOICEVOX()
v.CreateWave("ねこかわいい",Cid=8,Play=True)

{
  "accent_phrases": [
    {
      "accent": 1,
      "is_interrogative": false,
      "moras": [
        {
          "consonant": "n",
          "consonant_length": 0.046383969485759735,
          "pitch": 5.892535209655762,
          "text": "ネ",
          "vowel": "e",
          "vowel_length": 0.0923309400677681
        },
        {
          "consonant": "k",
          "consonant_length": 0.05872572213411331,
          "pitch": 5.988620758056641,
          "text": "コ",
          "vowel": "o",
          "vowel_length": 0.08133900910615921
        }
      ],
      "pause_mora": null
    },
    {
      "accent": 3,
      "is_interrogative": false,
      "moras": [
        {
          "consonant": "k",
          "consonant_length": 0.052951984107494354,
          "pitch": 5.742798328399658,
          "text": "カ",
          "vowel": "a",
          "vowel_length": 0.07227755337953568
        },
        {
          "consonant": "w",
          "consonant_length": 0.04524584859609604,
          "pitch": 5.88362979888916,
          "vowel": "a",
          "vowel_length": 0.06538953632116318
        },
        {
          "consonant": null,
          "consonant_length": null,
          "text": "イ",
          "vowel": "i",
          "vowel_length": 0.15184266865253448
        },
        {
          "consonant": null,
          "consonant_length": null,
          "pitch": 5.94594669342041,
          "text": "イ",
          "vowel": "i",
          "vowel_length": 0.11915982514619827
        }
      ],
      "pause_mora": null
    }
  ],
  "intonationScale": 1.0,
  "kana": "ネ'コ/カワイ'イ",
  "outputSamplingRate": 24000,
  "outputStereo": false,
  "pauseLength": null,
  "pauseLengthScale": 1.0,
  "pitchScale": 0.0,
  "postPhonemeLength": 0.1,
  "prePhonemeLength": 0.1,
  "speedScale": 1.0,
  "volumeScale": 1.0
}

Looking at this, you can see that it contains the detailed information necessary for pronunciation.
Generating this information is step 1.

Next is changing the options. Can you see the notations 'speedScale' and 'prePhonemeLength' at the bottom?
These are used to specify the speaking speed and the silent interval of the output file, respectively.
We adjust these to match the length of the file. Naturally, there is no silent interval.
This is done by directly editing the dictionary keys.

Next is the latter part of the speech synthesis.
Here, by POSTing the speech synthesis information generated earlier, we can obtain the binary of the synthesized audio.

What we do is not that difficult; we just attach the information edited earlier to the request body and POST it.
However, since there is a possibility of generating the same text multiple times, I have separated the request part into a function.
This is because no matter how much you can calculate the approximate speed from the number of characters, the overall length of the speech will inevitably deviate. It is natural if there are many kanji, only hiragana, or a mix of English.

Therefore, in the loop inside this, I am getting the length of the synthesized data and checking whether it matches the length specified by the target.
If it does not match, I check how much it deviates, adjust the speed, and then try again.
By the way, since it will never match perfectly, a length difference of up to 10% is allowed by default.
Well, even so, there are times when it doesn't match, and if it's impossible after recalculating about 6 times, it's usually impossible, so it stops at 6 times. I will leave the detailed calculation part for download.

We perform speech synthesis by following these steps.

Step 8 Concatenation

Now, once the synthesized audio is ready, the next step is to concatenate everything and return it to a single audio file.

Source code main

This is the main function. Irrelevant parts have been omitted here as well.

Here, we first instantiate a dedicated class to create an empty file.
After that, we sequentially insert the read files while reading the report. The report referred to here is the initial speaker analysis. At that time, we obtained information about at what second the speech started right? We use that here.

Source code: Empty WAV insertion class

This is the insertion class I created.
When instantiating, we determine the total length and create empty data.
At this time, it is necessary to pass the file along. This is to match the format with the data to be inserted later and to obtain that information.

Once instantiation is complete, you can use Insert to sequentially insert files at specified seconds.
After inserting all frames, you can save them using the wave library to combine them into a single file.

With this, the creation of the dubbed audio data is complete.

Step 9: Recombination with video, including subtitles

Now, for the final step.
We will reattach the silent video left aside in the first step to the dubbed audio.
However, there is nothing particularly difficult about this step. After all, FFMPEG does it for us.

Source code main

This is the main source code. Once again, irrelevant parts have been omitted.

What we are doing is creating the save path and executing the command.
We select the silent video and the dubbed audio created this time for FFMPEG. Also, as options, there is a setting to not touch the video at all and to add audio in standard AAC format.
By not touching the video, we can prevent quality degradation and expect faster processing.

Surprisingly, the dubbed video is complete with just this. Clap clap.

Source code: Subtitle input

Actually, there is a process of editing the silent video and adding subtitles before creating the video mentioned earlier. Since this is an option and may not be done, the procedure is not incorrect. This is the final procedure.

First is the introduction of the first half. Irrelevant parts are omitted as well.
Here, we use OpenCV.
We open the silent video and obtain the necessary properties.
At the same time, we also prepare the output video.

Subtitle settings.
Since the size of the subtitles is determined by a percentage of the vertical dimension, I am calculating that.

And the second half.
Here, we engrave the subtitles one by one while reading the video frames.

First, the necessary information is whether we are in a section with subtitles or not. Here too, we use the information from the speaker analysis about when they spoke from and until when.
Based on this information, we branch based on whether the current frame is in a speaking section or not, using the frame rate.

If we are in a speaking section, we engrave the subtitles.
By the way, to avoid unnecessary load, we detect when entering and exiting, and only perform tasks like generating text objects at those times.

OpenCV also has a function to write text, but since it only supports alphanumeric characters, I am using Pillow to engrave the text as image processing.
The key point here is the "textbbox" method.
This is a method that calculates how much area the text will occupy when rendered.
By using this, you can write subtitles in the bottom center of the screen. Also, although I haven't verified its operation, it includes a feature to wrap the text once if it is too long.

By the way, this "textbbox" was added in a later version, and in my past projects, I sometimes used a similar, different method. Since that has been deprecated in the latest version, it won't start up...
Even when asking AI, it sometimes provides answers using this old method, so if it doesn't work when you copy and paste, this is often the issue.
The old method exists up to Pillow version 9.

Summary

This time, I explained a tool that automatically converts natural voice commentary into VOICEVOX commentary.
Python really has so many libraries that you can do anything depending on your ideas and how you combine them.
There are still a few features I want to add but haven't been able to yet, so I think I will update it slowly.
See you again.