Generating Manga-Style Videos! (Runway Gen-3 x Midjourney x DomoAI)
Hello! I am Yoshikawa, the head of the UX/Direction Office at NijiBOX Co., Ltd.
The theme for this time is "Generating Manga-Style Videos!"
Since I originally started my career as a manga artist (a mysterious career path, I know...), with the recent evolution of generative AI, I've come to want to express myself in my own way, which is why I wanted to take on this theme.
And, after much trial and error, here is the video I created!
What do you think? I think it turned out quite well.
Ultimately, the method of combining Runway Gen-3, Midjourney, and DomoAI worked the best.
This time, I will generously share the detailed method for creating this video!
1. Generate the base video with Runway Gen-3
First, generate the base video using Gen-3.
I tell ChatGPT the scene I want to create and have it generate an appropriate prompt.
▼ Instructions for ChatGPT
I want to make a video with Runway, so I want you to create a prompt.
I want the prompt structure to be: Camera movement: Scene setting: Additional information.
The scene is a Japanese drama-style scene where a woman is waiting for someone she loves in the snow, enduring the cold.
I want to create a close-up of the woman's face with dramatic direction.
The point here is to have the AI generate the prompt.
Not only does it save the trouble of creating the prompt from scratch, but if the video doesn't turn out as expected, you can just give instructions like "this part is weird, so fix the prompt," so considering the effort of making corrections, doing it with AI is the best way!
And here is the resulting prompt.
The camera slowly zooms in on the woman, focusing on a close-up of her face. It starts with a full-body shot and gradually transitions to a close-up capturing the woman's expression. Snow falling around the woman reflects in the camera, emphasizing the coldness of the scene. The scene is in the style of a Japanese drama. On a cold winter day, with snow quietly piling up, a woman is waiting for someone she loves in a back alley of the city. Faint street light glows in the background, creating a quiet atmosphere. The woman is wearing a thick coat, but she is still shivering from the cold. Her hands are in her pockets, her face is red, and there are tears in her eyes. Her expression is a mix of anticipation and anxiety, and dramatic music enhances the emotion of the scene. The background sound includes the faint sound of cars in the distance and the sound of snow blowing in the wind.
Translate this into English, and the prompt is complete.
Then, input this prompt into Runway Gen-3!
Here is the video that was created!
With this, the role of Runway Gen-3 is finished.
2. Create manga-style illustrations with Midjourney
Now, next, we will create manga-style illustrations.
We will merge this into the video mentioned above later.
I will input the prompt created for Runway above into Midjourney as well, but I will further specify the taste.
Using the "Style Reference feature" implemented in April 2024, I will reflect my preferred manga style into the prompt to generate the illustration.
The method is simple!
First, upload the image you want to use as a reference for the taste (in this case, a manga image) to the Midjourney Discord and get the image URL.
After that,
Add " --sref (URL of the image you want to use as a reference for the taste) --sw 1000" to the end of the normal prompt for image generation (i.e., the prompt created for the Runway video).
*sw stands for how much of the sref element to inherit, and 1000 is the max.
*I will not explain the style reference method in detail this time, so if you don't understand the above, please refer to other articles!
Here is the image generated by combining the prompt for Runway with the URL of the image I wanted to use as a reference for the style!
That's it for Midjourney! Now, let's put it all together!
3. Combine the video and illustration using DomoAI to generate a manga-style video
Finally, it's DomoAI's turn!
I'll be using the "Fusion style v2" added in June 2024.
First, enter the prompt you used in Runway into the Prompt area at the top left (we've used this prompt three times now; it's so convenient to be able to reuse it!).
Specify "Fusion style v2" as the style. By selecting this, a "Reference Image" area will appear below, which will reference that image.
Upload the image created in Midjourney to the reference image area.
Then, upload the video created in Runway to the video upload area on the right.
After that, just press the "Generate" button at the bottom!
The resulting video is the one I showed you at the beginning!
I'm quite satisfied with how it turned out!!
Though it took me about a day of trial and error to arrive at this method... lol
4. Conclusion
What did you think?
AI tools are evolving rapidly, and we've reached a point where we don't know what will be released next week.
Amidst all this, I have a feeling that we are starting to reach a situation where an individual's personality and uniqueness are being sought after.
I want to continue to pursue my own way of expression and the kind of expression that feels right to me.
Thank you for watching until the end! See you next time!