"Star Agate" Production Diary ① | How was the AI anime MV made? Behind the scenes of trial and error (Vidu, NanobananaPro, Filmora verification notes)

Good evening, this is Kurono!
Today, I've picked out and summarized how I made certain scenes in the MV.
I forget how I do things right away, so this is basically for my own record... but I hope it's helpful to you all as well✨

◆ Tools Used

Illustration creation: Nano Banana Pro (Leonardo.Ai)
Illustration adjustment: CLIP STUDIO PAINT
Animation: Vidu AI
Music: Suno AI
Music adjustment: Studio One
Video editing: Filmora

◆ Production odds and ends

There are many things, but for now, I've roughly summarized the following three points.

[1] Layering multiple animations

The method follows the flow below.

① Place the torii gate animation in the background (e.g., at 0:39)

I create the illustration in NanobananaPro (abbreviated as banana from here on) and turn it into an animation with Vidu.
As an aside, even though I instructed it to "move forward into the torii gate," it ended up moving backward, so I reversed the playback in Filmora.
(Maybe it understood it as "the torii gate goes into the back"...)

② Create the animation to be placed on the top layer. Use a green screen for the background

I create an illustration of camellias with snow on them in banana.

I set the background to green screen in banana.
(I followed the steps above because I initially thought I'd use it as is, but I think you can just instruct it to be a green screen from the start.)

I turn it into an animation with Vidu.

The instructions are: "The camellia flower on the right falls. The camellia flower on the left stays still. The camera does not move."

③ Remove the green screen background in Filmora

In video editing, when you go to the "Video" -> "AI Mat Tool" screen, there is something called "Chroma Key". (Top left of the image) It is OFF by default, so turn it ON. After that, you can "Select Color," so use the eyedropper to select the green background.
By doing this, the green part becomes transparent, and only the movement of the camellia is displayed.
(As an aside, sometimes the color selected by the eyedropper goes to a different place than the cursor, making it hard to select, which is sad... is it just me? I'm used to it now though, lol)

After this, I adjust the "Opacity" item under "Video" -> "Basic" to make it look good.

By the way, I've layered various things here as well.

[Self-talk...]
If you use these AI matte tools too much in Filmora, there is an error where the preview becomes completely black in those parts or at transitions.
It's a mystery... When that happens, saving the data, closing it, completely closing Filmora, and then opening the data again usually fixes it.
It fixes it for now, but it's annoying when it happens frequently 😭
I feel like Filmora is screaming because there are too many heavy files, haha.

Also, depending on the original illustration, using chroma key can make the image outlines look very rough...! If you are going to use chroma key, it might be better to be creative with how you make the illustration for it. (Like making the outlines a bit thicker...?)
If you look closely at the cherry blossom part above, the back isn't very clean... It's frustrating! 😣

[2] Speed change: Eliminate monotony of movement with speed ramping

In my videos, I often use the original video with the speed changed. I often use normal speed changes, but in some places, I use something called "speed ramping" to make fine adjustments.I often use normal speed changes, but I use something called "speed ramping" in some places to make fine adjustments.

The animation playback time wasn't enough, so I slowed down the whole thing, but the part where the eyes open became a bit sluggish.
So, I set it to:
Start 0.88 → Before the ripple becomes the pupil 0:80 → When the pupil opens 1:70 → End 0.80.
※1 is the normal speed.
By doing this, you can make it faster only at the moment the pupil opens.
However, it's frustrating that the ripple is a bit sluggish (´;ω;｀)
Well... I just messed with it after making it because I realized I didn't have enough seconds, haha.

By the way, the original video was made by specifying frames for 4 illustrations using [Image to Video].

It might be a little small and hard to see...
"Simple illustration of a well" → "Illustration with ripples emerging" → "Illustration with eyes closed" → "Illustration with eyes open"
...is how it feels.
I made everything by giving instructions in banana, but it didn't take that long.

[3] Create the desired movement with keyframes

I added a little bit of movement to the moment the eyes open in this scene.
What I used was "keyframes".

Select the animation you want to operate on the timeline, and if you click the keyframe mentioned above at the specified location (where the red line is), a ◇ will appear.

If you change the "scale" or similar while the keyframe ◇ is selected, you can make it zoom in slowly.

This time, I wanted to make the part where the eyes open impressive, so I did the following.I wanted to make the part where the eyes open impressive, so I did the following.

① 1st keyframe (start) 100
② 2nd keyframe 110
③ 3rd keyframe 120
④ 4th keyframe (end) 115

If there were no ② keyframe, it would slowly zoom toward the ③ keyframe at 120, so the ② keyframe point would be 114.
Therefore, by inserting the ② keyframe at 110, you can make the movement from ① to ② slow, and from ② to ③ a bit faster.

If you overdo it, it will look strange, so it seems best to use subtle values.

Also, sometimes when I change the scale on a keyframe ◇, the ◇ mysteriously multiplies, so I still haven't really gotten along with keyframes yet...

◇ Bonus (About the structure)

I thought there would be quite a lot of scenes to create this time, so I summarized them in a spreadsheet.
At first, it was mostly blank, but I think doing this made the tasks easier to understand and helped things go smoothly.
(I usually just used bullet points in a memo.)
Well... things change as I make them, so this spreadsheet also has some differences, but I think it was good for the purpose of seeing the big picture.

In Filmora, if you change the folder where the original file is located, the link will be broken, so I tried to roughly number the video file names beforehand, or put them into a "Star Agate" folder before importing them into Filmora.

There are some unused ones in there too, but I don't know which ones to delete anymore, so I'm leaving them alone lol

◆ In conclusion

I was able to take on various challenges in this MV production as well, and it was fun✨
While I think I made something good, on the other hand, there were many things I worried about, like "I should have done this!" or "What should I do here?", so I hope to challenge myself again in my next work.

Somehow... the evolution of AI is so amazing that I can't keep up.
I still haven't properly tried ViduQ3 yet...
Kling 3.0 is also amazing, and Seedance 2.0 (is it not available in Japan yet?) is also a hot topic.

For clean-looking animation, Sora 2 seems good, and from what I saw briefly, Seedance 2.0 looked incredibly beautiful, so I'm curious about it...

...and although I feel like I'm about to stray, I recently signed up for the Vidu Ultimate annual plan, so I'm going to play with this for a while.
There was a 40% off sale in early February, so I clicked it. (If I use it for about 6-7 months, I should be able to get my money's worth!)

Just when I thought I would proceed with the animation production of the Gem Fairy Tale, a new OP-style MV idea came up, and my head is in a mess these days lol

【Related Articles】
I have summarized how I basically make things below✨

【Notes】
This article introduces overseas AI services. There are risks unique to overseas services, such as English-only support and changes to terms of service, so please check the official information and use them at your own discretion.