I tried making a game where you can create assets with AI and play with a REALITY avatar - REALITY Advent Calendar #6

Hello, this is Kato from the REALITY Unity team. This article is for the REALITY Advent Calendar 2022 on December 6, 2022.

At the start of the camp, I felt like trying out the AI image generation that has been popular lately, so I made a 2D scrolling game where you can play with a REALITY avatar using background assets created by AI.

For the image generation AI, I used Stable Diffusion (multiple base models) in the "AUTOMATIC1111 version Stable Diffusion webUI" environment.
I think this is the most common method currently.

Generating asset images

This time, I generated images using a method called img2img, where I pass text and reference images to the AI to have it generate images.

First, since I didn't have a specific image of what kind of background I wanted, I used an abstract image like the one above as the base for image generation. (It's definitely not because I'm bad at drawing... really!)

By loading the doodle, entering a few spells (keywords), and executing the generation, an image that looked quite like a game background was generated.

As shown in the image, if you pick up your favorite images from the generated ones and use them as a base to execute the generation again, it gradually becomes a nice-looking image.

In the end, I decided to use this cyber-city style image as the background.

Expanding asset images

I was able to generate a background image, but since it is square image data, it lacks volume to be used as a 2D scrolling background.
So, I used "Hua," a UI that allows you to use "outpainting" to predict and generate the continuation of an image and "inpainting" to redraw unnecessary parts, to make it a horizontally long image.

Chibi character

Since standard avatars felt a bit too tall for a 2D side-scroller, I decided to use chibi avatars (which exist as an internal feature) for the player character.

Game

The game was kept simple to fit within the duration of the camp; the objective is to fire bullets (which I modeled) at REALITY-kun, defeat the moving REALITY-kun, and progress to defeat the boss (Golden Bear).

Conclusion

At the time I started writing this article, only about two weeks had passed since the end of the camp, but Stable Diffusion version 2.0 was released during this time, and it feels like the environment surrounding image generation AI has advanced another step in just a short period.
Even from the outside, I had the impression that image generation AI was evolving at a tremendous pace, but after actually getting my hands on it, I felt that even more strongly.
I was able to get hands-on experience with the technology I was curious about through the camp, and I think it was a fruitful experience!

Tomorrow's Advent Calendar!

Tomorrow, IKEP from the same Unity team is scheduled to write a cool article!