Unveiling "Magical Forest" - Using Generative AI for WebAR Development
Introduction
■ Experience the "Magical Forest" from the 8th Wall Website
Brainstormed with ChatGPT
Because the Desginium team creates prototypes very quickly, in the initial stages we started by testing AR/MR technology for catching insects and walking in the forest. After the core (part of the experience) of “Magical Forest” was realized through the prototype, we used ChatGPT (GPT-4) to generate ideas to flesh out the story settings and experience content.
Think about Experience Content
First, we listed the "AR technical specifications" and "experience flow" as prompts, hoping that ChatGPT can provide ideas to add some interesting elements besides catching insects.
[Prompt]
Please come up with ideas for planning AR game apps for smartphones and headset devices.
AR technical specifications
?The camera can detect the flat surface of the floor, but cannot detect walls or obstacles.
?The camera can recognize some gestures.
?Characters can be controlled via interface or controller.
Experience Flow
?A dwarf named "Kobbito" will appear.
?When the player taps on the book, Kobbito and the magical forest will appear in AR.
?Players walk with Kobbito in the magical forest and search for insects.
?Caught insects will be returned to the book.
?Does the game end when all insects are caught?
?There are single player mode and multiplayer mode.
ChatGPT proposed to us game design ideas such as "story development", "character setting", "skills and challenges", "competition and cooperation", "progress rewards", "magical forest expansion", and "magic casting settings".
[One of the responses that interests me]
Magic casting settings:
Certain insects can only be caught by performing certain actions or
steps. For example, some insects appear by singing magical songs or
assuming certain postures.
We selected one of the most interesting responses and asked ChatGPT to provide more relevant examples. Based on ChatGPT's responses, I keep tweaking the question until I got the answer I was interested in.
[Prompt]
Could you give me concrete examples about the "Magic casting settings"?
[Prompt]
Can you give me 10 ideas for AR apps that take advantage of the
"Magic casting settings"?
After asking this question several times, ChatGPT came up with the following thoughts. Because not every idea offered by ChatGPT can be implemented, we need to choose the ones that look feasible to guide the conversation.
[Response]
Make AR In-game Items:
Catch specific insects by combining 3D objects to create new items.
Wave a Magic Wand:
Wave your smartphone like a wand to capture insects that respond to
specific movements.
Magic Spell:
Say specific words or phrases into your phone's microphone to summon insects.
Magic Potion:
Find specific ingredients and combine them to create a magic potion.
When you use magic potions, specific insects will appear.
...(omitted)
The idea of "Make AR In-game Items", using items to catch insects, seems to be able to be integrated into the experience. And it also fits the setting of the magical world very well. So I narrowed down the ideas, hoping that Chat GPT could come up with more specific examples of in-game item settings.
[Response]
Light Wand:
Item: magic stone + wooden stick
Result: Create a glowing wand that illuminates insects hiding in the dark and makes them appear.
Wind Whistle:
Item: wooden pipe + bird feather
Result: Play the flute summons wind, which will cause flying insects to fall.
...(omitted)
Once we come up with specific items, we consult with engineers to see which ones are technically possible to implement. We choose to allow players to use magic items through "spell cards" during the game experience. Magical powers can help catch insects. Players can use magical powers such as "Magic Magnifying Glass" to see transparent insects, "Magic Mushroom" to attract nearby insects to come closer and etc.
Think about the Background of the Story
[Prompt]
Could you please give me 10 ideas for the story of "Magical Forest" based on the conversations we've had so far?
[Response]
The 10 story ideas proposed are as follows.
Time Fragment:
The dwarf is a messenger from the future. The future without insects
in picture books is chaotic. Players from the past hold the key to
saving the future.
Guardians of the Magical Forest:
Insects are the spirits that protect the forest. Without them, the
vitality of the Magical Forest would diminish. By bringing back
insects, light returns to the forest.
...(omitted)
First, I sent the proposed content back to ChatGPT and asked them how they could make the story more engaging. I received the following advice.
[Response]
The relationship between Kobbito and picture books can be explored
in a little more depth. The characters the Kobbito play in picture
books and the problems and emotions they face in the real world are
also factors that can make the story more engaging.
Then, I asked ChatGPT for specific examples based on "their own advice", based on "the setting of the magical forest" and "what is interesting about the world of the Magical Forest ?"
[Response based on their own advice]
Guardians of Picture Books:
Kobbito and his friends take on the role of protecting picture books
and strive to maintain peace and balance within picture books.
...(omitted)
[Response based on the setting of the magical forest]
Magical Protection :
The Magical Forest was originally a special place in picture books,
protected by Kobbito. When the insects trapped in the forest escape,
the forest suddenly appears in the real world.
Space-time Distortion:
Because Kobbito used magic incorrectly, the time and space in the
picture book was distorted, causing the real world and the world of
the picture book to overlap.
...(omitted)
[Response based on What’s interesting about the Magical Forest?]
Magical Forest = Past World
Players travel back to the magical forest of the past world to look
for insects and creatures from the past era.
Magical Forest = Planets in Space
The forest actually exists on a distant planet, and players explore
that planet in a spaceship.
...(omitted)
It’s interesting to come up with ideas that I wouldn’t have thought of on my own. Once we confirmed what the story would look like, we could manually modify the setting and story to fit the specification and incorporate previously discussed ideas. Finally, share it with ChatGPT and let ChatGPT tidy it up. Here is the backstory we created and polished through ChatGPT.
[Response]
A picture book created by a legendary magician hides a door
connecting the real world and the magical world. In that magical
world, there lives Kobbito, small creatures full of curiosity. One
day, driven by the spirit of adventure, Kobbito opened the door to
the picture book world and stepped into the real world. However,
because they forgot to close the door, the insects from the magical
world followed them to the real world. With the power possessed by
these insects, a magical forest begins to spread in the real world.
In order to maintain the balance between the real world and the
magical world, Kobbito and the players embark on a grand adventure
to find insects and bring them back to the picture book.
Materials Generated by Midjourney
3D Textures
In the game, you can meet Kobbito and insects with mysterious patterns in the magical forest. The patterns for Kobbito costume and insects are generated using?Midjourney?as textures for the 3D models
STEP 1. Image generation
At first, I tried to generate a pattern by entering the word “Polka dots” as a prompt, but the pattern was too thin and too regular and beautiful. It didn’t feel right to use it as a pattern for insects. So I looked for a new direction by changing the words. I used “insect surface” in the prompt. You can also use the prompt “- -no” to specify content that you do not want to be included in the generated image. Maybe because I used “insect surface pattern” in the prompt, the resulting image sometimes looks like the insect limbs included in the pattern, but I can avoid this by using “-?-no insect body”.
In the end, after adding the prompt “strong distortion dots”, it is closer to the image I want. It is interesting to be able to create unexpected patterns by using?generative AI.
STEP 2. Apply to 3D models
We create the visual appearance of various insects by applying the generated images as textures to the insect model. Although I ended up not using the image generated with it this time, you can also create?seamless textures?by using the prompt “- -tile”.
I wanted to create a marble texture for the Kobbito costume, but if I just used the word “Marble” the color would be too light and not stand out. I tried different ways to re-express the marble pattern and in the end “like an amoeba” worked out well. Just like this, the generated results will be affected by how the words are worded, so I think it is important to get creative with your wordinig. Next, the generated image needs to be edited based on?UV mapping to apply to Kobbito's 3D model. However, this is still more efficient than drawing the design from zero.
Spell Cards
I designed the following spell cards using images generated by?Midjourney.
STEP 1. Image generation
I started with mushrooms, this seems easy. Initially, the mushrooms looked poisonous, but after trying various prompts over and over, I was finally able to generate?pop mushrooms that looked like game items.
The wait time to generate an image was quite tight, so I used “Set Items” and “Game Assets” which would generate an image with multiple samples. ↑ As you can see in the image above, it is efficient because you can choose as many mushroom designs as you want in one generation. Once you decide on the image you want to use, increase the resolution based on your expected size. In Midjourney, you can increase the resolution to?4096 x 4096 px?by pressing the “Upscale” button.
For “Magic Light”, it was hard to create the image I wanted. I want it to look like a floating sphere so that it wouldn’t deviate from the image of the game item. But even if I use the word “floating”, the resulting image of the item doesn’t look like it’s floating.
I also decided to avoid using glass or light expressions as they are difficult to edit when cutting out the image and turning it into material. I tried words related to light balls and ended up choosing an item that looked like a light ball from the images generated using the word “crystal.”
STEP 2. Retouch with Photoshop
When I put the two generated images together, I noticed a difference in their appearance.?The mushroom is 3D-like and the light is 2D.?So I applied brushstrokes and some effects in Photoshop to soften the visual differences. Additionally, I adjusted color and brightness so that it looks more natural even when the two items are placed side by side. Manually editing small adjustments like this in Photoshop is much faster than changing the prompts in Midjourney.
STEP 3. Design the Spell Cards
After adding the background and text, the spell card is complete. In the final version of the design, the color of the illustration was further adjusted to match the background color.
Key Visual Design
STEP 1. Image generation
First, I create a rough image in my head. The image I wanted to create was like Kobbito exploring the magical forest. I wanted to use the designs of the Kobbito and plants directly from the game, so I decided to have?Midjourney generate the forest background?and composite images rendered from the 3D model onto it.
If I add flowers in the prompt, the resulting image seems difficult to composite, so I decided to generate a simple forest without unnecessary elements. I've also added a tip to generate a 16:9 ratio image, since landscape is easier to use later.
STEP 2. Composite with Photoshop
Next, we will add elements to the forest background in Photoshop. I will?add images rendered from the 3D models of Kobbito and plants?used in the game. Because the individual materials don’t blend together and stand out, I also adjusted the brightness and color of each material to get balance throughout the image. To integrate the composite elements with the background, I added a stroke filter on top of the layer to blend it all together.
By the way, I later realized that I needed a wider image than ↑ the image above. This time I used a vertically stretched image, but I could also use?Midjourney’s Zoom function?to further expand the resulting image up, down, left, and right, in which case I can add more forest part to the background. The strength of AI tools is that they can respond to such needs immediately.
Title Logo Design
STEP 1. Image generation
Midjourney is not good at generating text, so I created the text in Illustrator as shown below.
I wanted to add motifs of books and plants to the text, which are key to the experience, so I decided to use Midjourney to generate these materials. An illustration-style logo was initially generated, but by adding “flat”, “simple”, “clean” and other prompts, the logo was refined into a flat one. When thinking about vectorization, it seems easier to use monochrome. Since I will only be using part of the image this time, my judgment will be based on ease of use of the subject rather than overall completion.
STEP 2. Vectorize the generated image
This time I used this website to vectorize the generated images. You can easily convert images to vectors by simply dragging and dropping them to upload them.
After downloading and opening it in Illustrator, you can see that it has been converted into a path as shown below. Then you can edit the path as needed. Through this process, I vectorize multiple generated images and extract motifs that can be used.
STEP 3. Combine the motif with the text and finishe in Photoshop
Add the motif extracted from the generated image to the text path and adjust the overall balance. I also added a leaf motif to the text using leaves extracted from images generated by Midjourney. Finally, texturing and coloring were done in Photoshop.
Audio Generated with Eleven Labs
Because there are scenes at the beginning and ending of the experience where Kobbito talks to the player, we thought it would be a good idea to add audio to increase the immersion of the experience. Therefore, we used “Eleven Labs” to generate the voice, an?AI voice generator?popular for its ability to create natural AI voices. You can try it for free, but if you want to use it for commercial purposes, you’ll need to subscribe to the pricing plan.
The “Magical Forest” has a need for?English narration, so we chose an English demo voice based on accent and tone. It is generated based on the selected demo voice, which you can also adjust the voice model in the “Voice Settings”. In addition to the default voice models, you can also use voice models generated by other users from the “Voice Library” in the menu.
Even if the settings are the same, the tone will change each time a new voice is generated. So we recommend producing multiple voices of the same sentence for comparison. It’s easier to adapt and use if they are generated sentence by sentence rather than all at once.
In addition, sometimes the sentence breaks and intonation of AI reading are unnatural, but if punctuation or spaces are added between words and regenerated, it may become more natural. When you feel uncomfortable with the way it starts speaking, using a space or line break at the first character may make it sound smoother.
You can check past generated samples and their settings from the “History” list, which is useful if you later want to check the settings of a previously created sound.
Conclusion
Advantages of using generative AI
Through this project, I was able to use AI tools to independently produce materials that would normally be outside my scope or require outsourcing. For example, I am not good at creating illustrations, but by using Midjourney to generate images, I can be in charge of everything from illustration production to final design by myself, as in this example of the spell card illustration. Another point is that you can give direct instructions (prompts) so that you can try out various ideas without hesitation.
Assignments
AI-generated content may lack consistency in style and subject matter. For example you cannot generate the same content again even with the same prompt. Therefore, it is difficult to fine-tune works generated by AI. And it will take a long time to continuously pay attention to the generation of small details. I think relying on design tools such as Photoshop to make such corrections can make more effective use of AI.
Also, it is difficult for AI tools to fully understand the overall context of a project, including its background, objectives, cultural elements, and emotional nuances. So it’s essential to tailor AI recommendations to your specific situation and guide you in the right direction. When using generative AI as design materials, it is necessary to carefully consider whether they are in line with the purpose and message of the project.
Demos
Finally, I would like to introduce to you some scenes from the final version of “Magical Forest”. The illustration for the Magic Mushroom card, the Kobbito and insect textures were created using Midjourney. If you have a?Meta Quest, we recommend using headsets to experience it!
■ See link to full text of the article
Designium Inc. in?Tokyo
Copyrights ? Designium, All Rights Reserved
[Event Info] XR Kaigi 2023 https://www.xrkaigi.com/