Nature in the Void


|

Introduction
Our final assignment for Hypercinema was to make a Cornell Box that represented something about ourselves, using Unity as a tool. This project went through two concepts: Putting Myself Back Together and Nature in the Void. 

I began the assignment wanting to draw inspiration from my own life and represent the act of healing through movement.

Later, I dropped this concept and moved on to create a nontraditional gallery of my photos. However, I learned a lot about Unity and developed the first idea in doing this assignment; I’m looking forward to returning to Putting Myself Back Together in a future project.

First Idea - What I Worked On & What I Learned

My first idea was a disassembled body that users would put back together using their hands. When the body was reassembled, it would transform into a character and start dancing. The whole thing would take place in a dirty, broken, small dance studio, which would expand and clean itself as the body put itself back together. 


I experimented quite a bit with the interaction, and even though I ended up shifting ideas to match the scope of time I had to do the assignment better, I still learned a lot.

I wanted to use hand tracking for the interaction, and chose to use the Leap Motion Controller. When testing, I learned about OSC communication. I also learned about importing SDK packages for different technologies into Unity (and how much it sucks). I attempted a couple of different methods of bringing Leap Motion data into Unity.

The method that worked the best was an OSC Receiver script that ChatGPT generated for me, and I edited to fit my use. OSC communication would happen from TouchDesigner, where I was receiving the Leap Motion data using built-in operators within TD. 

I also learned about Motion Capture and retargeting animation data onto a 3D skeleton. I created an avatar on Ready Player Me for my motion capture data.


In the end, I realized I wouldn’t have enough time to create choreography, do the motion capture session, retarget it, and match it to the interaction with the Leap Motion, and since this project was so personal to me, I wanted to do it justice. I started to re-scope after about 2 weeks into working, with 2 weeks left to create a project.

Re-Scoping - New Idea
Restarting from scratch, I started to think about other ways I could represent myself that were still personal, but not so complicated. I gave myself the challenge of doing more with less; How could I create a compelling, unique experience using just the tools I had in my belt already?

I’m a photographer and I’m also passionate about audiovisual experiences. I decided to combine these two to create an audiovisual gallery. I was also inspired by the lighting work we did in class on Unity and this sample project I made while learning:


I emailed Gabe, and he asked me how this gallery could be different from a traditional white-box gallery, which started my process.

Process
I started to think about how I could use Unity to create an environment that’s unlike the real world. The first thing I did was curate photos from my portfolio and bring them together with a theme. I’ve always been inspired by nature in my photography work and noticed that I had four sets of images that I could use to represent four elements. Once this theme was set, I started to brainstorm on how I could make my gallery different from other ones.


I knew I wanted it to be dark in the gallery, and use the first-person player as the light source in the gallery. I found my concept when browsing on the Unity Asset Store. I found a skybox set of a galaxy, and applied it to a very basic setup I made for the gallery. As I continued playing, I noticed that when I changed the material of the gallery itself, I was able to get a cool “trippy” effect that mirrored the skybox. 


This reminded me of the wormhole in Interstellar, and I started to solidify my concept more. I thought of the gallery as part of a speculative future where humans no longer exist, but left behind this time capsule of sorts in space to show what Earth and humans were like to another species.


Result

This was the final product. I set up triggers for video overlays and audio to play around the gallery.


Conclusion & Further Considerations

I like the direction this project took, and I’m excited to continue developing it based on the feedback I received in class. I want to extend this experience to Augmented Reality and try to set up a build on iPhone, similar to Vivian’s project. The idea behind this is to immerse the user. I’d like to install it in a hallway with four solid color canvases that serve as the image targets for each elemental animation.

I also want to use TouchDesigner visuals that I make on my own as the overlays for the video, and actually use tools like the Kantan Mapper to display the right resolution on each of the surrounding walls. I started this process, but wasn’t able to complete it in time for the final project.

Lastly, I want to bring in the narrative of the speculative future into the project somehow, adding in a narrative component with either UI elements or sound. I plan on working on this project over Winter Break and I’m excited to see what comes out of it!