DEEP DIVE: A Virtual Underwater Escape Room

Narrative: You have fallen into the ocean and are now trapped in an underwater cave. By finding a light, breathing apparatus, and a helpful sea creature you may get out alive, but you only have 6 minutes. Can you escape?

How? The game was designed in Unity. Graphics are projected onto a wall. A physical foam floor pad with embedded switches controls the game play when stepped on it. A button on the goggles is another controller used to manipulate parts of the game. The floor and button electrical components are powered by an Arduino and serially communicated with Unity to create a cohesive working system. On screen prompts and sound guide the players to the finish.

Why? Deep Dive was an exploration of game design, virtual reality, and immersive environments. We wanted to take gameplay out of a headset and create a more communal experience. How do game dynamics change if you are untethered? If other people are watching? Because it was a physical computing project, our number one goal was to allow the body to move freely and encourage physical interaction from the participants. Our controller was designed to expand from just a small handheld device, a screen, or a small dance dance revolution style floor, and grant our participants the opportunity to walk MORE. We believed that this type of physicality would enhance the exploration of the environment.

Read more about the process of creating “Deep Dive” through the links below:

The Deep Dive team: Dylan Dawkins, Cara Neel, Maya Pruitt

RETURN TO PORTFOLIO

PComp Final Progress: Component Iterations

This project demanded a interconnectedness between the physical and the virtual. It was a complicated balance and a long process. Below I try to breakdown the steps, the division of our labors, and the process of iteration for each component of Deep Dive.

CREATING THE INTERACTIVE FLOOR:

After the first play test we began construction of better plate switches. We used two layers to keep it flat. Our goal was to make the switches barely noticeable under the feet so users would have “stumble across”, so to speak, the game controls.

Version 1: Our first iteration of the floor pad used chip board (a non-corrugated cardboard) of different thickness. We used a thin chip board to lay on the floor underneath the mat. In certain areas we taped a layer of tin foil, stacked a stiffer chip board frame over it, and then added a tin foil on the underside of the mat.

Result: The tinfoil sections kept touching creating constant signal. We needed to create a thicker separation between the pad and floor. Luckily, we purposefully used tape early on to account for future adjustments.

Laser cut design of plate switches

Laser cut chip board pieces. The “Frame” shape was key.

Version 2: For our next playtest we glued all components down, and soldered our connections to a perf board.

Result: Eventually the tinfoil began to tear and it was quite cumbersome having everything wired together with two separate layers that had to be aligned exactly.

Breadboard —>

Solder —>

Perf Board.

Version 3: For our final version, which was for the winter show, we knew we needed to seriously amp up the robustness. We purchases thin aluminum sheet metal because it is conductive, and we could easily cut it to size. We maintained the original tinfoil on chip board layer, glued the chip board frames on top, and then glued the sheet metal on top of that. This allowed us to have the mat completely separated from switches and we could even alter their position under the mat if we wanted.

Result: We did have one issue where hot gluing connections to the sheet metal actually ruined its connectivity, but using tape was a quick fix. In the future, it should be soldered. Overall, it held up really well during the two-day show.

HOW DO YOU LIKE THEM BUTTONS?

After all our playtests, we realized getting people to touch the wall was really difficult. Do we have to make an obvious button? Are we trying to hide it so it still looks like a wall/the projection over it? We had a lot of questions about how to make this button effective. By a sort of happy accident, we created small plate switches that users had to end up holding during our tests, so we ultimately decided that it actually worked quite well. Players never questioned holding a device (even if it was tethered), because it’s quite natural in gaming. Cara came up with the idea to put it on “scuba goggles”. The goggles are not functional, merely an enclosure for the button, but it made it more fun, and fit perfectly into our story.

Floor mats are fun.

Button on the goggles turns on a light, I swear.

WORLD BUILDING IN UNITY:

Design of the world was conceptually thought out by all of us and then executed by Dylan. We knew we wanted a cave underwater with a tunnel to the surface. We had some iterations, in terms of figuring out how best to create the world and how it would be navigated. At first we thought perhaps each hidden object was located in a different scene, but ultimately settled on keeping the same environment and shifting view within it to find objects. A change in environment happens when you start on the surface and when you go through the tunnel at the end back to the surface (overall image of the model was developed in version2).

Version 1: As shown in our first playtest, the world was just the basic Unity demo with floating objects. For the true version 1, Dylan began to play around with water prefabs and the aesthetics of above and below the surface.

Version 2: The model of the world was created in Maya with a rock texture added. Using effects like fog created an underwater feel. For our initial runs, some of the objects, like the light, were only place holders.

Yellow sphere place holder for a light source. Notice the fog element that make the cave especially dark and difficult to orient.

Overall shot of the world. A tunnel from the top is where players initially fall into, the cave below is the main environment to be explored to find objects. Another tunnel leads outward for the final escape to the surface.

I played around creating assets in Blender, attempting 3D modeling for the first time. Ultimately, my coral didn’t make it into the final piece, but it was a good exercise for me.

Coral inspiration.

Rendered 3D coral model

SOUND DESIGN

I felt sound was crucial to the underwater ambiance for our project, so I took the lead on creating a narrative soundscape. I thought about how to make it most realistic, selected the sound clips, and mapped out where to place them in the story (for example what sounds are continuous vs. which are triggered by events), as well as how to apply them in the Unity code. Sound became an extremely useful device to create cues to players about what was happening as well as instilling emotion, like for a win versus a loss.

Pseudo code for sound placement.

Version 3: For our final Unity version, play through streamlined, all assets were in place (such as a real light source), on-screen text prompts were added, and the sound design was implemented.

RETURN TO DEEP DIVE

PComp Final UX Research

Playtests are a really important part of developing an interactive device. When you are so close to something it’s hard to know if what you’ve planned actually works for real people.

It was important for us to observe how people would use the plate switches or buttons on the wall and whether or not they felt inclined enough to explore.

PLAYTEST for 11/15/18:

Objective: Discover how people will explore our underwater world

We want to give limited instructions and see how people react.

Do they walk around? Will they figure out how stepping in a certain place changes the world? Can they figure out that certain objects in the world can be manipulated by physical touch?

Set up:

• 2 plate switches on the floor under yoga mats, hidden in random places. These control the orientation of the projected world (One moves left, one moves right).

• 1 Plate switch on the wall that changes the color of one object to red when it is pressed and the object is in view.

• Attach comp to HDMI projector in the classroom to display the Unity world on back wall

Instructions for the user:

“One object in the scene can change visually. Find it and change it.”

Video includes users from Playtest #1 – 11/15 and Playlets #2 – 12/6

OBSERVATIONS/NOTES FROM PLAYTESTING:

• Figuring out that stepping will move the view of the world was intuitive

• Pressing the “button” on the wall was not

• How are we giving directions?

  • Screen prompts? Audio?

• How do we indicate that the wall is interactive?

  • Outline of hand, obvious button, projection mapping (create panel with buttons behind it)

After our playtest we also finalized our storyboard and created a table to plan out exactly what is happening at each moment.

RETURN TO DEEP DIVE

PComp Final Progress: Initial Steps

To help ideate, Dylan, Cara and I actually went to Escape the Room NYC and tested out their Submarine themed escape room. While our use of technology will be far different, our biggest take away was that it is most fun not knowing what parts of the room are actually meant to be interactive.

We escaped the room!

This aspect confirmed our decisions to hide the plate switches on the floor and aim to use projection mapping for items on the wall. We don’t want things to look like obvious buttons in order to encourage exploration and let people “discover” what works as they go.

Cara mapped out the placement of our sensors to look something like this:

I worked on creating our first plate switch out of cardboard and hooking it up to an LED in Arduino as a simple way to test. Stepping on the switch turns on the light.

Dylan began working on having the Arduino talk to Unity. Using push buttons to represent the plate switches, he made it so that the left and right physical buttons changed the camera view in Unity respectively.

RETURN TO DEEP DIVE

PComp Final Proposal

Group: Working with Dylan Dawkins and Cara Neel

IDEA: AN UNDERWATER WORLD!

Using projections, we want to transform ideally any corner of the room into an underwater escape room experience.

Quick sketches by Cara to illustrate our initial vision

After talking with David about this, we realized that world building is a pretty huge undertaking. We needed to decide whether the user is meant to simple explore it or if they are a character in a narrative. To help frame the world, we fused these concepts together, creating a story that guides the user through exploration in our world, and then to escape it!

Here is a link to our initial storyboard.

Thinking through the technical aspects and interactions: we thought about gesture capture or building some sort of remote control but we ultimately decided that the main physical interactions would be created with pressure plate switches on the floor and wall (linked to an Arduino). The Arduino would then communicate with Unity to build the 3D modeled world to be projected

PURPOSE: We want to create an experience that alters perception, and changes the way you expect a surface to act. We want to heighten physicality by allowing the user to walk around and engage with the world with their hands and feet. It is an opportunity for people to reflect how their presence affects space, while playing a surreal game.

Next steps:

-playing around with projections

-building the Unity world

-creating plate switches to trigger changes to the projection

RETURN TO DEEP DIVE

PCOMP FINAL BRAINSTORM

For this post, I’m just going to hash out some ideas for the physical computing final.

Our assignment is to make an “interactive system”, meaning we should think about it as a feedback loop. Instead of having just one interaction (user input, to device output), can it create two-way communication?

IDEAS:

1. Not sure if this can be turned into an interactive system, but I like the idea of revisiting my undergraduate thesis. I have actually submitted an IRB for a second experiment to jump off the original thesis. Using eye-tracking, my advisor and I were interested in understanding how art training affects the way humans look at art. It would be super cool to create this eye-tracking device that paints. What I mean is that I predict, that expert artist often view art imagining the process and how they would create it themselves. I predict that a representation of their eye movements might look different than people who don’t have art training.

   • The result would be different “drawings” that represent the eye movements. Perhaps different colors for each part (are the looking at the edges of something, the middle.

   • Since I also used protocol analysis, this technique could be used again but this time something is taking the dictation as it happens. Maybe certain words trigger a different output.

   • Perhaps this is too ambitious and I could table this for my thesis, or maybe it’s a combination of ICM and PCOMP? I don’t know, but I think it could be interesting, and I would love to have ITP collab with the Cognitive Science department at Vassar somehow.

2. I just really want to create a giant projection of water on the floor and I want people to be able to move the water with their hands, or touch a fish. Is this even PComp idk?

3. Maybe I can expand on beat visualization and make it an entire room installation. Could delve into the world of the psychological phenomenon of synesthesia – associating colors or smells to different sounds.

4. Mental health has really been on my mind lately. I don’t really know how to incorporate that, but it would be very meaningful to me to crate a device that provides comfort to those who are struggling. There is a lot of stigma around mental health perhaps because humans are so judgemental, maybe if you talk to a machine that can take away that fear? or maybe it makes one more uncomfortable? Would love to expand of this somehow: Sit in a chair that warms to provide comfort.

5. Giant interactive rube goldberg machine

PCOMP MIDTERM PART II: FABRICATION

See below for more about the process.

The first halloween midterm post was getting a bit long, so I wanted to break it up into a second post, but also as a way to distinguish that creating the Halloween disco ball was truly a balance of two major components: the programing of digital elements AND the fabrication of the physical parts. Without the former, it is an empty shell, and without the latter, it is a bunch of LEDs and wires. So we thought critically about the fabrication of our project as well.

We started with a search for the jack o’lantern. We wanted a preexisting enclosure to save time. We settled on a plastic cauldron with a raised and painted flame design on the side.

Immediately, I disliked the way the flames were created on this cauldron, but I saw it as an opportunity to emphasize the future LEDs to be placed inside. I decided to cut out the flames, so that light could shine through, and since it’s a flickering candle in its normal state, this worked out nicely.

However, we didn’t want our shell to just have a bunch of flame-shaped holes, we would still need to enclose the LED bulbs inside, so we thought using some sort of translucent material would diffuse the light nicely. I tested with translucent acrylic and thought this could work well. We could laser cut the pieces and use some sort of plastic adhesive to hold it in place. But then, my dad has a genius suggestion and showed me a think translucent plastic that he used to use to diffuse light for photographic sets. This changed my path completely, the plastic was thinner, easier to cut and manipulate and I could paste it inside flush against the plastic.

Khensu-ra and I were also keen on adding a motor to our project. Originally, we planned to make the disco ball rotate. However, we quickly realized that this could be problematic with twisting all the wires inside. So we ditched this idea and thought what if we could make something come out of the cauldron! We became determined to create a fan that could blow confetti out of the top of the cauldron.

So the challenge had been set: we have a disco ball cauldron that’s sensitive to the environment, that lights up to the beat of the music, and blows confetti. HOW THE HECK DO WE BUILD THIS THING?

I sketched out a design that layered out components inside this cauldron. It was important to me that the final product would look very clean, but that we wouldn’t have to go crazy on cost.

Description of sketch (from the bottom layer to the top):

1. We wanted the first layer, so to speak, to be a box to enclose wires, the Arduino, and the fan battery. Luckily, I had scavenged this perforated metal box from earlier in the year. It had an open top though, so we would have to create some sort of lid. We ended up finding some left over acrylic to laser cut for this.

2. We would drill holes in the four corners to attach to the box, a hole for the fan switch, 3 holes for the cauldron to be attached and one larger hole to run wires through.

3. The cauldron would have the translucent plastic attached from the inside.

4. Then, we would rest a small tower to hold the breadboard with LEDs at the bottom, and mount the motor for the fan on the next layer (made from two laser cut acrylic panels stacked with dowels).

5. Drill a small hole on the edge of the cauldron lip to expose the photocell.

6. Mesh layer on the top to rest confetti on top of.

So this became our task list, and in execution we accomplished each step pretty much to a tee, but it was certainly a challenge!

Creating the stacked interior tower:

We drilled into the acrylic to create a circular indentation that the dowels could fit into.

Khensu-ra holding dowels and acrylic pieces together while the glue sets.

Get that motor runnin’:

Creating the motor mechanism was an interesting part. Getting our motor to run was easy, but getting it to blow confetti was difficult. We were originally advised that we should use a tube where air could be underneath the fan, otherwise it wouldn’t blow anything. When we tried this, however, the motor created a vacuum and ended up pulling the confetti in the opposite way that we wanted. We tried enclosing the motor instead with a back to the tube. This worked, the confetti has nowhere to go but up and out!

Late night madness in the shop.

Soldering the motor switch.

Motor mounted and spinning.

Fan blows confetti out of the tube. I celebrate every victory.

Of all these elements, which part do you think would have been the hardest to complete?

Well, whatever you thought, you’re wrong.

It was the DAMN LID for the box.

Surprised? Yea us too.

Khensu-ra and I measured this lid probably a million times, with electronic calibers even, to make sure all hold would be in the exact right place. Alas, we adjusted it over 7 times, mocked it out on cardboard about 6 times, and sadly the final acrylic piece still did not fit correctly. TRAGIC. Ashamed to say, I had to file out the four corner holes in desperate measures to make this thing sit on the box correctly. It worked out, but I’ll never forget the pain.

Found metal box to be used as the base of the cauldron and to enclosure the Arduino and battery.

One of a million laser cut cardboard prototypes of this lid and that was even before the other holes were added.

Final acrylic lid with holes. After all that it fit better having the switch in the opposite side of what we planned. Ugh the struggle, the sacrifices.

Another unpredicted issue was that the metal box was actually super conductive. I didn’t want to risk killing my Arduino, so I had to cover the whole inside with tape (and the bottom is cardboard) to make sure no wires or pins from the Arduino could touch the metal surface of the box. This solved the problem.

Glued transparent plastic for the flames, outside view.

From this photo you can see how the photocell was mounted to the rim of the cauldron. The stacked tower placed inside. The translucent plastic for the flames was adhered by cutting it larger than the flame and glueing the edge around the opening.

Pictured here is my finger on the fan switch with cords run through a hole in the green panel to be hidden in the box. Behind the cauldron is the metal box after being covered with blue tape and cardboard inside to prevent electrical shortage.

With all the code working, assembly seemed straight forward. The components worked, we just needed to put it inside this weird bowl. Our engineering of this device was actually quite sound, but all the movement of pieces, inevitably always made a wire or something pop out and left me quietly crying that the disco ball had suddenly stopped working. Ashley came to my rescue and said I could hot glue wires to the breadboard and it will help them stay in place. BLESS, this solved my issue entirely. As long as everything stayed connected, there would be no issue.

Shot of the final product.

FINAL THOUGHTS:

Ultimately, the halloween disco cauldron was a success! I think people enjoyed how responsive the cauldron was to the light in the room. It was a simple interaction, but an effective one, because it’s reacting to environmental change. In the end, we ended up using a lot of found material and strategies that we had learned from Intro to Fab (yay!). In addition, I very much enjoyed watching people’s reaction when they flipped a switch and saw confetti blow out. It was also a hilarious instinct that everyone would quickly clean up the pieces and put them back in. If I could do it over, my one big change would be to use stronger brighter LEDs, this would make the party state even more pronounced. Overall, I am very proud of this piece. It is clean, well constructed, and gets the point across. But the most exciting part is that it has the potential to be scaled up. The foundation is there, so where could we take it next?

PCOMP MIDTERM MEETS HALLOWEEN

Perfectly timed with Halloween, it was only fitting that our Physical Computing midterm would be something interactive with this theme. For this project I worked with Khensu-Ra Love El (check out his blog here).

Rather than a likely spooky scary approach, we decided to do something fun and make a Halloween Disco Ball! This post documents the process.

When coming up with ideas we talked about different sensors we wanted to try, as well as how we wanted people to interact with our project. I really wanted to use a photocell resistor in this project, which is a sensor that measures light. I loved the idea of creating two states with our project, how would exist by day versus night?

My main idea was to have the photoresistor set to a threshold to know when it is dark in a room. Darkness would trigger different party lights. Khensu-ra also liked the idea of adding music, which I was definitely on board with. This became a really great way to include serial communication to Javascript since we were motivated to use actual mp3 files, not just robotic tones on the Arduino. Specifically, I had to have this thing play Thriller!

We decided to create two states, one in light and one in darkness - an ode to Halloween as a celebration of creatures of the night. Initially we thought of doing a Jack-o-lantern looking enclosure. We imagined that in the light state, the lantern would flicker like a candle and in darkness it would light up in different colors. Kind of a fusion of the images below:

Creating the flickering candle effect seemed like a good place to start. Referencing this code, the Arduino and LEDs create a convincing candle light effect, especially when placed behind a translucent material (a sheet of paper in the video below).

To create the trigger of light to dark, we decided to use a photocell resistor, which measures light. Even though it’s tiny, it’s quite sensitive. On the breadboard, I added more LEDs and figured out a threshold for the photocell. If the photocell is above a certain threshold value: enter candle state, if the value is below the threshold: enter part state.

After creating these two initial states, we met with our Professor David Rios to figure out how to play music beyond Arduino tones. This proved to be a good opportunity to try serial communication. p5.js could read incoming photocell value readings and play music accordingly. We decided that turning the lights out in a room would also trigger the music playing.

It took us quite a bit of logic to figure out how to write the code for this. We needed our discoball to be aware of its state. It needed to recognize that it was both below the darkness threshold as well as know if it had already begun playing music.

function gotData() {
  photoVal = serial.read();
  if (photoVal < threshold && isDark == false) {
    song.play();
    isDark = true;
  }
  if (photoVal > threshold && isDark == true) {
    song.pause();
    isDark = false;
  }
}

Function gotData() reads the incoming photocell value from the Arduino and plays music when conditions of the if statements are met.

The result is a pretty sensitive system:

To make the lights more responsive and interesting, I wanted them to pulse to the beat of the music. I thought this would add more drama to our disco ball. I explored audio frequency visualization first in p5.js (see code for that here). The next step was to add serial communication again, but now going the other way, to have p5.js send instructions to Arduino (serial out). The beauty of this is that the code for my ICM beat visualization remains the same, p5 can use FFT to parse through the music, it then sends out a bunch of serial.write() commands that the Arduino then reads to light the LEDs accordingly. Below is a video of this first test with a single LED mapped to the bass of the song.

Then I added two more LED to indicate bass, midvalue, and treble! [video]

SERIAL COMMUNICATION LABS

For these labs we learned that Arduino can not only extend to the physical world, but it can also communicate with your computer. Here comes the internet of things!

SERIAL INPUT

We first learned “Serial In”, meaning the Arduino will receive input or values from a sensor and communicate that information to the computer. Specifically, this is done in p5.js, a javascript library.

Computer reads potentiometer values and changes the background color of the canvas accordingly:

SERIAL OUTPUT

“Serial out” is the opposite of the above lab. This time information from the computer will output to control something on the Arduino. The mouse location on the canvas determines the brightness of the LED.

ARDUINO DIGITAL & ANALOG LABS

ONE PUSHBUTTON, TWO LEDS

CONTROL LED WITH POTENTIOMETER

OTHER VARIABLE RESISTORS: PHOTOCELL

CONNECTING ANALOG INPUT SENSOR (POTENTIOMETER) AND SERVO