Video Album

For my final project I would like to expand on one of my projects from the past few weeks.

One of the ideas I had was to continue experimenting with playing videos from still photos as an exploration of stories content and memories in physical spaces rather than in digital folders on a desktop – I’d like to continue de-archiving the hard drive I found with footage from 10 years ago to create a physical photo album to house all these memories in a non linear way (or perhaps, somewhat linear because the pages of a photo album are usually turned from left to right).  I would like to finesse the way the video starts playing on actual photos that are developed (fading up, mapping accurately, etc), and perhaps use virtual buttons to begin playback or sound.

I’d ultimately like this to be something like an interactive documentary, the adventure of a few wide-eyed 18-year-old college kids traveling down to this forgotten town of Centralia in Pennsylvania in 2007, shooting social media style “stories” on a camcorder with our Magnetic Fields and Beirut songs playing throughout our car rides. I believe there are only 7 inhabitants left in the town today, but there was also a time capsule that the town buried in 1966 that was uncovered after we shot all this footage in 2014, so in a way I think the format of this video content could mimic the idea of preserving lost memories of this town in physical artifacts.

Another separate idea I wanted to expand on is the 360 video orb. I’d ideally like to have the orb floating in space and when the user touches it, it expands to a larger semi-translucent 360 video sphere that encompasses the user’s immediate area. For this next week though, this might be a bit too tricky, as from my previous experiments trying to do this with Vuforia, the tracking kept breaking, and between my personal device and what’s available at the ER, I don’t have access to a device that can run ARKit. I would also ideally prefer to build this for the hololens, since I think the effect of looking around and feeling immersed in this sphere would be much much more effective without holding a screen up on a device to view it.

Midterm in progress

I started collaborating with Terrick for our midtern, and so far we have a general geometrical sketch laid out of our world. We really want to work with videos and make an experience that’s on rails that slowly travels through a video corridor alongside a cat.  At the end of the corridor, we want to push past a 360 video sphere into an actual skybox.

Of The Four Types of Stories in VR, this experience would be Ghost without Impact, as the user is led through this hallway with the ability to look around, but without any local or global agency.

In hindsight, we were likely inspired by Passage.

Mood Board

This week I started exploring different tools in tiltbrush to see if I could create dream-like structures to walk be able to walk through. I like the aesthetic of warm colors that pop and lights that are emitted in strange and captivating spaces; I’m very inspired by the works of artists such as Pierre Huyghe and Philippe Parreno for their use of lighting, Pipilotti Rist for color and video textures, and Ryan Trecartin for general experimental narratives.

I want to explore what editing in VR means, by using a soundtrack as a base off which to create events, or perhaps reveal parts of the world to guide the experience. The goal would be to create an environment that doesn’t make much sense linearly, but evokes a chaotic hum with a playground feel, with objects and structures that emit nostalgic content.

Virtual Buttons // Telepresence

Keeping in line with my video theme, I wanted to start exploring 360 video in AR to try to evoke a sensation of being elsewhere.  I had some 360 footage I shot in the desert of Israel and I used it as the texture on a sphere with inverted normals in Unity, with some opacity to be able to see the real world environment through it.

I initially wanted to make it a giant 360 video that surrounds the viewer by using the ground plane, but apparently my iPhone6 doesn’t support “SmartTerrain”. So I kept my image target, but I’m happy I was pushed to explore virtual buttons because by adding a rotating animation to the sphere, it made the experience better so the viewer can “spin” the video a full 360 degrees around the Y axis to see the whole environment without having to move all around it. I tried to apply extended tracking to a giant sphere that surrounds the viewer but it kept breaking, so the end result is a scaled down version, which has a different effect – kind of like a snow globe or crystal ball.

Ultimately I would love to try to have a live 360 video feed shown in this way, so the viewer can feel more connected in realtime to another location.

Augmentable Object Fantasy

The biggest obstacle I encounter every day when I’m trying to get out the door is getting dressed. This has been a problem in my life ever since I can remember, and so if I could augment any object, it would be an entire closet system that has all my clothing items and accessories scanned and integrated with an AR mirror. I realize this isn’t the most original idea; there are probably many companies developing this already, but I first came across the idea in “Clueless”, and have been dreaming about it ever since.

To expand on this idea, I think it would be amazing to have an augmentable wardrobe- clothing items that fully allow you to express yourself in AR. People could follow or subscribe to your “AR channel” and see your crazy outfit, hair style, or anything really.

Unreal // Interactions

I expanded on my world from last week, tweaking some of my design choices and adding interactive sound. I changed the material of the floor and walls to be all glass instead, as I found the Ocean material to be a bit too dark and distracting. I wanted to make different sound tracks start playing throughout different locations in the scene, so I added sound cues to the three different thresholds of the doors. I made sure the First Person pawn starts off facing these doors, which are aligned and meant to draw the user in, straight ahead. I personally don’t love when an experience starts off with crucial elements being in the “curiosity zone” (behind the user), so I also placed the pawn pretty far back against the front facing wall.  Here’s my walkthrough of the scene!


Vuforia // Centralia

I’ve been really interested in integrating video content into this spectrum of extended reality, so I decided to use still images that would come to life in AR to play a video recording I took almost exactly 10 years ago – to imbue memories into physical photographs as a sort of time capsule. At the time, I was a sophomore an an undergraduate student at NYU, and I went with a group of my friends on a roadtrip around Pennsylvania for spring break. I had a video art assignment I had to fulfill, and while I didn’t end up finishing the project, I was left with a hard drive I uncovered recently, which had a wealth of these recordings that feel like a mix between homemade videos and snapchat stories.

There are two image targets, one is a still of a title sequence, which explain the narrative a bit:

“Centralia: An ancient municipality located in the outskirts of society in rural Pennsylvania. Population: 9 due to coal mining fire circa 1962. The remaining 991 residents (approx.) are expected to return in 2016 to open a buried time capsule.”

The other image target is a rough trailer I had made at the time, the audio didn’t come through on this screen recording (I’ll have to re-upload this), but it’s meant to be a dream-like glimpse into this time and space.

I linked the video player within Unity to each of the image targets using simple planes. Initially, the videos started playing automatically as soon as I hit play, so it was a bit tricky to figure out how to make the video start playing only when the image target was detected in order to create the illusion that the photo became animated. I still need to work on making it stop playing when the image target is no longer detected.

I definitely want to keep going with this and incorporate all this footage into a larger mixed reality project, perhaps to explore relationships with time, memories, and archival content within extended reality.

Unreal // VR Chat

Here’s a few stills from my floating Unreal nightclub!

Three women dance in absolute sync to the Donna Summer track, “I Feel Love” on a water dance floor under a series of doorways, while fireworks go off into the water ceiling and floating windows illuminate different colors through the thick fog.

Between the Starter Content and Example Content packages, I had a lot to play around with, and the only thing I made was my hip-hop dancing lady in Mixamo/Fuse. This was the first time I worked with particle systems, and I adjusted the smoke one to give it a more foggy effect, which helped bring out the lights more.


The Lab was extremely immersive physically, to the point where certain characters made me step back a bit, and one popped out in way that was too scary and real. The bow and arrow game was great with the haptic feedback, and I liked how the experience played with scale.

VRChat!!! completely blew my mind. Unlike The Lab, it was more emotionally immersive- it’s is a game-changer for me. I’ve always wanted to try social VR and have been really interested in its future applications, and the first time trying it was beyond everything I had imagined and hoped it’d be.  My sense of embodiment wasn’t perfect, but good enough to feel present and lose track of time in this vortex. The subject of anonymity is particularly interesting, as it’s not actally truly anonymous with its use of real mics (although I’m sure users could blend their voices). It was jarring and somewhat terrifying at the beginning, but it started to feel somehow empowering in a strange way- encountering trolls and knowing that they could say whatever they wanted without being able to actually see me or physically affect me. I hear and read a lot about the potential harm and detrimental dynamics of social VR harassment, and yet I came out of the experience feeling more in control somehow. And then there were users who were actually great to talk to and explore other worlds with.  Even in my memory, these experiences and encounters feel very real, and whether that’s something scary or exciting, it’s clear after trying VRChat that very soon, these virtual spaces will be exponentially populated with more and more users.

4th of July

I want to be up here with you when it’s pink and it’s orange and cooling from the sinking sun.  It’s too chilly to know but the days will be long and the night will be warm and the ice cream truck somewhere around with that song. I won’t tell you to look at the stars because the Chrysler’s brighter, the Empire State, and FDR drive. The first thing I remember, I was up there on that really tall building, couldn’t fall asleep, looking out this way to the Queens sky. The sun came up right around here, but it probably moved further East by now. I still can’t sleep, but out here barely mind January winds. My toes are cold but I barely mind knowing that May is so soon, and the rosé’s really too sweet because it was cheap.

I had never been that close to free, alone up here, an island of fireworks, between that job and the next. But still, being up here with you, it would be magic and cute, we can get drunk we can dance and fall asleep. The little speaker going and the blanket laid out, I know there’s Shelter Island out there, but this is ours. And it’ll be gone by the time it’s overrun.