Hello Virtual Reality

For my final project, I wanted to explore the idea of disembodied voices in our current social environments- chats that amount to chatter through shared media and collective perspectives. The earliest computer voice I encountered was probably the answering machine on our landline, which guided us through various voicemail messages.  In the future, I anticipate spending more and more time in a headset, so I wanted to find a bridge between “real” reality and virtual reality by taking this historic voicemail interaction to a speculative environment. I also wanted to explore different ways in which we can create feelings of co-presence in virtual scenes.

What I ended up with is an interface for people to view my stream in real-time and send me messages by commenting on my channel. Their messages would generate message objects in my scene, which they could then see and hear in the same stream.


Twitch is a streaming platform that is predominantly used by gamers in order to follow each other’s gameplay and comment in real time. I decided to use this as both a streaming tool to share my perspective from my headset to my own twitch channel. By using the Twitch API with Socket IO for Unity, I set up my project to instantiate spheres to fall every time someone would comment on my channel. I then used the IBM Watson Text to Speech SDK to say “new message” with each new sphere that would appear in my scene, and read out the chat message when I collided with it.  I streamed out my game view or headset display by using OBS to stream directly to my twitch channel.


I ran into a few technical hurdles trying to get Socket IO for Unity to work with the Watson API without breaking my project. After some trial and error I found a version of a Socket IO package that was compatible with other integrations. There was also limited documentation on how to use the Text to Speech function in the Watson SDK for Unity, but finally got it working with some help.


I definitely want to keep expanding on this project and interaction flow, now I’ve set up the technical framework. There are so many different directions to go with this- for example using the username data from Twitch to assign voices, or having the content of the messages instantiate different objects. While testing out my project, I asked my sister who’s in college 3 hours away, to send me messages while looking at my stream, and it was a really sweet way to connect with each other in real time. I look forward to more experiments!

Culinary Physics final proposal

During my first class at ITP, my physical computing professor asked us to form groups and come up with an idea for an ultimate “fantasy machine”, a machine that didn’t have to adhere to the laws of physics, or technical and monetary restrictions. My group devised a “Tea with Mum portal”, inspired by our longing to connect with our family overseas in a more meaningful way. It allowed two users to be scanned before entering their own booths, each equipped with scanned tea cups.  They could then see each other and their cups as holograms and change their scenery to wherever they’d decide.  The ability to see each other’s drink was key – beyond heightening the feeling of presence or enabling interactions like “cheersing”, it would set the tone for a communication based on a fundamental and cross-cultural experience of having tea, or coffee, or a drink of any kind with someone.

Based on this initial inspiration, for my culinary physics final, I want to explore social and cultural constructs around consuming food and beverages based on a shared assumption of reality. By taking a simple interaction of sharing a drink with another, I’m interested in shifting the context of reality by changing the setting to a virtual environment in order to see if any sense of social cohesion could carry over.

Live Web midterm idea

I want to explore concepts we touched upon with the Citizens app, where anyone can stream out their perspective and essentially become a camera.  To take it a step further, users who are following someone else’s stream would be able to affect the streamer’s environment through sending a message, ideally by voicing their message in a localized space in a virtual environment. In order to do this, I plan on using Twitch to stream out a video feed of a VR headset, and use the Twitch API with Socket.io for Unity.  The environment will be a simple space with a mirror in it, and the streaming user in the headset (most likely myself), will embody an avatar, seen through the mirror.

key things I’m going for:

  • a sense of copresence virtually through voice
  • share someone’s virtual perspective synchronously and be able to affect that environment via text to speech
  • explore any behavioral shifts that might occur from having chat texts voiced out loud

final idea

For my avatar final, I would like to explore a pipeline that’s traditionally used for gamers to stream their experience, in order to allow people I know in my real social life to control my behavior in a virtual social space.  I’m interested in fostering a collaborative and collective experience through one avatar, as well as what it means to be able to switch between experiences to follow.

I have been exploring some twitch feeds of VRChat users and find the interaction between the streamer / user and Twitch “followers” fascinating. The Twitch follower can be both a passive observer, as well as as a potential active participant.  The streamer is always notified of any new followers or comments they receive and can choose to acknowledge them.

I’d like to upload an avatar to either the High Fidelity or VRChat platforms, and invite my friends to participate in a time-controlled event (or series of events) to experience my field of view and control my actions. Ideally I would like to have another friend in the virtual space to interact with, in order to heighten the level of immersion for the followers- and as a bonus, switch between my feed and my friend’s.  I will view any incoming twitch messages on my HMD and either respond or react to them accordingly.

It’s important to me that this interaction takes place within my existing social circle, in order to contain the experience somewhat and hopefully make it more cohesive, even if this means far less followers (I’m ok with this experiment just involving even one twitch follower).