"a true messenger that live updates profile videos every time I type a message"

 

Motalk

Team work with Bryan Hsu (role: Art director and Full-stack developer) | Live Web with Professor Shawn Van Every | Dec 2017

Mobile Development | Interaction Design

Tools: JavaScript, jQuery, HTML, CSS, Socket.IO, WebRTC API and Digital Ocean

 

 

Synopsis

"Motalk" (Motion+Talk) is a true messenger. It lively updates profile videos every time I type a message, so I can’t hide my facial expressions when I am chatting with my friends. When I start typing a letter in the text input box, it records a video of me  until I press enter with the front camera of my mobile device, or a web cam on my computer. It instantly updates my profile into the video it just recorded. Live updates of my profile video prevents me from faking the profile picture (no more old or photoshopped photos!), by which enables representing the true essence of a 'profile'. Also, there's no need to use emojis anymore because I can simply express my emotions by facial expressions!

motalkmockup.png
 

This app was an experimental Socket.IO study done in "Live Web" class at ITP with Bryan Hsu, and the main goal was to make a fully functional quick prototype. We worked from ideation to front-end development and back-end development for sleepless full three days, and it was a big hit at the final presentation. People loved it because they could go silly with their gestures, texting with their friends during class (without being caught by the teacher). It was personally a successful work because I could build everything 'from the scratch' as a full stack developer, and every step taught me a lot from meticulously working on responsive CSS to live streaming videos to communicating with the server. 

full-stack.jpg
 

Ideation

 
IMG_6529.jpeg
IMG_6537.JPG
 

Wireframe

Screen 1 [Name] 

Write my name and press enter to go to the chatroom. The background big circle screen is a preview of my face suggesting that it is a video chat room. 

Screen 2 [Chatroom] 

Start chatting! A series of profile videos of me and my friends will play in loop in the center, making it into an interesting and dynamic look. Lower left side circle is a preview of my face playing all the time, and the lower right side circle is a preview of a silent movie that is made by cumulating all the profile videos. 

Screen 3 [Silent Movie] 

If I click the lower right side circle, it will expand and show the silent movie with profile videos and texts from my conversation. The texts act as subtitles. I can go back to the chatroom if I click anywhere on the big circle screen. 

 

Development

Client-side JavaScript (explaining only main parts)

  • WebRTC enables web browsers with Real-Time Communications (RTC) of video, audio and data via simple JavaScript APIs.

  • Get video elements, load and play.

  • if I start typing in the input box, start recording a profile video.

  • If I press enter, stop recording and send message and the profile video to server. 

  • Setup media recorder.

  • Get live stream videos and display.

  • When stopping the media recorder, save the data as a blob and send it to the server as a buffer array.

  • Get message content and make an object with user's name, message content, video and whether it is local (right) or from the server (left).

  • Send the object to the server with "socket.emit()".

  • Reset everything.

  • Push every video to the movie chunk and also send to the server.

  • Make a message box div containing user's name, message content, and a profile video.

  • Convert the buffer array back into a blob.

  • Get video URL and play the video.

  • Scroll to bottom of message container every time there's a new message.

  • Receive message data from the server.

  • Receive movie chunk from the server and play preview of the silent movie on the lower right side circle. 

 

Server-Side JavaScript (explaining only main parts)

  • WebRTC security offers end-to-end encryption between peers so requires 'https' to access.

  • Use filesystem module.

  • Need a key and a certificate for peer-to-peer SSL connection. All the communications are encrypted.

  • WebSockets work with the HTTP server.

  • When a client-side user emits data, data comes into the server.

  • Send it to all of the clients using "socket.broadcast.emit()".

  • Send accumulated movie chunk to me.

 

CSS (only some parts)

It is responsive so it works in every mobile devices as well as computers. It was a lot of math of meticulously playing with pixels.  

Screen Shot 2018-04-06 at 6.15.19 AM.png
 

HTML (some parts)

Socket.IO script is a must to communicate with server. For now, Motalk is composed of 3 screens.

 

Process Demo

Me talking to other me on separate windows. I was testing the basic chat functions to see if data was being correctly transferred. It was successful! But it was kind of slow. 

 

Testing a movie chunk on the lower right circle screen. First it had a lot of bugs because it's playing so many video clips in a row. This version is kind of flickering, but they are in the right order. 

 

Movie chunks of all the profile videos played on movie circle screen. It is a silent film with texts acting as subtitles. Fun to watch!

 
 

Next Step

We will definitely make Motalk prettier! We're going to design it with more engaging graphics and smoother user flow. Finally we plan to upload this to the app store. Personally I want to try to re-develop this into an iOS app with Swift!