First test of real-time facial capture streaming from Faceshift into Unity3D

CONSTRUCT is a Sci-Fi short film advancing the art of filmmaking, VFX and virtual production.

S4855: Creating CONSTRUCT: How NVIDIA GPUs are Defining a New Filmmaking Paradigm

Testing the new Mac Pro to see if everything works in our setup. 

Blackmagic 4k capture > PixelConduit > Syphon > Unity > PixelConduit all working nicely. 

Tested Unity with a large particle system spread over all 6 cores. Also worked nicely.

Quick video of the setup we used at the open day on jan 18th. 

Previzion virtual camera system setup in classroom with a greenscreen so visitors could see themselves composited inside a virtual environment. As an added bonus, visitors could control the big robot arm by moving their hands above a Leap Motion 3D sensor

Today we set up our new motion capture system for the first time in Studio 1. We setup an array of 12 cameras in a circle, Calibration worked the first time, which looks promising. Unfortunately our software is a bit older than expected. Tomorrow I will test if my Unity prototype interface is receiving data from the motion capture system and I have a studio booked in April for a full week of testing.

While developing the Unity interface for the OptiTrack Arena motion capture system I discovered a neat way to listen to network traffic in a seperate processor thread. Hopefully this solves the lag/stutter we experienced controlling Unity from the Previzion virtual camera system.

Requested studio time to do actual testing with OptiTrack and diving into its docs in preparation.

Seems to be working ON PAPER. Need to test in the studio next month with the actual 12 cam setup to be sure.

Next up is adding motion capture to the system. With no space to setup our 12 camera OptiTrack system yet, and considering integrating the new Kinect with Unity is going to take some time to develop, I dug out a good old Kinect 1 and used OpenNI to control a humanoid character in Unity in real-time. 

Meanwhile I’m diving into the OptiTrack data streaming protocol to start developing a Unity integration with a proper mocap system

Made a demo for the open day on the 17th, demonstrating the Previzion virtual camera system in combination with Unity, compositing visitors into the virtual environment in real-time, while controlling a virtual robot arm using a Leap Motion sensor. Due to a misunderstanding the camera wasn’t calibrated so we did not get a perfect match between the real and the virtual scene. Also the Inverse Kinematics script I came up with could use some love. 

I’ll post some footage when I find some time to edit it.