quick and dirty demonstration of the state of things so far:

Unity3D camera being controlled by Previzion, respecting position, orientation, zoom and focus. Live camera feed is chroma keyed using PixelConduit, streamed via Syphon to Unity3D where it is composited with the virtual environment.

Enjoyed a 3 day guest lecture at INSEEC London where students got a crash course in Unity, designed their own level and on the last day got to explore it on the Oculus Rift.

Progress : realtime chroma keying of video in Pixelconduit, streaming to Unity via Syphon

Implemented & tested successfully:

-camera position

-camera rotation

-lens focus

-streaming external video into 3D environment

Implemented, not tested:

-camera zoom

Still to implement

-chroma keying of incoming camera footage

Since five years the Netherlands Film Academy uses the real time game engine Unity3D for all interactive courses on the Interactive Media & Visual Effects department. Unity has a relatively low learning curve and allows our students to make visually outstanding interactive experiences.

A couple of years ago our department acquired a Lightcraft Previzion virtual camera system that can show real actors in a virtual environment in camera in real time. This technique, called Virtual Production, has created a revolution in the film industry.

Previzion uses a proprietary software to pull this off and this has some disadvantages. First of all we have found the workflow importing assets from Maya sub optimal and, more importantly, the closed nature of the software does not allow us to add any functionality.

Luckily Previzion has a protocol to send it’s information to 3rd party application so I started researching if we could make a connection with Unity. Early 2013 we successfully tested an early prototype where most of the information from Previzion was visualised in Unity in real time. 

After this successful test we realised the combination of Previzion hardware and Unity software has more potential uses than just previsualisation. Combined with 3D sensors such as Kinect, PrimeSense and Leap Motion it should be possible for an actor to actually interact with a virtual environment.

This blog will document our research of realtime technology in cinema production as well as provide links to new developments that are relevant to our research