Crashing the Gates

By | Origins | No Comments

Gonzo: I’m going to Bombay, India, to become a movie star!

Fozzie Bear: You don’t go to Bombay to become a movie star. You go where we’re going: Hollywood!

Gonzo: Sure, if you want to do it the easy way!

– The Muppet Movie


Press your nose against a window long enough, and you’ll start to think about hurling a hammer through it. In the summer of 2003, I knew where I wanted to go, but there was no clear way to get there. The kind of work that James Cameron, George Lucas, and Peter Jackson were doing was fascinating, but prior knowledge of how a studio worked made me wary of the standard movie industry path. Painful experience had taught me that great achievements inside someone else’s company were not always recognized, and I was too proud to undergo the standard entertainment initiation of scurrying after coffee. I wanted to spend my time actually creating instead of schlepping.

Scrambling around on my own projects was equally frustrating. Helping a friend on his student film project had triggered my own interest in movies, and the low cost digital video revolution was in full swing. Trying to make something actually look good while on a micro budget, however, was an exercise in awkwardness. I knew exactly what I was aiming for — the list of movies with otherworldly visuals and good storytelling is not lengthy — but between my camcorder footage and ‘The Lord of the Rings’ lay a rather imposing gulf.

In short, I wanted to do cinematic works of epic scope while retaining creative and financial control. The astute reader may note that these are the rarest perks in the industry; the group of people that have achieved this, aside from the above trio, is another short list.

I also knew that these three had run into large problems, despite their success. To construct the exotic worlds of their movies, each had created giant visual effects organizations, which rapidly became monsters with huge payrolls that had to be fed or perish. Create a service based business, and you will end up serving the business. I needed something better than this — some sort of magical machine that would do the grunt work of creating stunning cinematic backgrounds, without generating a massive financial burn rate.

I was far from the first person to wish for such a system. Nearly a billion dollars had been previously spent in attempting to achieve this in the entertainment industry, with no commercial success stories. Fortunately, I was aided by two tremendous resources: naivety and hubris. I did not know of the failures that littered the field, and was fresh from victory on my last project (the iRobot Roomba.) I modestly planned to crack the major unsolved problems in the visual effects industry over the next couple of years; it was time to show these pixel pushers a thing or two.

Origin Stories

Since I had put together a couple of digital editing systems, I understood something of how images were handled on a computer. Having learned the basics of visual effects, I also realized that the vast majority of the shots were highly repetitive. After completing two shots that took me the better part of a week each, and mentally multiplying this by the 1500 shots that an average movie required, it became clear that spending thirty years on one movie was a poor plan. Making an automated VFX shot processor was the clear answer.

I knew that the system had to be based on digital cameras; the costs of shooting 35mm put it out of my reach. I wanted freeform camera motion tracking, without depending on encoders to track the camera. Somewhere in there, we had to solve lens optics.

The original plan was to simply automate existing VFX pipelines — record camera moves, then run them back through a post production rendering farm. This safe and logical step was tossed out the day I ran into Bill Warner, the co-founder of Avid Technology. “Why would you do it that way?” was his typically direct response. “It should be all in real time . . like a 3D Ultimatte.”

He forgot that statement about 5 minutes after making it, but I didn’t, and that weekend the company’s focus changed permanently.  After spending considerable time researching existing real time software systems, I came to the realization that no existing system could achieve our targets. Fortunately, at this point Phil Mass (the software lead on the Roomba) had joined the team, and the die was cast.

We tore into the task of building Previzion from scratch, bridging the flexibility of the visual effects pipeline with the need for immediate, instant feedback on a production stage. We had to break down each process systematically — keying, tracking, rendering, compositing, color correction, and data logging — and engineer ways of solving all of the problems with real time processing techniques from our years in robotics. We built our own inertial measurement systems to solve tracking and sync, engineered complete lens calibration and distortion rendering systems, and wrote a full compositing stack capable of rendering HD frames in milliseconds.

If the theory has difficult, the practice was gruelling. The company moved to Los Angeles, and was immediately immersed in the high pressure production world. We operated on numerous soundstages, and adapted the system daily to the needs of features, episodic television, soap operas, and commercials over a span of years. What was supposed to be a three year development project stretched into eight, as we discovered that visual effects balances warily on the edges of four or five technical disciplines. The pixels pushed back.

Why We Write

A great software system is never ‘finished’; the continuous evolution of the code is what makes it improve year after year. At some point, however, it becomes good enough to get the job done, and to bring a new way of working to the process of storytelling. This is already happening, and the rapidly increasing list of Previzion credits tells its own tale.

In the process of working on these shows, however, we found an intense need for additional knowledge: how to build backgrounds, how to integrate the system into a show, how to operate it, and how best to use the potential of this new technology.

My education in this field started with the same frustrations. When I started, there were few guides outside of a VFX facility on how to really handle production level keying. Reading ‘expert’ mailing lists revealed a wealth of conflicting ideas: What green? What blue? What keyer? What happens when the standard methods don’t work? How do you handle lens distortion? Vague, hand-waving arguments were generally the result. The standard fallback response, ‘Consult a qualified professional’, was the single most frequent and frustrating response I heard on these forums.

We did eventually understand why. When VFX shots are set up correctly, in a certain fairly narrow window of greenscreen color and illumination, with accurate logging of camera parameters, a top flight keyer, and well matched lighting between foreground and background, the images will lock together easily and be convincing. Step outside of that narrow sandbox, however, and the problems become nearly insoluble without hurling money at them.

Of course, without a real time system on set to check your work, the only immediate reference is previous experience. Very, very few people in the industry ever go through the complete circle, from camera setup to greenscreen lighting to shooting to keying to compositing, and it is the closing of this loop that leads to comprehension of how the various factors in the process all come together. Most of the people who did understand this were inside dedicated facilities that benefited from secrecy.

Our goal with the Previzion system, the Lightcraft web site, and this weblog is simply to tell people what can be done with the systems, show them how to do it, and lay out the future of where this could go. It’s to make clear that path that was so dim eight years ago.

Clear does not mean easy; this is a powerful system for people who demand professional results. There is no ‘1-click’ solution for visual effects. The process can, however, be made straightforward. The original goal — to make it possible for a small team to make shows that are visually competitive with much higher budgeted work — is now possible, and is being done on a daily basis.

Deus Ex Machina

What began as my personal end run around Byzantine studio politics has somewhat unexpectedly been adopted as the standard on set visual effects production tool in Hollywood and the rest of the world. The entertainment industry turned out to be facing the exact same dilemma that I had — large ambitions, limited budgets.

The systems have changed as well. Over time, due to our rapid response to customer requests, the technology has gone beyond anything we could have imagined at the start of the project, and is now triggering a creative revolution. A stereoscopic feature film, ‘Walking with Dinosaurs 3D’, is using Previzion to track stereo camera rigs flying over herds of virtual dinosaurs with GNSS satellites. The episodic sci-fi show ‘V’ had VFX shot counts that exceed most VFX-centered movies, and two of the most visually striking television pilots of 2011, involving both epic fantasy and period piece storytelling, were both shot using Previzion. Brazilian telenovelas are finishing hours of real time composited backgrounds, and the production world is slowly discovering what is truly possible.

In that discovery lies a chance.

The troubled state of the industry is well known; VFX companies going under, strange accounting practices, guild strikes, and lawsuits over royalty payments are becoming frequent events. Most of these problems are artifacts of the origins of the motion picture industry. I’ll go into this in more detail in upcoming posts, but a brave(r) new world awaits.

Since the days of nitrocellulose stock, filmmakers have faced a similar set of trials. With these new tools, the trials will not disappear, but can at least be made different. We have the rare chance to remake part of the industry in a new image, and to forge a space where the best artistic work is both protected and rewarded.

In this column, I wish to write of what has been, what is, and what will be in this new arena, and to shine a sometimes-flickering light down the untrodden path that I set out upon eight years ago.

I write that others of kindred spirit may find common ground.

I write to create the entertainment industry that I want to work in.

We have opened a window that most thought impossibly barred. I hope that you will join us for the view.