Last week, in the first part of our series, I gave you an overview of how the VFX industry has typically worked, and explored some of the logistical considerations surrounding the implementation of remote workflows.
I also covered the broader strokes of how a business that’s historically depended on brick-and-mortar facilities has had to adapt in terms of communication, oversight, and creativity.
In today’s article, we’ll look at some of the technical considerations for remote VFX workflows. We’ll start by examining two of the remote VFX workflows that are currently in use, and outline their advantages and drawbacks. Finally, in part three (next week), we’ll explore how cloud technology is accelerating the move to remote workflows.
The common solutions
The remote work solution for VFX has, until recently, boiled down to two main approaches—artists using local machines (a collection of distributed artists working and rendering on their own home computers) or remote machines (artists working on home computers or monitor-keyboard systems with the actual processing and rendering occurring on a remote machine).
A box under the desk
Dating back to the early days of digital VFX, this approach has been used for years by independent artists—even at brick-and-mortar facilities. Artists worked on visual effects tasks such as rotoscoping and compositing, and when they completed a shot, they rendered the final product on that same computer.
If multiple artists were working on the same project on different machines, files were shared between them via sneakernet; literally walking hard drives between machines.
As digital VFX systems became more powerful and rendering tasks heavier, render farms and more advanced systems were developed that allowed the artists to keep working on their shots locally while other shots or tasks were sent to a render farm or other workstations for processing.
A local machine running independently is fairly easy to set up. Each artist has their own machine (or machines, if they have more than one for rendering purposes). They install the software tools they need on their primary computer—3D applications for modeling, animating, and rendering, and 2D tools for roto or paint tasks and compositing—depending on their area(s) of expertise and whether they handle all aspects of the process or specialize in one or two.
The cutting room may be onsite in a brick-and-mortar setup, but is equally likely to be at a different location. The editor is responsible for feeding the VFX artists the locked sequences or shots, providing them as “turnovers” of original camera footage at a specified resolution.
In a sneakernet setup, hard drives are carried or shipped to the artist, especially if the amount of data is large. But as file-sharing services have become a viable option, smaller quantities of data are frequently accessed online.
As we discussed in part one, VFX processes are generally iterative, with the artist sending versions to the supervisor or director for review and feedback using the same file transfer process. Often, a shot or sequence requires multiple specialists (rotoscope, matchmoving, animation, lighting, compositing, etc.) all of whom need to download the latest versions of the shots or assets.
Depending on what kind of feedback is required, they may need to download fully rendered shots, but sometimes can view lower resolution proxies to save upload and download times.
Not as simple as it looks
If the artists already have machines, it’s easy to get them up and running, especially if they’re a “one-man band” who can do entire shots or sequences without contributions from other artists. And if they already have the necessary software installed, the costs to the effects studio are negligible. Cloud-based storage or file sharing adds only minimal expense.
But that’s the very best case scenario for the artist, production, or studio. There are many other questions and issues that arise, in terms of expense, speed (and the logistics surrounding it), and security.
First, who is covering the cost of the local machine, software, and internet connection? Is it provided by the visual effects company or considered a cost for the artist to bear—or can they bill for it?
Additionally, all the required communications software needs to be installed, or be available as a web app, along with a webcam for video conferencing. Who bears those costs?
The adage fast, cheap, good—pick two comes into play here on a couple of levels.
Home high-speed internet can fluctuate and seldom reaches the advertised speed, at least in most of the U.S. download speeds of 50-100 Mbps are the minimum required to even consider doing remote work. Most home internet is also asymmetrical, meaning that the upload speeds are frequently 5-10 times slower than the download.
This is an important consideration because, in order for a supervisor or director to review (or for the artist to exchange) a rendered file, the shot must be downloaded and uploaded. And that might take several hours, depending on the connection speed and the resolution and duration of the shot.
This means that if a director needs to review a shot at a specific time, the artist has to include upload and download times in order to back into a deadline—and allow for the possibility that an upload or download could take longer than expected or need to be restarted.
So, if the workflow requires an artist to have a faster internet connection to speed up the process, who pays for that?
Time for an upgrade?
Next, the local machine must be powerful enough to accomplish the work. Disk space, amount of memory, and type of GPU are critical in terms of both working and rendering speeds.
During rendering, the machine may become less responsive to working, which means that in some cases artists may need to have another machine (or more) to allow them to share or offload rendering. And extra hard drives may be required for greater storage capacity.
Similarly, software needs to be licensed and installed on the local machine, and the versions have to match the rest of the team’s and remain updated accordingly.
Which brings us to the issue of sharing files and assets among artists working on local machines in different locations.
It’s easy to control file sharing if everything is on a hard drive on a local machine. But, if artists working on local systems are accessing assets that are not located on a local drive and are being shared online or via the cloud, complications can arise.
Like herding cats
Typically a coordinator or asset wrangler monitors the shared file process to ensure that users are downloading the correct versions of shots and that no one else might be manipulating the same files at the same time. But as the number of individual artists rises and the number of files increases, it can become daunting to keep track of who has which files.
Custom local scripts can be created to automatically download or upload files and to monitor when files appear. The catch, however, is that many file sharing systems operate in a sync mode as a default. This creates a ‘drive’ or ‘folder’ on the local machine that matches the cloud version, and any changes that are made locally are synced to the cloud version. Changes ripple to anyone who shares this same folder.
This creates two problems unless the settings are carefully managed.
- Local storage is being used up by any files in that folder even if they don’t relate to that artist or to the current shot—and disk space is already at a premium.
- If anyone sharing this same ‘folder’ changes or deletes a file, then the copy on the cloud, along with everyone synced to that same folder, will have their files changed or deleted. It’s not unusual for an artist to want to delete files when they are done with them, not realizing that those files will also be deleted for anyone else in the pipeline who might be working on them.
Last, but perhaps most importantly, the most significant drawback to distributed teams using local computers is the relative immaturity of robust security procedures. In addition to the shots now existing in whole on multiple home machines, the studios must depend on artists to securely delete the files when they’ve completed their work.
Many consumer-oriented file sharing services don’t necessarily provide the most secure storage, either. Historically, lack of security is the biggest reason why most larger studios have resisted using off-premises workflows. We’ll talk more about security later in the article.
A box somewhere else
A remote machine solution is typically used by medium- to large-scale visual effects companies (although there are exceptions).
In this scenario, the artist takes home a computer or special terminal that accesses a remote machine. The remote machine may be a workstation or render farm at a physical location, and the original assets live there rather than on the artist’s local computer or on individual hard drives.
Artists log onto the system through a VPN (Virtual Private Network) to the remote machine. The view from the remote machine is streamed to the user and all input from the user’s keyboard and tablet is sent back to the remote machine.
From the user’s viewpoint, they fire up their software application and work as they normally would at the facility. No files are downloaded or uploaded. They log off at the end of their day and when they log on the next day everything is exactly how they left it.
As simple as it looks
Unlike the local machine setup, there are numerous (somewhat obvious) advantages to the remote machine setup.
- First, there’s no need for the artist to upload or download files, so any changes or updates are immediately viewable by the director or others. Files are also immediately usable by other artists working on the same shot, just as if they were physically at a facility.
- Because the data files are all available in one place, pipeline tools can automate steps in the process. Databases can track changes and versions, and record and control access to different files for different artists.
- Software licensing is handled centrally and all software applications are readily available to artists and synchronized for versions. And remote systems allow memory, disk space, GPU and other features to be controlled, depending on the requirements.
- Scaling a production by bringing on new artists is much simpler when artists aren’t required to supply their own high-powered machines with compatible software already installed. Artists can also easily switch to different types of machines if necessary and if they are available.
- If a shot requires heavy and time-consuming rendering, the work can be sent to a render farm or to multiple machines as controlled by render queue software.
- This system is more secure because the files are held in a secure central location, which means that the individual artists can neither distribute shots nor are they responsible for deleting files.
Before you rush off…
In ways similar to the local machine setup, there are logistical issues that must be considered.
Again, the artist’s home internet speed has to be robust enough to acquire fast image updates from the remote machine with minimal lag time. But, while the 50-100 Mbps speed is the minimum, speeds beyond that may not provide much benefit. And, again, communications software and a webcam for video conferencing are necessary.
The artist’s machine also has to be able to connect through the VPN and display a high-enough quality image. Some solutions (like Teradici) are available for just this purpose.
Beyond that, there are additional considerations and costs for the VFX studio in a remote machine setup.
- The VFX studio must invest in configuring computers to accommodate a remote workflow, in addition to supplying the remote boxes for each artist. Pipeline software may need to be developed or may have to be modified to accommodate this approach, which means an additional expenditure, at least initially.
- There must be smooth and reliable VPN access, which involves proper setup and investment. Even then, typically there’s a limit to the number of artists that can VPN in at the same time, based on bandwidth and connections. (Anecdotally, one facility I spoke with runs day and night shifts to work around the limits of their VPN services.)
- One big concern for artists is latency. Artists who are used to working on a local machine can make changes and see the results immediately. But imagine a scenario where an artist is working in Photoshop to adjust something and experiences a delay with every touch of the slider. Sluggish responses can make the process not only more time consuming and frustrating, but can also result in errors.
Under lock and key
It’s impossible to discuss the challenges of moving to a remote workflow without discussing security. It’s a big concern for every client, but when you’re working with large tentpole movies, security is the top concern. From securing the facilities’ premises to the digital files (and transmission thereof), allowing off-premises work to continue comes with heavy considerations.
The MPA (Motion Picture Association) mandates and manages strict content security protocols on behalf of its members, which includes all the major studios.
Their security program, designed to protect intellectual property from leaks, issues specific best practices to vendors, facilities, and services, who must prove their compliance in order to be certified.
Since 2018, the MPAA has joined with the Content Delivery and Security Association (CDSA) to create the TPN (Trusted Partners Network) to boost security initiatives for film and television. TPN certification is voluntary, but ensures that the certified entity is maintaining the highest possible standards for security.
VFX facilities often place cameras to monitor doors and issue security cards to limit and record who has access to various parts of the facility.
Extensive procedures are implemented to ensure that no assets—from concept artwork to completed shots— are leaked to the public before the film is released. And file transfers to or from a facility are encrypted or shared via a private link that is password protected. Viewing copies of work-in-progress generally also have visual watermarks to ensure that the material cannot be shared and, should it leak, can be easily traced to its point of origin.
Given that degree of security consciousness, opening up to remote work is a big step. Studios have had to make a tradeoff of putting the work on hold until everything returns to ‘normal’ or getting the work done in order to release the film when and how they wish. The large outlay of production funds on which studios pay interest has prompted them to accept the higher risk in order to allow remote VFX work.
This requires a certain amount of trust—in addition to issuing NDAs and whatever other technical processes they can implement to secure workstations. It’s why the remote machine approach is currently the preferred method for larger studios; it’s far easier to control file sharing and prevents assets from living on difficult-to-secure home machines.
Look to the cloud
Although both of these solutions we’ve discussed can incorporate cloud technology as a part of the workflow, it’s a much larger topic, and one we’ll save for Part 3 of this series. But it’s enough to say that remote cloud workflows solve many challenges from ease of implementation to cost, efficiency, and security.
