Making the Move to a Remote VFX Workflow: Part 3

In Part 2 of our series covering remote VFX workflows, I took you through the two most commonly used remote solutions. The first is distributed artists using their own local machines with a sneakernet setup. And the second is artists working on remote machines that are connected to the facility and/or render farm.

For those of you who’ve been wondering when I’d talk about cloud-based solutions, your patience will be rewarded. Today we’ll explore the numerous advantages of cloud-based workflows for VFX. And because remote workflows are proving to be advantageous for both the VFX facilities and for the artists, I’ll also outline how the future of the VFX industry might change going forward.

An industry on the edge

Anyone who’s worked in or closely followed the VFX industry over the past couple of decades has seen how difficult it can be to keep a facility financially healthy. I’ve written extensively on the subject of how delicate a balance it is for a facility to stay in business and what kind of impact that’s had on the community of artists.

There are numerous factors for why VFX, in particular, seems to be such a vulnerable sector. But, if you look at some of the inner workings, a pattern emerges.

First, the VFX industry is notorious for being squeezed by studios to give them the lowest price for the best possible results. Unless a studio has an ongoing relationship with a particular VFX company (think Disney with Industrial Light & Magic), it’s likely that several companies will compete for the privilege of working on the higher budget, higher profile projects. Competitive bidding can lead to facilities’ trimming their bids to the barest minimum, enough to just cover their overhead—never mind being profitable.

Beyond that, VFX projects are generally “flat bid,” meaning that there’s an agreed-upon scope of work that’s been outlined in a contract. In reality, it’s the rare VFX project that doesn’t experience “scope creep,” in which additional tasks or revisions are added over the course of the work.

Producers “can” ask for overages, and often do. But just as often, when a project is occupying a facility for months (or even years) at a time at costs of hundreds of thousands to millions of dollars, a producer may be loath to nickel and dime the studio. Their hope is that over the duration of the project, other efficiencies will offset the overages.

Unfortunately, that’s equally rare. Over the years, we’ve seen the demise of successful companies, perhaps most notably Rhythm & Hues, who declared bankruptcy just after winning the Oscar for The Life of Pi.

Historically, VFX companies have had to invest heavily in on-premises computing, which creates high overhead both in terms of hardware costs and the square footage necessary to house it.

Compounding that, the larger VFX houses have needed to be located near the studios in production hubs like Los Angeles and London, or where large subsidies are offered (like in Vancouver), where the cost of real estate is already high.

So let’s see how shifting to a cloud-based workflow could open up opportunities for a more sustainable model.

Heading to the cloud

There’s been an ongoing debate about the pros and cons of on-site machines versus cloud services. I’ll summarize here, but have included links to a number of articles at the end of this one, if you want to delve further.

Many industry pros still feel the cloud is too expensive, especially if they have already invested in a local-machine and render farm solution—as is the case with a number of the largest and most established VFX facilities.

The simplest approach for them is to continue to use the machines they have—at least until they need greater capacity or their technology becomes obsolete. At that point, they’ll have to evaluate what the cost ramifications are for building out more on-premises solutions versus moving operations to the cloud.

First, they’ll need to break down all the costs of their on-premises servers and related infrastructure, backups, IT support, power, replacements, etc. Once they know their current costs, they’ll want to start by determining how much development work needs to be done to get functional cloud services—and how long it may take. This requires a deep analysis of everything necessary to make a change of this magnitude—which may involve consulting with a cloud specialist or leveraging an existing service.

The many costs that need to be taken into consideration include:

Transferring all data, including assets and videos, to the cloud, as well as the cost of the labor to do the initial uploading and organizing.
Storage types and costs on the cloud, both short and long term, as well as databases and other cloud services. Note that storage costs can vary depending on the speed, so using different levels of storage for shots that are in production versus those that can be archived can help reduce costs.
Costs of downloading data, if required.

There are also logistical considerations, like what kinds of tools and protocols will be used to monitor cloud usage. For example, if machines are accidentally left running and you’re paying on a usage basis, failing to discover it for days or weeks could become very expensive.

But for newer VFX companies, such as Untold Studios, who are able to design their workflows from the ground up (or from the cloud down), the advantages are clear:

There is little to no up-front investment in machines.
Operating expenses can be written off as a business expense rather than having to deal with depreciation for capital expenses over several years. And there’s no need to pay for expensive real estate or square footage simply to hold machines.
Less IT support is required to maintain or upgrade machines (like updating GPUs or storage).
There’s no limit to the number of users.
A full range of “machines” are accessible at different price points. Artists doing simple tasks can use a simple machine with minimal resources. But for tasks such as huge simulations that require heavy render processes, they can easily select a machine with massive RAM, disk space, and multiple GPUs. The facility also has the ability to quickly scale according to the project’s needs, spinning up 100 or 1000 machines for heavy renders.
Costs are billed on a per-hour or per-minute usage, which means that they’re only paying for what they use rather than having machines idling when not in use. For example, AWS computing allows you to reserve machines at a cost (if you know you’ll use them) but you can also get machines as needed. You can even bid on machines that are not in use and pay less, with the understanding that you may be “bumped” off of them on short notice.
While machine learning and AI algorithms are being folded into desktop apps, the cloud provides the scale, speed, and flexibility to put it to use on large productions for customizing workflows and automating processes.
As cloud computing becomes more available, the usage costs continue to decrease.

Moreover, while R&D teams creating proprietary software still exist in some VFX companies, the market has filled the many gaps with specialized and advanced tools. Developers are creating toolsets that can be accessed in the cloud and that take advantage of Docker/Kubernetes, batch processing, and other cloud technologies.

From the artists’ standpoint, accessing VFX tools on the cloud is no different than accessing a remote company machine (assuming they’re properly set up). And both mimic working locally, except for the login process. Artists from all over the world can find their closest cloud portal in order to get the best speeds, giving them super fast data connections to the files they need.

Security

Here’s the good news: Security on the cloud is better than at a typical facility equipped with on-premises machines.

The cloud companies’ entire focus is on security and management of data, and they employ multiple layers of security. Access to data is very carefully controlled and user access is very specifically defined and monitored.

And since the data is physically in a different location, likely even in a different city or state, neither production nor IT staff have any physical access to drives or machines from which to copy assets. When you consider that a staggering 77 percent of leaks of motion picture content are attributable to insiders, the extra security that the cloud provides can be significant.

If you recall from last week’s installment, we discussed the MPA and TPN certifications, which ensure the highest levels of security over content and data. One of the reasons I’ve used Frame.io is because they carry both.

Not all silver linings

So what about the drawbacks of using cloud-based solutions? Mostly they have to do with a current lack of standardization.

Different cloud services offer their own complement of tools, machines, costs, naming conventions, structures, and programmable access. As you assess cloud vendors, you’ll need to compare things like the range and costs of machines and storage, the tools they have available, ease of development on top of what they supply, support for VFX packages, add on services, admin tools, and different levels of security.

Also, many of the cloud solutions are designed to be customized by the company that uses them, which means that there’s an initial investment of time and money for implementing a fully efficient cloud-based VFX pipeline.

There are, however, companies working on tools and processes to fill in the gaps in order to help a VFX facility get up and running quickly. Arch Platform, which acts as a middle man for the VFX cloud; Conductor, Azure, and Zync, who provide cloud rendering services; AWS, who owns ThinkBox Deadline render queue software and provides start-up guides.

The overall advantages of remote work

Regardless of the solution—local, remote, or cloud—there are advantages for both VFX companies and artists to continue working remotely, even if and when it is safe to go back to on-premises work.

For companies

Anecdotal reviews already indicate that there are unexpected advantages for those companies that have made a full switch.

In the case of Zoic Studios, who switched their 300+ person/three-city operation to a fully remote model using a hybrid-cloud workflow, they found that they were as productive as when they were in their brick-and-mortar facilities. But unexpectedly, they also found that they saved on overtime costs and downtime.

Why? Because remote setups make it easy to access artists from anywhere. Zoic was able to assign work to available artists in any location, versus restricting artist “casting” to those in a particular location.

And then there’s the time advantage. When a VFX house ceases to be dependent on having artists on-premises, it’s possible to cast a far wider net for talent, which translates to having the ability to quickly build a team. Plus, with the potential to hire artists in different time zones, the possibilities for round-the-clock work expand, as well.

Remote workflows also enable a VFX company to scale up without worrying about physical capacity.

There are numerous projects over the years that have had to hire more people than originally budgeted. This results in squeezing artists into spaces that weren’t designed to comfortably accommodate them—or the extra equipment required.

For VFX workers

VFX artists, whose workdays are notoriously long, benefit in many ways, beginning with travel time to and from work. According to a recent survey conducted by Frame.io, remote workers in post-production saved an average of 53 minutes a day on commuting—many of them using that time to achieve a better work-life balance.

We’ve already talked about the cost savings for VFX companies who could reduce their overhead by occupying smaller office spaces not necessarily in expensive production hubs. But what this can mean for VFX workers could be literally life-changing.

Since the early 2000s, VFX companies have kept fewer employees in staff jobs and have hired artists on a project basis. With facilities scattered across the world from Los Angeles to Singapore to New Zealand, artists have become digital nomads.

It’s become increasingly difficult for many artists to make a steady living in one location, to buy a house, to keep their kids in one school. They’ve had to choose between uprooting their families—or leaving them for extended periods. Moreover, not every VFX artist earns enough to afford a home in some of the more expensive production hubs like the Bay Area, Vancouver, London, and New York.

Working remotely means that artists could now live in any state or country with high-speed internet access, opening up the possibilities of living in far more affordable locations.

We’ve seen already that tech workers are leaving Silicon Valley for places like Oregon and Texas. The same could happen for VFX workers, who might finally be able to purchase a home and keep their kids in a nearby school. What’s more, they no longer have to move to a new location for a new project. The days of being a captive digital nomad could be over.

These, and other changes, could drastically improve VFX workers’ quality of life, and even open up new possibilities. As many people are aware, parents (especially those who are single parents) with young children have an especially challenging time working in VFX. The long, unpredictable hours, the demand for working weekends—remote work and more flexible hours makes it far more viable for them to be able to take on projects.

Likewise, It also opens up opportunities for talented artists who might have medical issues or disabilities that make it difficult for them to spend the 10-plus hours a day sitting at a desk.

The disadvantages of remote work

Of course, there are drawbacks and catches for both the VFX companies and artists, but many of those are a function of initial setup and the inevitable growing pains.

Teams who are used to working together and have a kind of shorthand for communicating can be more difficult to develop when people are no longer in the same location. It’s part of why ensuring clear communication among and between teams, and coordinating meetings and feedback—especially if people are working across different time zones—is extra important.

The same applies to tracking tasks and assets to ensure that artists have everything they need in order to keep working. And providing extra IT support for troubleshooting problems remotely and ensuring that remote servers are in sync is an additional consideration and potential expense.

But then there’s the more significant consideration surrounding film subsidies.

Subsidies exist in a number of countries and states, which carry stipulations around the location of the workers and the company. U.S. states spend over $1.5 billion on film subsidies, while British Columbia spends $500 million annually and covers over 50 percent of the salaries for VFX workers in B.C.

If remote VFX artists can be anywhere in the world, will the qualifications for subsidies compel them to be local?

From the VFX artists’ perspective, there’s the question of receiving benefits as a remote worker. Full-time, onsite employees may receive overtime pay, unemployment benefits, paid time off, and health care through their employer (although, in the case of some short-term projects, many of those benefits may not be provided).

Independent contractors, however, are generally ineligible for such benefits. If artists work from home, will facilities attempt to classify them as independent contractors to save on paying for benefits?

There are also the questions of productivity—working from home often means being distracted by family, roommates, or friends, who all think that because you’re home, you’re more available to them. There are also more opportunities to be distracted by what others around you are doing —watching videos, playing games, or doing chores around the house.

Conversely, because some people don’t have those kinds of distractions, they may find that they’re putting in more “chair time” than they had while onsite. They might be less likely to take breaks, move around, or stop working at the end of the day, as they try to squeeze in just one more change.

It’s a particular hazard when people are working across different time zones—some artists may have already put in several hours of work before a supervisor is up in the morning, and end up working hours beyond what their normal shift would be. The round-the-clock workflow, while an advantage for some, can be a liability for others.

It’s also difficult for some people to feel as creative or as motivated when they’re not working directly with teammates. As many creatives know, having artists in close proximity to one another helps them to see what others are doing and sparks ideas or inspiration.

And, finally, people are social creatures who generally benefit from work friendships and interaction. Working from home, especially for those who live alone, can be isolating. It’s part of the VFX company’s responsibility to keep employees feeling as though they’re still part of a team and have the support they need, both in terms of their work and their mental health.

As we talk about the downside of remote work, it’s worth considering that the immediate shift (both in VFX and other sectors of the industry) happened with insufficient lead time to put all the best practices in place. Workers were suddenly thrust into a whole new way of working, with their lives further complicated by the logistics around daily life. With their children staying home from school, needing to shop for food and cooking at home, and worrying about their and their family’s safety—all against the backdrop of the pervasive and unsettling news coverage—working from home became even more challenging.

Which is to say that as we have had time to adapt to working remotely, the experience under less stressful circumstances could potentially be more relaxed and agreeable for both companies and workers.

What lies ahead

There’s no doubt that the pandemic and quarantine have accelerated the need to implement remote VFX workflows. But what happens when it’s safe to work on-premises again?

Numerous questions arise: Will facilities bring everybody back into their buildings? Or is remote work here to stay? Will VFX companies prefer the lower overhead costs and new efficiencies over returning to business as usual? And will VFX crews prefer the potential for better work-life balance and permanent, affordable living?

At this stage, VFX companies should be looking 10 years into the future. How will the consumption of media change and how will that change the need for VFX? How will production change? The new technologies enabling virtual production, such as virtual sets, will have a huge impact on how VFX are handled. VR and AR technologies are currently in use to help scout and plan for VFX shoots.

And then, how will cloud-based workflows enable faster, less expensive, and less location-dependent productions? What will machine learning and AI bring to the process?

What new best practices need to be put in place? Will there be some kind of international directory of VFX artists who are available for remote work? If so, how would they be vetted?

My guesses:

Studios will want to revert back to self-contained companies as much as possible, but the pandemic (along with efforts to slow climate change) will require a reduction in travel.

I suspect we may see a hybrid of people working together in the office along with remote workers, and that it will ebb and flow depending on the project.

For example, some teams may want to be together in the early stages of creative and technical development, but then shift into working from home as the project ramps in production.

And I absolutely believe that the cloud and AI will be major components of future VFX, along with virtual production. The massive scale of the cloud and the ability of AI to handle video analysis and to automate many basic tasks will certainly change how we work.

Just as motion control allowed a wider range of visual effects, and reduced tedious manual labor, so too will these new tools provide a broader scope of visual effects and a reduction of the time-consuming steps we currently need to do. Virtual sets mean that on-set tasks such as setting up green screens and the proper lighting will be vastly simplified, and rotoscoping and compositing tasks will be dramatically reduced.

If there’s anything that those of us who have been in the industry for a long time know, it’s that big changes can take time to catch on. But when proof that the changes are immediately impactful is irrefutable (or when the solution addresses an urgent need) the adoption curve flattens. Think about the first time photorealistic CG characters appeared in Jurassic Park—before that, animatronics and stop motion were the norm. And think about how quickly and definitively that paradigm shifted.

I’m betting that what results from this remote experiment will just as dramatically change how we create visual effects—and what we’re able to create going forward. And I know that I’m excited to see what the future holds.