- What is HEVC?
- How does it work?
- The impending format war: HEVC vs. VP9 vs. AV1.
- Who will win out in the end?
You may have heard a lot about high-efficiency video coding or HEVC recently. Apple integrated this next-generation codec into MacOS High Sierra, and professional NLEs FCP X and Premiere Pro now support it. Other hardware and software vendors keep announcing new support almost every month.
This fanfare should come as no surprise to anyone in the video industry. HEVC promises to deliver high-quality 4k video that’s up to 75% smaller than before. Which could pave the way to an even more ambitious 8k HDR future. This sort of performance is what we dream about as video professionals, and could simplify and improve the jump to ultra high definition production.
If HEVC takes off, it will shape the industry for years to come. But there is growing competition in the realm of futuristic codecs. Challengers like VP9 and AV1 have powerful allies in the fight for our screens. So it’s time to sit down and take a good look at this new video technology and discover how it can benefit your video workflows.
What is HEVC?
In short, HEVC is the successor to the H.264 codec. If you’ve read our definitive guide to codecs, you should already be familiar with H.264. HEVC was specifically created to provide UHD HDR deliverables with wider color gamuts, rather than just the HD SDR Rec.709 deliverables of H.264.
The choice to use any particular codec is always partly informed by what particular hardware is available. With CPUs in the last decade or so becoming ever more powerful, cheaper, and more abundant, there has been an industry-wide trend to trade storage space on hard drives for computation via CPUs.
HEVC is just the latest continuation of the trend in trading storage for computation. In years past, digital intermediates like ProRes or DNxHR would eventually be converted to H.264 for web or broadcast. HEVC, on the other hand, requires even less storage space than H.264, yet it requires even more CPU power.
What this means is that the networks that we already use to deliver our HD SDR Rec. 709 video will be able to deliver more data into our video: higher dynamic range, wider color gamuts, and larger resolutions. The infrastructure won’t immediately require more bandwidth. The more powerful CPUs that live in our new smart TVs, tablets, and phones can decode more data out of the networks without requiring more bandwidth, all else being equal.
Claims about being able to more efficiently compress data into cutting-edge video codecs are exciting—but should you believe all the hype about equivalent quality?
An initial working group study that took place from 2013 to 2016 found that HEVC “clearly exhibited a substantial improvement in compression performance, as compared to AVC [H.264].” The study tested two different kinds of images: “natural” content and “synthetic” content. The “natural” content consisted of real-world images captured by a camera, and the “synthetic” content was from Sintel, a test movie created with the 3D graphics application Blender. The study found that HEVC reduced the bit rate of natural content by 51% to 74%, and synthetic content by 75%, with no loss of perceptible video quality.
How does HEVC work?
What’s under the hood that magically allows for lower data rates with equivalent visual quality? The answers are extremely technical, and if you are so inclined, you can check out the slides from Vivienne Sze of MIT and Madhukar Budagavi from Samsung or the July 2012 issue of IEEE Consumer Electronics Magazine. But for video professionals, here is a quick breakdown of HEVC’s compression methods.
From Macroblocks to CTUs
Whereas H.264 would break an image down into squares of pixels called “macroblocks,” HEVC breaks down the images into “Coding Tree Units” (CTUs) that can be up to 64×64 pixels. Each macroblock within H.264 can have only interframe or intraframe prediction, but not both.
In HEVC, CTUs actually use a combination of inter- and intraframe compression. This means that there are more methods of preserving detail in complex images.
You may have heard the term “macroblocking” to refer to artefacts—those nasty compression artifacts when the 16×16 squares can’t reproduce the fine details of a particular texture, like smooth gradients in clear skies or blocky clouds with a wide variety of luminance. In contrast to H.264’s macroblocks, HEVC’s CTUs don’t even need to be squares, so there’s much more flexibility in how different parts of the image are compressed.
From 8 to 35 intra prediction modes
In H.264 intraframe compression, there are only eight ways for each block to borrow nearby pixels. With additional computational complexity, HEVC can use up to 35 different methods to borrow nearby pixels.
New kinds of filtering
HEVC also uses new kinds of filtering to eliminate would-be artefacts, such as sample adaptive offset (SAO) and adaptive loop filtering (ALF). These filtering methods provide a big improvement over what is available for h.264.
HEVC in post-production today?
All these technical capabilities are impressive, and you might want to start using HEVC for all your projects immediately.
Unfortunately, it doesn’t yet have universal support for encoding and decoding in every post-production app, so you might need to practice some patience before you can benefit from its impressive technical capabilities. However, updates are steadily rolling out that enable HEVC support.
- Apple FCP X: Support for HEVC first appeared in Final Cut Pro X 10.4, which was released back in December 2017, but requires macOS High Sierra.
- Adobe Premiere Pro: Premiere Pro supports the import of HEVC media with resolutions up to 8192×4320, and can export to 720p, 1080p, 4K, and 8K, at 8-bit or 10-bit.
- Avid Media Composer: As of this writing, HEVC has not yet been supported in Media Composer. If presented with it, Avid MC users will need to convert to DNxHD.
- Blackmagic Design DaVinci Resolve: As of April 2016, HEVC can only be decoded on macOS.
Who wants HEVC?
The Broadcast Television Industry
One of the big drivers of HEVC will be the transition to ATSC (Advanced Television System Committee) 3.0—the next collection of technical standards for how television stations will broadcast out to viewers. Since the very outset of ATSC 3.0 planning, ATSC selected HEVC as the core codec. ATSC 3.0-compliant networks can broadcast 4K now. Later, if and when there’s a demand for 8K, similar HEVC pipelines will let broadcasters comfortably upgrade without much difficulty.
There are several indications that HEVC could take off in a big way for broadcast. Over 2 billion devices already support it, so producers and networks have a big incentive to make content for that user base. Several high-profile deployments have showcased its technical feasibility, like the 2016 World Series and the 2018 PyeongChang Olympic Winter Games.
To date, Apple has provided a few different tools for HEVC encoding. Devices running iOS 11 can already handle 8-bit HEVC content, so long as the device contains at least an A10 Fusion chip. For newer macOS devices, Apple supports 8-bit HEVC hardware encoding, and the High Sierra update introduced 10-bit HEVC software encoding.
Apple supports HEVC in one form or another across their entire product line, so it’s quite possible to deliver HEVC content to hundreds of millions of users almost anywhere.
It should be no surprise that Microsoft has mirrored Apple in HEVC adoption, though their support has been less consistent. Originally, HEVC was natively supported inside of Windows 10. But, Microsoft dropped native HEVC integration with the 2017 Fall Creators Update, and now requires downloading and installing a free Windows 10 extension.
In either case, Windows 10 should have no problem handling HEVC content, which pushes the codec’s reach to over 700 million more mobile and desktop devices.
Living Room Companions
Of course, the living room is still a major media consumption environment, and HEVC has a growing presence there. The heavy support of HEVC by broadcasters means that TV manufacturers have a huge reason to get behind the codec. Samsung, Sony, and LG have all started shipping TVs that natively decode HEVC, enabling much smoother live 4k content than before. And video-on-demand apps are showing some HEVC support too. Netflix has been playing around with x265 (an open-source HEVC encoder), Amazon Prime has adopted the codec, and Hulu began UHD streaming with HEVC as well.
This broad support for HEVC has already captured a huge portion of the global media consumption market. However, rivals are fighting back.
The Format Wars
A primer on HEVC wouldn’t be complete without the necessary context that HEVC is but one challenger in the next-generation format war. HEVC has a good head-start over the competition but the jury is definitely still out.
The Moving Picture Experts Group (MPEG) is the body that oversaw the development of HEVC. They’re also the ones who oversaw the development of MPEG-2, MP3, and H.264 in years past. Because codec development is a big, complicated, sprawling process, all the assorted businesses and organizations that help develop such formats typically join patent pools to recuperate the costs of development.
The patent pools are able to license the usage of the codecs. Device manufacturers and software developers pay license fees to use the codecs in their products. This licensing cost is then incorporated into the price of the hardware or software.
This business model has worked for decades, but HEVC hasn’t yet taken root like H.264 did. Mainly because it now faces a worthy competitor: VP9.
YouTube (owned by Google) was never fully satisfied with the licensing agreement for H.264. Back in 2010, Google acquired a video compression company that owned VP8, a comparable alternative to H.264. YouTube then open-sourced VP8 in the WebM container, unleashing into the world an alternative to H.264 without any licensing fees.
By 2015, it was clear that YouTube didn’t want to license HEVC, and didn’t bother with it. Instead, they adopted VP9. VP9 is to VP8 as HEVC is to H.264.
In 2015, Amazon, Cisco, Google, Intel, Microsoft, Mozilla, and Netflix created the Alliance for Open Media (AOM). In 2016, AMD, ARM and NVIDIA joined AOM and AOM announced the new codec in development, now known as AV1. Bitmovin, the co-creator of the MPEG-DASH video streaming standard, joined AOM in 2017. Hulu followed, along with Facebook.
In July 2017, it was looking like Apple was going to be the main backer of HEVC, which could have fueled a long and wasteful format war, but in a surprise twist, Apple quietly joined AOM this past January as a founding member. AOM updated its website to include Apple on its list, without any kind of press release. And Apple refused to respond to a request for comment. What this means for the future of HEVC is anyone’s guess, but since most major players are members of AOM, it’s difficult to understand why they would want to pay the licensing costs for this technology.
In spite of AV1’s significant backing, HEVC remains a strong contender largely because of its significant head-start. HEVC now has wide support in software and hardware, and AV1 is starting from scratch. Because of the significant processing power required to decode either of these two new codecs, it is impractical to expect devices to play them back unless they have been specifically designed to support hardware decoding. While most software can be updated reasonably quickly to support new codecs, hardware is another story.
Leonardo Chiariglione, the founder and chairman of MPEG, worries that AOM won’t be able to fund research and development as well as MPEG would, and the fast pace of innovation we’ve seen for 30 years from MPEG is about to slow down.
Though many consumer devices, NLEs, and operating systems support HEVC, the streaming industry is poised to adopt AV1. HEVC may be useful as a broadcast deliverable for ATSC 3.0 television networks. But Google, Amazon, Apple, Hulu, and Netflix all seem like they’re going to swing toward the mass adoption of AV1. On March 28 of this year, just in time for the NAB 2018, the AV1 1.0 spec was released. Remember that it’s just a 1.0 spec. So while we wait for the industry to start shipping products that support AV1, HEVC could still take off. (YouTube recently posted an AV1 beta launch playlist. By downloading a beta of Chrome 70 and tweaking a setting on YouTube, you can actually see YouTube’s implementation of the AV1 codec in action.)
In fact, we may see a future where broadcast TV defaults to HEVC while streaming media defaults to AV1. And this isn’t just because of the disagreement over licensing. Early results show that AV1 will be 100x slower at live media encoding than HEVC. This makes AV1 technically unsuitable for wide swaths of broadcast 4k content. But it’s not an issue for streaming companies who deal with relatively little live content.
As you can see, a lot has been invested in the development of these technologies. The ability to save a significant amount in licensing fees makes open-source codecs compelling. But the speed and adoption of HEVC can’t be ignored. Your guess is as good as ours as to which will ultimately win.
Have you already adopted HEVC? Share your thoughts with the community, and let us know how it has changed your workflow.