Every week, Frame.io Insider asks one of our expert contributors to share a tip, tool, or technique that they use all the time and couldn’t live without. This week, Laurence Grayson makes rotoscoping in After Effects much easier with Roto Brush 3.0.
If you’ve been working in motion graphics or VFX for any length of time, you’ll know two things to be true. One, that rotoscoping is unavoidable—every VFX or motion artist will need to rotoscope something at some point. And two, that rotoscoping sucks.
What is rotoscoping?
For anyone who doesn’t know, rotoscoping is the practice of separating something in a video frame from its background—creating foreground and background layers or “plates.” Historically this was done in After Effects by outlining the foreground subject by drawing nodes with the Mask tool. You’d then adjust those mask nodes to match the movement of the subject. Frame by frame. For (nearly) every frame. In short, it can a repetitive, time-consuming task, which is why it’s typically passed down to whomever sits at the bottom of your organization’s pecking order.
After Effects Roto Brush 1.0 and 2.0
It’s actually been 14 years since Adobe introduced Roto Brush in After Effects in CS5, and it was a huge leap forward. But like most groundbreaking tools, there was a fair amount of room for improvement.
If there was clear contrast between your foreground and background, and everything was in focus, and your subject didn’t move too much, and you didn’t have a lot of fine detail around the edges, it could save you hours. Otherwise, it was pretty hit and miss. And not having manual nodes to fall back on meant that a lot of die-hards (like me) tended to play it safe, and stuck to the old ways.
Fast forward about ten years to when After Effects Roto Brush 2 landed, and everything got noticeably better. Improved edge refinement tools took a usable tool and made it a useful tool—as Frame.io Insider Jason Boone demonstrates in this video.
After Effects Roto Brush 3.0
These days, we have After Effects Roto Brush 3, and to be fair, it’s not quite the giant leap between nothing and 1, or 1 and 2. But in rotoscoping any improvement is something to celebrate.
There’s actually quite a lot to tweak with Roto Brush 3.0, and this is an Insider Tip, not a full tutorial, so I’ll stick to what you need to know about Roto Brush in After Effects.
How to use After Effects Roto Brush 3.0
- First, make sure the clip you want to roto is in the Layer monitor (not the Composition monitor). Double-click on the clip if it’s not.
- Move the playhead in the Layer monitor to a point in your video where most of your roto target is visible in frame.
- Select the Roto Brush tool from the top of the After Effects UI and change the brush size with the mouse wheel, if needed.
- Draw over your roto subject to define the foreground, and hold down Opt/Alt and draw to define your background.
- You can reduce the range—or “span”— that Roto Brush 3 will analyze by dragging the handles on the Layer monitor’s timeline.
- When you’ve got a rough result, you can switch from the Roto Brush tool to the Refine Edge edge tool (click and hold the tool button at the top of the After Effects UI).
- Use the Refine Edge tool to draw around any areas, particularly hair, that need fine-tuning. If you have X-ray enabled (default), this will show a negative view of the details you’re highlighting.
Now it’s possible to just hit Space and let Roto Brush do the hard work for you. But even Roto Brush 3.0 can’t produce miracles, so you might prefer to advance frame by frame, keeping an eye open for any areas where your mask is starting to slip. (Jason does a great job of demonstrating this in the video above.)
When you’re happy with your selection, you can hit the Freeze button to lock it in place. You can unfreeze it later if you see something you missed on the first pass.
Set your expectations
If you’ve never rotoscoped anything before, then you might be expecting Roto Brush 3.0 to be a completely automated, fire-and-forget, single-button tool that does everything perfectly the first time. It’s not.
There are going to be times when you need to step in, and times when the footage will resist your best efforts. But trust me when I say this, After Effects Roto Brush is so much faster than doing it by hand and it’s the first thing I reach for when I’ve got rotoscoping to do. It also lifts the bar on footage, saving otherwise unusable clips from the bin.
As Morgan Prygrocki put it in a recent episode of Frame.io Live, “No one will miss rotoscoping by hand.” She’s not wrong.
Insider Tips are helpful weekly posts brought to you by Frame.io’s awesome writers and industry experts. Come back for a new Insider Tip every Wednesday, or subscribe to our newsletter to get a reminder each week.
Frame.io V4 is now available for all! Check out the launch announcement here.
