For years, filmmakers – particularly timelapse and stop motion shooters – have been looking for ways to help smooth out their footage with interpolation. Or perhaps you just want to slow some footage down that wasn’t shot at a high frame rate. Many of us have used Twixtor at some point in order to do this before native optical flow features started to appear in editors like Premiere Pro and DaVinci Resolve. Well, this one from researchers at Google, UC Merced and Shanghai Jiao Tong University just blows all previous techniques away. It’s called DAIN and it’s pretty awesome.
Twixtor and Optical Flow techniques to generate missing frames has sort of become the standard, but it’s not perfect by any stretch of the imagination. You can usually tell when footage has been slowed down in this way by the obvious telltale artifacts (ok, they’re not always quite as obvious as that one, but they’re often there).
This new technique, however, seems to cope with them extremely well, slowing down 30fps footage all the way to 480fps with virtually no signs that it’s been slowed. DAIN analyses each of your frames and then uses a specially designed AI to intelligently examine the footage to generate the in-between frames for silky-smooth playback. According to the project website… The initial goal was to be able to modernise old footage, to give it a more pleasing sense of motion and bring it more towards the types of video we’re used to seeing today. And it really does give those old videos a whole new look and feel. And when you see it processed through DAIN, you really get to understand how accurate (or how way off the mark) some of the movie and TV show recreations set in those periods really are. Among those important metrics for video quality, the most important one is the temporal resolution measured in frame-per-second or fps for short. Higher-frame-rate videos bring about more immersive visual experience to users so that the reality of the captured content is perceived. Therefore, the demand to improve the low-frame-rate videos, particularly the 12fps old films, 5~12fps animations, pixel-arts and stop motions, 25~30 fps movies, 30fps video games, becomes more and more urgent.
DAIN largely seems to differ from other existing methods of interpolation by its use of depth estimation. So it knows when something is in the foreground vs the background and doesn’t try to smear your subject against what’s behind them as other systems that only see flat footage often do. This means that whether you’re conforming old 16fps footage to 24 or 60fps for use in modern productions, or slowing down footage shot at 24fps down to 480fps slow motion, it should do a much better job than other methods you might have at your disposal. And the best bit? It’s free. You can find out more about it over on Wenbo Bao’s website, grab the code for yourself at GitHub, or download a pre-compiled Windows app (it’s in Alpha) here. [via Reddit]
