HopperRender

An optical flow frame interpolator with DirectShow integration that allows you to watch any local media file in 60 fps. This is my first DirectShow and CUDA project. The goal is to achieve pretty decent frame interpolation with a variety of user customizable settings. The filter can be added to a DirectShow media player like MPC-HC or MPC-BE.

Please keep in mind that this project is still in ongoing development and there are very likely some bugs depending on the environment you're running and the setting you use. The interpolation quality is also not perfect yet, but pretty decent most of the time, especially for 24 fps -> 60 fps conversion.

Features

• Realtime frame interpolation of any source framerate to 60fps
• Compatible with HDR video (usage of madVR recommended)
• All resolutions (even DVDs and 4K Blu-rays) are supported
• No installation or internet connection required
• Warps frames in both directions and blends them for the smoothest experience
• User customizable settings to adjust the quality and performance of the interpolation
• HSV Flow visualization lets you see the calculated movements of objects in a scene
• Automatically adjusts internal settings to match the PC's performance
• Compatible with madVR, Enhanced Video Renderer, MPC-Video Renderer, and more
• Automatically detects the source frame rate (as well as playback speed) and disables interpolation if not needed
• Automatic scene change detection to prevent them from being interpolated
• Small Exporter GUI lets you render videos files with HopperRender

How to get started?

To use this filter, you need to use a DirectShow player like MPC-HC or MPC-BE. This filter uses the CUDA API and requires a NVIDIA GPU (GTX 950 or newer). The usage of madVR is recommended, but not necessary.

Installation

1. Open MPC-HC or MPC-BE's settings.
2. Select the External Filters submenu.
3. Click on Add Filter.

4. Click on Browse.

5. Select the HopperRender.dll file from the extracted zip file.

6. Select Prefer.

7. Close the settings and the media player.

That's it! You can now play a video with MPC-HC/BE and HopperRender will interpolate it to 60fps.

Note: Do not move or delete the folder containing the HopperRender.dll file, otherwise the media player won't find it.

Settings

You can access the settings when playing back a video with HopperRender by right clicking on the video in MPC-HC/BE, selecting the Filters menu and then HopperRender.

• You can activate and deactivate the interpolation

• You can select which type of frame output you want to see:

  • Warped Frame 1 -> 2: Shows just the warping from the previous to the current frame
  • Warped Frame 2 -> 1: Shows just the warping from the current to the previous frame
  • Blended Frame: Blends both warp directions together
  • HSV Flow: Visualizes the optical flow as a color representation, where the color indicates the direction of movement
  
  • Blurred Frames: Outputs the blurred source frames
  • Side-by-side 1: Shows the difference between no interpolation on the left, and interpolation on the right (split in the middle)
  • Side-by-side 2: Shows the difference between no interpolation on the left, and interpolation on the right (scaled down side by side)

• You can set the number of iterations (0 will automatically do as many iterations as possible)

• You can set the Frame and Flow blur kernel sizes which controls how much the frames or the flow will be blurred (Values between 0-32)

• In the status section, you can see the current state of HopperRender, the number of calculation steps that are currently performed, the source framerate, the frame and calculation resolutions, as well as the currently detected frame difference and scene change threshold

• The settings will be automatically saved to the registry HKEY_CURRENT_USER\Software\HopperRender so next time the filter is used, it loads the settings automatically

How it works

Note: The following is a very brief overview of how the filter works. It is not a complete, or 100% accurate description of the interpolation process. Refer to the source code for more details.

• To prevent the algorithm from focusing on pixel level details or compression artifacts, we first (depending on the user setting) blur the frames internally to use for the optical flow calculation
• HopperRender uses an offset array that shifts the frame according to the values contained in it
• The offset array has 5 layers that contain different shifts that can be 'tried out' at the same time to find the best one
• The first step involves setting the 5 layers to a horizontal shift of -2, -1, 0, 1, and 2
• Then, the first frame is shifted accordingly and we subtract the y-channel difference of the shifted frame to the unshifted next frame
• We then reduce all the absolute pixel deltas to one value and find out which layer (i.e. which horizontal shift) contains the lowest value and therefore difference
• Depending on the resulting layer index, we can either move on to the same procedure for the vertical movement, or continue moving in the negative or positive x direction
• We repeat this process until we are certain we found the best offset, or are out of calculation steps
• After having found the best offset for the entire frame, we decrease our window size to a quarter the size and continue the search again for every individual window starting at the previous position
• Depending on the user setting, we do this until we get to the individual pixel level
• Finally, we flip the offset array to give us not just the ideal shift to warp frame 1 to frame 2, but also to warp frame 2 to frame 1
• We then blur both offset arrays depending on the user settings to get a more smooth warp
• Then we use these offset arrays to generate intermediate frames by multiplying the offset values by certain scalars
• We add a bit of artifact removal for the pixels that weren't ideally moved and blend the warped frames from both directions together

Exporter

The exporter allows you to interpolate video files with custom settings instead of just watching it with the DirectShow Filter.

Note: Showing the preview will negatively impact export performance and will not show a frame time accurate playback. The Exporter does not support HDR videos and will currently not encode any audio.

Installation

1. Download OpenCV and extract it to C:\opencv.
2. Add C:\opencv\build\x64\vc16\bin to the Path System/Enviornment Variable.
3. Download OpenH264 1.8.0, extract it and copy the .dll to C:\opencv\build\x64\vc16\bin.
4. In the extracted release, launch HopperRenderExporter.exe and select the video you want to interpolate, as well as the desired settings.

Acknowledgements

This project is based on the EZRGB24 Filter Sample and the DirectShow core parts were inspired by the LAV Video Decoder.