Camera stabilizer

A camera stabilizer built with the OpenCV library, including support for stabilizing streamed video. The approach is similar to Nghia Ho's implementation, with a modification that supports streaming, so only previous frames are used to compute the stabilization.
I assume the shake can be compensated with a rigid transformation, smoothing the x and y offset and the z rotation.

The project

This started as lab work at university. It uses the OpenCV open source library, which is widely used for computer vision and machine learning. I wrote an extension for an experimental C++ node-based video compositor built on the Qt framework.

Feature detection

Features are regions of an image with distinctive traits that can be identified even after transformations (e.g. translation, rotation, scale). Feature detection is the process of finding such regions. Feature matching is identifying the same feature in another image after transformations have been applied.
The goodFeaturesToTrack method finds strong corners in a frame (1st parameter) and stores them in a vector of 2D points (2nd parameter). The remaining parameters are, respectively: maximum feature count, quality level, and minimum distance between features.

goodFeaturesToTrack(PreviousFrameBW, PreviousFrameCorner, 200, 0.01, 30);

Feature matching

Now that the features are linked to the previous frame, I identify the same ones in the current frame with the calcOpticalFlowPyrLK method. The first three parameters are already known; PreviousFrameCorner was computed above. The matched features are written to CurrentFrameCorner, status stores 1 if a feature was matched and 0 otherwise, and err logs the errors.

cv::calcOpticalFlowPyrLK(PreviousFrameBW, CurrentFrameBW, PreviousFrameCorner, CurrentFrameCorner, status, err);

for(size_t i=0; i < status.size(); i++) {
    if(status[i]) {
        PreviousFrameFeatures.push_back(PreviousFrameCorner[i]);
        CurrentFrameFeatures.push_back(CurrentFrameCorner[i]);
    }
}

I keep only the features for which a match was found, and discard the rest.

Features at previous and current frame

Estimating the rigid transform

I now have two lists (PreviousFrameFeatures, CurrentFrameFeatures) of 2D points, where matching indices represent the same feature on each frame. This method estimates the transformation between the two frames:

CurrentTransformMatrix = estimateRigidTransform(PreviousFrameFeatures, CurrentFrameFeatures, false);

Some elements in the scene may be moving on their own, for example the subject or water in the background. Even so, the method estimates the most probable transformation and tolerates features that represent an incorrect offset. This gives the delta translation and rotation between consecutive frames. Accumulating these deltas produces the trajectory.

Smoothing

Simply correcting each frame by the inverse trajectory removes the shake, but over time the trajectory can grow so large that the frame drifts off screen. To prevent this, I also push the frame back toward the center linearly (the number of frames it takes to re-center is a parameter). Once the center is reached, the trajectory is reset and the process starts over, keeping the image in frame except during very large shakes.

Trajectory and smoothed trajectory

The inverse trajectory (red) combined with the vector pushing toward the center yields a smoothed trajectory (green).

Trimming

Even with the subject smoothly re-centered, large corrections can expose the edges of the stabilized frame. To handle this, I added trimming.

With and without trimming

A crop radius (also a parameter) removes the region where the frame edges would be visible. The best value is the smallest one that still hides those edges.