While working at my last company, we specialized in building a software platform to analyze (primarily visual) data from industrial inspections. Our first integration was Boston Dynamics' Spot robot, but I was tasked with generalizing and expanding our platform to alternative hardware (drones, fixed cameras, alternative ground robots, etc.)
Another one of my main projects was one I conceived of myself: what I called “drift correction”. We had consistent trouble with “sensor-pointing” where the underlying robot would makes its best attempt to align its camera/sensors to the same location/angle as the original “mission recording”, but it would frequently make minor (and sometimes major) mistakes.
I spent a lot of time learning about traditional computer vision methods to determine what the best way to solve this might be.
First, we start with a “ground truth” image.
Then, we have a new image that we are trying to detect drift on, related to the ground truth.
Also, we have a Region of Interest (RoI) for each mission action. Thus, we can narrow down the search to only look in the region of interest in the ground truth image (since we know that's correct), and try to match it to anywhere else in the new image.
After dozens of iterations, the final product was to detect Keypoints in the Ground Truth RoI, detect keypoints everywhere in the new image, and describe all of those keypoints using a Descriptor algorithm.
Then, we match the keypoints using a brute force algorithm based on their Descriptors, and store the vector between the keypoints of each Match. At this point, you might imagine that we're done. We just calculate some summary vector between the points, and voila!
Unfortunately, our algorithm fails, a lot. The matches are rarely correct. To fix this, the solution I found was to look for the largest clusters of vectors. I cluster them twice, once based on their angle and once based on their distance. I then get any overlapping matches from each cluster, and, if it's above a certain threshold number of matches, we have found relevant drift.
From there, we simply calculate the average vector, and we have our drift calculation! Now, depending on the downstream task, we can make changes to fix the issue. For example, if we were just to attempt to overlay the two images (now corrected)
