Case study with POSM: Stitching 360-degree photos taken by a camera travelling through a pipe

A USA-based company, POSM (Pipeline Observation System Management) - asked us for help with extending their Panomorph software. We have happily accepted this task and made the integration code between customer's software and SharpStitch.

This was a very interesting project. The story begins with a camera travelling through a pipe. The camera is equipped with proprietary type of lens called Panomorph and produces images such as this one:

PanoMorph shot

The client's software "unwraps" these 360-degree circular images into a flat format:

Finally, these flat images are stitched together. The result is 2D overview of the internal area of the pipe:

As simple as it looks, there were some challenges to overcome on our side (the stitching part). The source images were far from optimal. The camera riding through pipe sometimes produced duplicate images, it was shaken from time to time, it can go temporarily out of focus and the lens can be covered by water droplets (presenting themselves as repeating pattern over the mosaic). Finally, SharpStitch has been updated to deal with these defects and produce stable and quality results.

The project required us to add several features in the SharpStitch library. You can use them to your benefit as well and I will cover these briefly.

Color Calibration

The source images were taken with wide-angle lens and hence suffer from serious light falloff near border of the lens. This effect is called vignettingand is more tanglible on wide-angle (e.g. fisheye) lenses than on classical circular lenses with moderate focal lengths.

The effect of vignetting can be removed by image processing. We have implemented an optimization framework for removing all kinds of nonlinear photometric distortions such as vignetting. Color calibration is SharpStitch can be thus adjusted to special and custom cases. For example, the unwrapped pipe images posess light falloff in vertical direction instead of radial and spreads in top-bottom fashion instead from center of the image.

The following images show the mosaic with and without color calibration used:

Fast Pixel-Based Matching Of Images (A.K.A. Template Matching)

There are basically two methods used to align images in the process of "image stitching": Direct and feature-based. By default, SharpStitch uses feature-based stitching because of its many advantages, namely speed and generality.

There are cases, however, where direct matching can do better. The unwrapped images of pipes are very small and their movement is simple (a vertical shift of some 5 to 25 pixels). This is ideal for template matching. We have implemented this in SharpStitch while adding further optimizations. You can also combine direct and feature-based matching in a stitching process.

Multi-Core Processing

Thousands of images are stitched in this case and it is needless to do so sequentially on today's multi-core CPUs. We have reached good CPU utilization for two stages of image stitching (feature detection and template matching). The multi-core processing is turned on by default in SharpStitch, but can be turned off. We have used both options to measure speed improvement. The multi-core processing is about 6 times faster on a Intel i7 Quad Core CPU. It can't reach the theoretical optimum 100% CPU utilization because parallel code usually needs synchronization (accessing shared resources). This causes temprorary drops in the CPU utilization.

We plan further speed improvements by using GPU for graphical (rendering, filtering) operations as well as for general purpose computation (GPGPU).

The following graphs shows CPU utilization (all cores) by SharpStitch (green) and system processes (grey) using sequential and parallel processing:

The calibrated images are easier to blend because differences in overlaps have been minimized by the calibration. Image blending is then based purely on image content and not affected by color distortions.

Sample Application available on request

Based on your specific scenario and requirements, we can create a derived sample application (including source code) solving this kind of stitching problem with SharpStitch.

Please contact us to request the sample application. Include any information about your objectives if you can.

Conclusion

POSM was able to switch from outdated and slow stitching application to our library while greatly improving performance and reliability of their software.

Furthermore the client benefits over time as we are continually improving SharpStitch performance.