Image Stitching

[Image Capture] [Feature Point Detection & Descriptor] [Feature Point Matching] [Recognizing Panoramas]

[Image Matching] [Blending] [Result] [Download] [Reference]


Image Capture

We use Nikon D80 to capture images from the NTU campus. The mode of the camera is set as manual, and it is controlled by a software, Camera Control pro 2 [Nikon]. Since we don't need too large image for image stitching, all images are resized to equal height as the beginning of the process.

Besides, we also capture two set of photos from the famous FPS game, [Crysis]. The first scene is shot surroundding with ruins and forests, and the second scene is near the sea.


Feature Point Detection & Descriptor

We adopted the method in [Brown04] for feature points detection, which is also orientation, scale, and illumination invariant. Although its ability in scale-invariance is not as good as SIFT, it still did a very good job in our works since the input pictures of panorama are usually at the scale.

We also adopt the descriptor in [Brown04] as our feature point descriptor, it's a vector with length equal to 8x8. Instead of having the gradient information as SIFT, the descriptor is composed of its neighboring color information.


Feature Point Matching

To speed up the matching process, we implemented Haar wavelet transformation and use its first three wavelets to build the lookup table. We choose 10 bins for each dimension, therefore, theoretically it gives a speedup of (10^3)/(2^3)=125 compared to exhaustive matching.

We match the points according to the ratio of the distance of the nearst points and the second nearest points. Since we use let the threshold of the ratio be very small, the number of the matching pairs is very small compared to the number of the feature points in images.

The resulting images of the previous two stages is as below:


Recognizing Panoramas

Given many images at different locations, we have implemented an algorithm to detect the possible panoramas from those images. The algorithm is as follows :

Input: images I and their feature points
Output: panoramas

For image i, j in I
   If number of match points between image i and image j is larger than N
      They are identified as a matching pair in P.
   End
End

K=0
While there is more matching pair in P that is not exploited yet
   1.Build another panorama k.
   2.Find the image pair s with the largest number of matching points
   3.Insert the two images in pair s into k
   
   While more pair in P that has image t in k
      Insert the image t into k
   End

   k = k+1
End

There exists a weakness in our algorithm, the threshold N is image dependent and is hard to predict. In some images the number of feature points is very small (Ex: beach), and the N needs to be small, too. On the contrary, in some images the number of feature points is very high, if N is too small, unmatched image pairs would be identified as a matching pair.

Another reason that may causes N to change is the overlapping size between two images, it is determined by the rotating angle of the camera when we take shots.


Image Matching

In the beginning, we apply RANSAC [Fischler81] to get the homography between each image pairs. However, since we don't know what to do next because we are not familiar with Bundle Adjustment, we go back to the method that describes the relationship between two images by only translation. We still use RANSAC to find this relationship.

Blending

We implement the algorithm mentioned in [Burt83] rather than the simplified two band version in [Brown03]. We blend the images by 5 levels of pyramid, including the original size level. Every level's images are blended by weighted sum. The higher level image is, the wider weighted sum region is.

The resulting images are as below:


Result

We have adopted two real scenes in NTU campus and two scenes snapshotted from [Crysis]. The following are the resulting images.

We don't do bundle adjustment in this system. Besides, we do not fix the camera or the snapshot vertically. For all these reasons, some distortion might occur. However, all resulting images seem quite nice in our system, so the problem of distortion may be minor.

From our images


[Full Size] [Panorama View]


[Full Size] [Panorama View]


[Full Size] [Panorama View]

From test images


[Full Size] [Panorama View]


Download

Captured images

Click the following url to download the zipped packages
[Department] [Grass] [Camp]

Source codes and execute files

To learn how to use, please check readme.txt
Stitching (VS 2005 project) [download]
The whole packages including our test images [download]


Reference

  1. Nikon official website (link)
  2. Crysis website in Electronic Arts (link)
  3. Matthew Brown, Richard Szeliski, Simon Winder, Multi-Scale Oriented Patches, MSR-TR-2004-133, 2004.
  4. Martin A. Fischler and Robert C. Bolles, Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, in Communications of the ACM, 24, 6 (1981), 381-395.
  5. P. J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics in ACM Transactions on Graphics, 2(4):217-236, 1983.
  6. M. Brown and D. G. Lowe, Recognising Panorama, ICCV 2003.