DIP V Geometric Image Modification

I. Motivation

Image warping is an important technique in computer vision, which is widely used in applications such as image registration, panorama stitching, and video stabilization. Usually, a geometric or affine transformation is applied to an image such as scaling, rotation, translation, or distortion. In this process, the pixel locations are remapped in a way that preserves the visual appearance of the image, where intensity is not remapped in most cases. But more advanced applications also include intensity mapping. For example, the real-world images are often captured from different viewpoints, with varying lighting conditions and perspectives. It is often necessary to align these images and transform them into a common coordinate system, like stitching or registration. More specifically, image warping can be used to create panoramas by stitching together multiple images taken from different viewpoints. It can also be used to stabilize videos by compensating for camera motion. There are many ways to achieve it or approximately achieve it. In this section, we approximately achieve an image warping with non-linear mapping.

II. Algorithm

The goal is to achieve something like this:

Basically, we should achieve image warping by using affine transformation such as translation, rotation, scaling, and shear. But this problem includes non-linear warping, so we use a polynomial approximation to achieve this, which is formulated as follows.

Where the and represent the 12 parameters that are used for approximately warping image in a non-linear way; the and represent the target points that we want to map to; the and represent the source points that we want to map. 6 points are needed in both target and source images. The 6 by 6 matrix of M is the polynomial representation of the points in source image.

Besides, because we use inverse mapping to avoid black dots in warped image, so the points in M matrix are actually the points in the target image (warped), and the points in the 2 by 6 uv matrix are from the source image (original).

Control points. The choose of control points is important, which will directly lead to different result of warping. If we separate the images into 3 segments, the control points look like:

Where

the 1,2,3,4,5,6 represent the control points of triangle 1;

the 3,5,6,7,8,9 represent the control points of triangle 2;

the 1,4,6,8,9,10 represent the control points of the bigger triangle 3.

In such case, the triangle 1 and 2 can be normally warped to what we exactly want, but the triangle 3 cannot by using the approximately non-linear mapping with 2-order polynomial.

If we separate the images into 4 segments, it looks like:

Where the big triangle 3 is separated into two smaller triangles, which are represented by 1,4,6,13,10,11 and 6,8,9,12,10,13, respectively. In such cases, the approximately non-linear mapping with 2-order polynomial can lead to a better result in such warping case.

Cartesian and image coordinates. In the implementation, the coordinates need to be transformed from Cartesian to image space, and vice versa. This is done by using following formulas:

Where the and represent the coordinate (i,j) in Cartesian space; the and represent the coordinate (i,j) in image space; the is the height of image.

This may look complicated, but it can be easily visualized in a simple graph:

Interpolation. Because we use inverse mapping to achieve the warping and avoid black dots in the warped image, interpolation is needed when the corresponding coordinates (non-integer coordinates) in the source image of the warped image (integer coordinates) are calculated. The interpolation is implemented in three channels, respectively, with bilinear interpolation at the image space, which is formulated as follows:

Where the points (i,j), (i,j+1), (i+1,j), and (i+1,j+1) are the neighbors of points (y,x). The coefficients are as follows:

This can also be visualized as follows:

The overall algorithm steps are

(1) separating images into a few proper segments and choosing corresponding good control points;

(2) mapping coordinates of control points from image space to Cartesian space;

(3) calculating approximately non-linear coordinates mapping matrix using the 12 parameters for each segment in Cartesian space, which can be done in either forward mapping or inverse mapping;

(4) for each segment, using the corresponding mapping matrix to map all points to another image to get the warped image.

In the last step, forward mapping is straightforward and does not need any other operation, but inverse mapping needs additional interpolation operations to get a precise value of pixels, which is the key to avoiding black dots in the warped image.

III. Experiment

Inputs

Results

Figure 1. The warped images of dog from using different combinations of strategies. For example, the left upper is from applying 3 segments and inverse mapping; the bottom right is from applying 4 segments and forward mapping.

It’s easy to see that the results with 3 segments are not as good as those with 4 segments regarding the warping shape. This is understandable because the mapping algorithm used for this warping is not perfect and is just an approximation, so it cannot model a good nonlinear relationship between the original and warped images. In such a case, finer-grained segments can definitely improve the results, but also require more manual selection of the segments and corresponding control points.

We can see that the warped shapes from forward mapping are much better and closer to what we really want compared to those from the inverse mapping, regardless of whether there are 3 segments or 4 segments. But the disadvantage of forward mapping is also obvious: there are some black-valued pixels in the warped images, which lead to a less pleasing visual experience.

Overall, the 4-segment method is better, and inverse mapping is better than forward mapping, even with some not perfectly warped shapes, because the shape can be improved with better mapping algorithms, such as using polar coordinates, but the signal loss (black lines) cannot be avoided if we use forward mapping.

Figure 2. The warped images of dog (left column) and recovered images (middle and right columns) from corresponding warped ones. The middle column includes the recovered (map-backed) results with inverse back-mapping; the right column includes the recovered results with forward back-mapping.

Inverse/Forward back-mapping (after warping, we map it back to the original shape, where we could choose forward or inverse to achieve this). In Figure 2/Figure 4, the recovered images with inverse back-mapping (middle column) are much visually better than those with forward back-mapping (right column). Even though sometimes the former ones also cannot cover the whole original image size, they don’t have the obvious strip-like artifacts. This is because of the nature of forward back-mapping: it will cover all areas but doesn’t have enough information.

Number of segments. The recovered images from the 4-segment warped images are also better than those with the 3-segment method, no matter what mapping strategy is used. This can be seen in the middle column of Figure 2/Figure 4.

The third row (4-segments) result can cover more area compared to the first row (3-segments);

the 4-th row (4-segments) result doesn’t have an obvious artifact, which can be seen at the upper right corner in the second row (3-segments).

This can also be seen in the third column. The black pixels are more evenly distributed in the whole image if it’s recovered from the 4-segments warped image. The reason for being more evenly distributed in the upper right big triangle is that the second-order polynomial approximates the mapping and leads to a round-like artifact. The strip-like artifacts in the 3-segments method are because 5 control points are located on the same edge, which cannot lead to a good non-linear forward mapping.

Inverse/Forward mapping. The recovered images look different as well between forward-mapping warped images and inverse-mapping warped images. Looking at the middle column in Figure 2/Figure 4 as well, the second row can almost cover the whole image compared to the result in the first row; the 4-th row can also cover all areas better than the third row. But also, even though the recovered image from the forward-mapping warped image may cover more area, it also cannot avoid some black lines that are generated while warping images.

In conclusion, all recovered images cannot have the same visual quality compared to the original image, which is mainly because the warping operation will squeeze the original image to a smaller area, which means signal and information loss is inevitable. The lost information can be seen in those black pixel areas in the recovered images.

Similar things go to the following cat images.

Figure 3. The warped images of cat from using different combinations of strategies. For example, the left upper is from applying 3 segments and inverse mapping; the bottom right is from applying 4 segments and forward mapping.

Figure 4. The warped images of cat (left column) and recovered images (middle and right columns) from corresponding warped ones. The middle column includes the recovered (map-backed) results with inverse back-mapping; the right column includes the recovered results with forward back-mapping.