For Fall 2024, I am taking an image processing class where one of the assignments involves creating panoramas by implementing image transformations. The process involves manually labeling correspondences between images and performing image warping, rectification, and blending. Below, I document my approach to this assignment with in-depth explanations of each step.

## Table of Contents Link to heading

# Part A: Stitching a Panorama Link to heading

## Shoot the Pictures Link to heading

I used a Samsung Galaxy S22 with a 0.5x zoom to capture wide-angle shots. The goal is to capture overlapping images that can be aligned using homographies. Below are some of the pictures I took, which will be used for different stages of the project:

**MLK Building****Night Skyline****Table Top**

#### Guidelines for Capturing Images: Link to heading

**Camera Movement**: Keep the center of projection fixed and rotate the camera for consistent perspective changes.**Overlap**: Ensure about 40%-70% overlap between consecutive images to make alignment easier.**Avoid Barrel Distortion**: Use lenses that avoid distortion, keeping straight lines straight.**Lighting**: Capture images in quick succession to maintain consistent lighting and reduce potential artifacts.

## Recover Homographies Link to heading

Homographies are key to transforming points from one image into another. The homography matrix ( H ) is a 3x3 matrix that defines this transformation. It maps points from image 1 to image 2 based on their coordinates.

Given corresponding points ( p = (x, y) ) in the first image and ( p’ = (x’, y’) ) in the second, the transformation can be described by:

$$ p’ = H \cdot p $$

#### Steps to Recover Homography: Link to heading

**Select Corresponding Points**: Using a manual labeling tool, I selected pairs of corresponding points in both images.**Formulate the Equation**: With four or more correspondences, the homography matrix ( H ) can be computed by solving the system ( Ah = b ), where ( A ) is a matrix formed from the correspondence pairs and ( h ) is the vectorized form of the homography matrix.**Solve Using Least Squares**: For robustness, I use more than four points, resulting in an overdetermined system. The least squares method minimizes error in finding the best-fit homography.

Since our H matrix contains 8 unknown variables (the lower-right corner of the matrix is a scaling factor and set to 1), we need >= 8 equations to solve for the matrix. Each point correspondence gives us 2 equations. So, we need at least 4 points to solve the system of linear equations. In practice, we often use lots of points to avoid an unstable and noisy transformation. Least-squares can then be used to recover the homography matrix:

## Warp the Images Link to heading

Once the homography matrix ( H ) is computed, I use it to warp one image into the coordinate system of another. The warping operation applies a projective transformation, ensuring that the perspective between the images is corrected.

The warping process is defined as:

$$ p = H^{-1} \cdot p' $$

Where ( H^{-1} ) is the inverse of the homography matrix, used for **inverse warping**.

Key steps in warping:

**Forward vs. Inverse Warping**: I used inverse warping to avoid holes in the final image, computing pixel values in the destination image by interpolating pixel values from the source.**Interpolation**: Using`cv2.remap`

, I performed interpolation to fill in missing pixel values smoothly.

## Image Rectification Link to heading

Before blending the images into a mosaic, I rectified images to ensure the correctness of the homography transformation. Rectification involves mapping an image containing a known planar surface (like a poster or keyboard) so that the plane appears front-facing.

For example:

**Starcraft Laptop****Poster**

In both cases, I manually selected points on the objects and mapped them to a predefined square in the output space. This demonstrates that the warping process is functioning correctly.

## Blend the Images into a Mosaic Link to heading

With the images properly aligned through homographies, the next step is to blend them into a seamless mosaic. Simple stitching often leads to harsh edges and artifacts. To avoid this, I used **Weighted Distance Blending**.

#### Weighted Distance Blending: Link to heading

After aligning the images with homographies, I needed to blend them into a seamless mosaic. Initially, I experimented with **Laplacian pyramids** for multi-scale blending. While this approach is effective in preserving detail and avoiding harsh transitions, it was challenging to determine the exact boundaries between overlapping images, leading to imperfect blending results.

To address this, I switched to a more straightforward and effective method: **weighted feathering** using the `bwdist`

function. This method worked exceptionally well and produced smooth transitions between the overlapping regions of the images.

#### Weighted Feathering with `bwdist`

Link to heading

The idea behind this technique is to compute a **distance transform** for each image, which measures the distance from the nearest edge of the mask. The further a pixel is from the boundary, the more weight it carries in the blending process. The weights are normalized to smoothly transition between the two images based on their distances from the edges.

Here’s the Python code I used for blending:

```
def bwdistBlend(panorama, enlarged_image):
# Create binary masks for each image
mask_2 = np.any(enlarged_image > 0, axis=-1).astype(np.float32)
mask_1 = np.any(panorama > 0, axis=-1).astype(np.float32)
# Compute the intersection mask where both images overlap
intersection = np.logical_and(mask_1, mask_2)
# Compute distance transform for both masks (distance from zero)
dist_1 = bwdist(mask_1)
dist_2 = bwdist(mask_2)
# Normalize distance maps to range [0, 1]
dist_1_norm = dist_1 / (dist_1.max() + 1e-8)
dist_2_norm = dist_2 / (dist_2.max() + 1e-8)
# Compute the blend weights based on distance transforms
blend_weights_1 = dist_1_norm / (dist_1_norm + dist_2_norm + 1e-8)
blend_weights_2 = dist_2_norm / (dist_1_norm + dist_2_norm + 1e-8)
# Ensure the blend weights are only applied in the overlapping regions
blend_weights_1[~intersection] = 1 # Only panorama where no intersection
blend_weights_2[~intersection] = 0 # Only enlarged_image where no intersection
# Blend the two images using the computed weights
result = panorama + enlarged_image
result[intersection] = (panorama.astype(np.float32) * blend_weights_1[:, :, np.newaxis] +
enlarged_image.astype(np.float32) * blend_weights_2[:, :, np.newaxis])[intersection]
# Clip the result to ensure valid pixel values and convert back to uint8
result = np.clip(result, 0, 255).astype(np.uint8)
return result
```

Here are the final mosaics:

**Night Skyline Mosaic****MLK Building Mosaic****Table Mosaic**

# Part B: Automatic Feature Detection and Matching Link to heading

In this part, we implement automatic feature detection and matching to create panoramas without manual correspondence selection. We’ll follow the approach outlined in “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al., with some simplifications.

## 1. Harris Corner Detection Link to heading

The first step is detecting corner features using the Harris corner detector. These are regions with high contrast that exhibit significant changes in both horizontal and vertical directions.

**Implementation Details:**

- Used single-scale Harris corner detection
- Threshold set to maintain strong corner responses

**Results:**

## 2. Adaptive Non-Maximal Suppression (ANMS) Link to heading

To ensure a good spatial distribution of features and filter out weaker corners, I implemented ANMS as described in Section 3 of the paper.

**Implementation Details:**

- Computed suppression radius for each point
- Sorted points by the maximum radius that the point is the most “cornery” determined by harris corner detection value. Then picked to the top few points.
- This maintains even spatial distribution while preserving strongest features

**Results:**

## 3. Feature Descriptor Extraction Link to heading

For each detected corner, I created a distinctive feature descriptor to enable matching. (idea is that we want the first nearest neighbor to be super close and the second nearest neighbor to be super far)

**Implementation Details:**

- Extracted 40x40 patches around each corner
- Downsampled to 8x8 descriptors to handle noise
- Applied bias/gain normalization
- Used axis-aligned patches (no rotation invariance)

**Example Normalized Descriptors:**

## 4. Feature Matching Link to heading

Implemented feature matching using Lowe’s ratio test between first and second nearest neighbors.

**Implementation Details:**

- Computed distances between all descriptor pairs
- Applied ratio test for matching uniqueness
- Used threshold based on Figure 6b from the paper

**Matched Descriptor Examples:**

## 5. RANSAC Homography Estimation Link to heading

Used RANSAC to robustly estimate the homography transformation between images.

**Implementation Details:**

- Randomly sampled 4 point correspondences
- Computed homography for each sample
- Counted inliers using distance threshold
- Refined homography using all inliers

**RANSAC Results:**

- Yellow points: Inliers
- Red points: Outliers

## Manual vs. Automatic Comparison Link to heading

Here’s a side-by-side comparison of the results from manual and automatic stitching for all three scenes:

### MLK Building Link to heading

**Manual:**
**Automatic:**

### Night Skyline Link to heading

**Manual:**
**Automatic:**

### Table Scene Link to heading

**Manual:**
**Automatic:**

### Analysis of Results: Link to heading

**Accuracy**: Both methods produced well-aligned panoramas, with the automatic method showing comparable accuracy to manual selection. Both methods struggle with the MLK panorama as the overlap region is small and the transformation is large.**Efficiency**: Automatic method is extremely fast**Edge Cases**: Manual method can sometimes handle cases where automatic detection fails (the manual one in the mlk example looks better)

## Correspondence Comparison: Manual vs. Automatic Link to heading

### MLK Building Link to heading

**Manual Correspondences:**
**Automatic Correspondences:**

### Night Skyline Link to heading

**Manual Correspondences:**
**Automatic Correspondences:**

### Table Scene Link to heading

**Manual Correspondences:**
**Automatic Correspondences:**

### Analysis of Correspondence Methods: Link to heading

#### Manual Correspondence Selection Link to heading

**Advantages:**

- High precision in selecting specific features
- Ability to choose semantically meaningful points (e.g., corners of windows, distinctive architectural features)
- Works well in low-texture or repetitive regions where automatic methods might struggle
- Can handle cases with significant lighting changes

**Disadvantages:**

- Time-consuming process
- Subject to human error
- Limited number of correspondences (typically 8-12 points)

#### Automatic Correspondence Detection Link to heading

**Advantages:**

- Much faster execution
- Generates many more correspondences
- Consistent results across multiple runs
- Evenly distributed points across the image due to ANMS

**Disadvantages:**

- Can be fooled by repetitive patterns
- Sensitive to parameter tuning (Harris threshold, ANMS radius, matching ratio)
- May produce some incorrect matches (though RANSAC helps filter these)
- Requires sufficient texture in the images

### Impact on Final Results: Link to heading

**Alignment Quality**: Both methods produced well-aligned panoramas.**Processing Time**: Manual method took about 2-3 minutes per image pair for point selection, while automatic method runs in seconds**Reliability**: The manual selection was more reliable for challenging cases**Point Distribution**: Automatic method achieved better spatial distribution of points thanks to ANMS, while manual selection often concentrated on obvious features

## What I Learned Link to heading

The coolest thing I learned from this project was how robust feature matching can be achieved through RANSAC. It’s fascinating that Lowe’s ratio test and RANSAC can get correspondances that rival human labling. The process of seeing random corner points transform into meaningful correspondences through these algorithmic steps was particularly enlightening.

# Code Link to heading

You can find the full implementation of my project here.