← All CS 180 Projects

Project 2: Fun with filters and frequencies!

Part 1: Fun with Filters

Part 1.1: Convolutions from Scratch!

For each output pixel, convolution multiplies a neighborhood of image pixels by a flipped filter and adds the products. I wrote two NumPy implementations. The first uses four loops: two over the image and two over the filter. The second keeps the image loops and uses NumPy to multiply and sum each neighborhood.

Both functions flip the filter horizontally and vertically. I add zero padding before convolution so the output has the same dimensions as the input. For an even filter dimension, the padding has one more pixel before the image than after it. This matches SciPy’s mode="same" alignment. Zero padding treats pixels outside the image as black, so the box blur darkens the border.

Four loops

def convolve_four_loops(image, kernel):
    # Get image and filter dimensions
    height, width = image.shape
    kh, kw = kernel.shape

    # Flip filter vertically and horizontally
    flipped = kernel[::-1, ::-1]

    # Add zero padding so output keeps original image size
    padded = np.pad(image, ((kh // 2, (kh - 1) // 2), (kw // 2, (kw - 1) // 2)), mode="constant")

    # Create empty output image
    output = np.zeros((height, width))

    # Visit each output pixel
    for row in range(height):
        for col in range(width):
            # Multiply each nearby pixel by its filter weight
            for i in range(kh):
                for j in range(kw):
                    output[row, col] += padded[row + i, col + j] * flipped[i, j]

    return output

Two loops

def convolve_two_loops(image, kernel):
    # Get image and filter dimensions
    height, width = image.shape
    kh, kw = kernel.shape

    # Flip filter vertically and horizontally
    flipped = kernel[::-1, ::-1]

    # Add zero padding so output keeps original image size
    padded = np.pad(image, ((kh // 2, (kh - 1) // 2), (kw // 2, (kw - 1) // 2)), mode="constant")

    # Create empty output image
    output = np.zeros((height, width))

    # Visit each output pixel
    for row in range(height):
        for col in range(width):
            # Select image neighborhood covered by filter
            patch = padded[row:row + kh, col:col + kw]
            # Multiply corresponding values + add products
            output[row, col] = np.sum(patch * flipped)

    return output

My 9 × 9 box filter gives all 81 pixels equal weight. Dividing by 81 keeps its weights summed to 1. I apply it to my 500 × 500 selfie and compare the outputs with scipy.signal.convolve2d using the same padding.

# Define 9x9 box filter
box_filter = np.ones((9, 9)) / 81
# Apply box filter with four loops + measure runtime
start = perf_counter()
blurred_four = convolve_four_loops(selfie, box_filter)
four_time = perf_counter() - start

print(f"Four-loop runtime: {four_time:.3f} seconds")

# Display blurred image
plt.imshow(blurred_four, cmap="gray", vmin=0, vmax=1)
plt.title("9x9 box blur: four loops")
# Apply box filter with two loops + measure runtime
start = perf_counter()
blurred_two = convolve_two_loops(selfie, box_filter)
two_time = perf_counter() - start

print(f"Two-loop runtime: {two_time:.3f} seconds")

# Display blurred image
plt.imshow(blurred_two, cmap="gray", vmin=0, vmax=1)
plt.title("9x9 box blur: two loops")
# Apply box filter with SciPy + measure runtime
start = perf_counter()
blurred_scipy = convolve2d(selfie, box_filter, mode="same", boundary="fill", fillvalue=0)
scipy_time = perf_counter() - start

print(f"SciPy runtime: {scipy_time:.3f} seconds")

# Compare pixel values with SciPy
print("Four loops match SciPy:", np.allclose(blurred_four, blurred_scipy))
print("Two loops match SciPy:", np.allclose(blurred_two, blurred_scipy))
Original selfie
Original selfie
Box blur: four loops
Box blur: four loops
Box blur: two loops
Box blur: two loops
Box blur: SciPy
Box blur: SciPy
9 × 9 box filter runtime
ImplementationTime (seconds)
Four loops5.341
Two loops0.570
SciPy0.023

I use np.allclose to compare pixel values. Both implementations match SciPy within its default tolerance. The two-loop version is faster because NumPy handles the inner multiplication and sum. SciPy is fastest in this run.

I also convolve the selfie with finite difference filters. Dx measures changes across columns, so vertical boundaries appear strongly. Dy measures changes across rows, so horizontal boundaries appear strongly. In the derivative displays, gray is zero, white is positive, and black is negative.

# Define finite-difference filters
dx = np.array([[1, -1]])
dy = np.array([[1], [-1]])
# Apply finite-difference filters with two loops
selfie_dx = convolve_two_loops(selfie, dx)
selfie_dy = convolve_two_loops(selfie, dy)

# Display horizontal brightness changes
plt.figure()
plt.imshow(selfie_dx, cmap="gray", vmin=-1, vmax=1)
plt.title("Selfie: Dx")

# Display vertical brightness changes
plt.figure()
plt.imshow(selfie_dy, cmap="gray", vmin=-1, vmax=1)
plt.title("Selfie: Dy")
Selfie: x derivative
Selfie: x derivative
Selfie: y derivative
Selfie: y derivative

Part 1.2: Finite Difference Operator

I apply the same Dx and Dy filters to the cameraman image with SciPy. Here I use symmetric boundary extension, which mirrors values at the image border. I combine the derivatives with the gradient magnitude, √(Ix² + Iy²), to measure edge strength in either direction.

Cameraman: original
Cameraman: original
Cameraman: x derivative
Cameraman: x derivative
Cameraman: y derivative
Cameraman: y derivative
Gradient magnitude
Gradient magnitude
Binary edges: threshold 0.15
Binary edges: threshold 0.15

I binarize the magnitude with gradient_magnitude > 0.15. This keeps the camera and much of the person’s outline while removing many small responses in the grass. Some grass remains, and some weak edges disappear. I chose this threshold to reduce the background detail without losing too much of the subject.

Part 1.3: Derivative of Gaussian (DoG) Filter

Finite differences respond to small changes in brightness, including texture in the grass. I first smooth the image with a Gaussian filter. I use cv2.getGaussianKernel(19, 3) and multiply the column by its transpose to make a 19 × 19 filter with σ = 3. Pixels near the center receive more weight than pixels near the edge.

Gaussian filter: 19 × 19, σ = 3
Gaussian filter: 19 × 19, σ = 3
Gaussian-smoothed cameraman
Gaussian-smoothed cameraman

I take finite differences of the smoothed image and calculate the gradient magnitude again. Smoothing reduces fine texture and spreads the remaining edges over more pixels. It also lowers their peak strength, so I use a smaller threshold of 0.03. A threshold of 0.04 removed more of the outline than I wanted.

Smoothed gradient magnitude
Smoothed gradient magnitude
Smoothed binary edges: threshold 0.03
Smoothed binary edges: threshold 0.03

I then convolve the Gaussian filter with Dx and Dy using mode="full". These derivative of Gaussian (DoG) filters combine smoothing and differentiation. Convolution is associative: (I ∗ G) ∗ D = I ∗ (G ∗ D). I can apply each DoG filter directly to the original cameraman image.

DoG x filter: 19 × 20
DoG x filter: 19 × 20
DoG y filter: 20 × 19
DoG y filter: 20 × 19
Smooth, then differentiate
Smooth, then differentiate
Differentiate Gaussian, then filter image
Differentiate Gaussian, then filter image

The x derivatives, y derivatives, and gradient magnitudes from the two methods all pass np.allclose in my notebook. Compared with Part 1.2, both methods reduce the grass texture and produce wider, smoother edges. The magnitude images use their own display ranges so the edge structure is visible.

Part 2: Fun with Frequencies!

Part 2.1: Image "Sharpening"

A Gaussian blur reduces fine detail. I subtract the blurred image from the original to isolate high frequencies, then add a scaled copy of those details back to the original. The sharpened image is I + α(I − G ∗ I). Here, I is the original image, G is the Gaussian blur filter, and ∗ means convolution. G ∗ I is the blurred image, so I − G ∗ I contains the high-frequency details. The sharpening amount α controls how much of that detail I add back. With α = 0, the image stays unchanged; larger positive values increase the sharpening.

I combine these operations into one convolution with the filter (1 + α)δ − αG. The impulse filter δ has a 1 at its center and zeros elsewhere, so convolving it with I returns I unchanged. This combined filter produces (1 + α)I − α(G ∗ I), which is the same sharpening formula written above.

I use a 7 × 7 Gaussian with σ = 1 and symmetric boundary extension. After sharpening, I clip values to [0, 1]. The high-frequency displays use gray for zero so both positive and negative details remain visible.

Taj Mahal

Taj Mahal: original
Taj Mahal: original
Taj Mahal: Gaussian blur
Taj Mahal: Gaussian blur
Taj Mahal: high frequencies
Taj Mahal: high frequencies
α = 0
α = 0
α = 1
α = 1
α = 2
α = 2

Berkeley

Berkeley: original
Berkeley: original
Berkeley: Gaussian blur
Berkeley: Gaussian blur
Berkeley: high frequencies
Berkeley: high frequencies
α = 0
α = 0
α = 1
α = 1
α = 2
α = 2

Increasing α strengthens contrast around edges. At α = 2, the building details stand out more, but bright and dark outlines also become stronger. Sharpening can exaggerate existing texture.

Blur, then sharpen

I blur the Berkeley image with the same Gaussian, then sharpen that blurred image with α = 2. The result has stronger boundaries than the blurred image. Fine detail lost during blurring is still missing. The full images make the overall change look small, so I also compare the same crop from each image.

Original
Original
Blurred
Blurred
Resharpened: α = 2
Resharpened: α = 2
Original: close-up
Original: close-up
Blurred: close-up
Blurred: close-up
Resharpened: close-up
Resharpened: close-up

Part 2.2: Hybrid Images

A hybrid image combines the low frequencies of one image with the high frequencies of another. I blur the first image to keep its low frequencies. I subtract a blurred copy of the second image from itself to keep its high frequencies, then add these two components. The first image’s broad shapes are easier to see from farther away, while the second image’s fine details are easier to see up close. This implements the Gaussian filtering method in Section 2 of Oliva, Torralba, and Schyns’s Hybrid Images.

I use the starter align_images function and select corresponding eyes to align the faces. The function adjusts position, scale, and rotation, then crops the images to a shared size. This follows the paper’s discussion in Section 2.2: matching features and pose helps the two images form a coherent face.

Each Gaussian extends three standard deviations from its center. I filter vertically and horizontally with symmetric boundary extension. The σ values below are in pixels after alignment. A larger σ makes a wider blur and a lower low-pass cutoff. For the high-pass component, subtracting that wider blur retains a broader range of detail.

Derek and Nutmeg

Derek: original
Derek: original
Nutmeg: original
Nutmeg: original
Derek: aligned
Derek: aligned
Nutmeg: aligned
Nutmeg: aligned
Derek + Nutmeg
Derek + Nutmeg

Derek supplies low frequencies with σ = 12, and Nutmeg supplies high frequencies with σ = 8. I pass Nutmeg first to the alignment function so the rotation adjusts Nutmeg toward Derek’s pose. Nutmeg’s eyes and fur are visible up close; Derek’s face becomes clearer when I step back.

Paige and Azzi

Paige and Azzi are my favorite pair. Paige supplies the low frequencies with σ = 4, and Azzi supplies the high frequencies with σ = 3. I first tried σ = 6 for Paige, but her face was too faint. Reducing it to 4 kept more of her facial structure.

Paige: original
Paige: original
Azzi: original
Azzi: original
Paige: aligned
Paige: aligned
Azzi: aligned
Azzi: aligned
Paige: low frequencies, σ = 4
Paige: low frequencies, σ = 4
Azzi: high frequencies, σ = 3
Azzi: high frequencies, σ = 3

I calculate each Fourier display with log(abs(fftshift(fft2(image)))). The shift puts low frequencies at the center, and the logarithm makes weaker frequencies visible. The input transforms below use the aligned images. I display each transform with its own grayscale range.

FFT: aligned Paige
FFT: aligned Paige
FFT: aligned Azzi
FFT: aligned Azzi
FFT: low-pass Paige
FFT: low-pass Paige
FFT: high-pass Azzi
FFT: high-pass Azzi
FFT: final hybrid
FFT: final hybrid

The low-pass result concentrates energy near the center. Subtracting Azzi’s blur removes low frequencies, while her edges remain in the high-pass result. The final hybrid contains both components. Their mouths and face outlines do not align perfectly, so some doubled features remain visible up close.

Final combined hybrid: Paige + Azzi
Final combined hybrid: Paige + Azzi

Josh and Rushil

Josh: original
Josh: original
Rushil: original
Rushil: original
Josh + Rushil
Josh + Rushil

Josh supplies low frequencies with σ = 6, and Rushil supplies high frequencies with σ = 6. Rushil was hard to see with a high-pass σ of 3. Increasing it to 6 retained more of his face, while Josh remained visible from farther away. These settings differ from Paige and Azzi because the aligned faces have different scales and details.

Part 2.3: Gaussian and Laplacian Stacks

I build six Gaussian levels for each fruit. Level 0 is the original image. To create each next level, I blur the previous one with σ = 2, 4, 8, 16, and 32 in order. These are successive blur widths, so each level includes the smoothing from earlier levels. Every level keeps the original 300 × 300 dimensions.

This uses the scale separation described in Section 2 of Burt and Adelson’s A Multiresolution Spline With Application to Image Mosaics. Their pyramids reduce image dimensions between levels. My stacks retain the full dimensions, as the project requires.

Gaussian stacks

Apple: Gaussian level 0
Apple: Gaussian level 0
Apple: Gaussian level 1
Apple: Gaussian level 1
Apple: Gaussian level 2
Apple: Gaussian level 2
Apple: Gaussian level 3
Apple: Gaussian level 3
Apple: Gaussian level 4
Apple: Gaussian level 4
Apple: Gaussian level 5
Apple: Gaussian level 5
Orange: Gaussian level 0
Orange: Gaussian level 0
Orange: Gaussian level 1
Orange: Gaussian level 1
Orange: Gaussian level 2
Orange: Gaussian level 2
Orange: Gaussian level 3
Orange: Gaussian level 3
Orange: Gaussian level 4
Orange: Gaussian level 4
Orange: Gaussian level 5
Orange: Gaussian level 5

For Laplacian levels 0 through 4, I subtract each next Gaussian level from the current level: Li = Gi − Gi+1. Level 5 stores the final blurred image. Each difference contains detail removed by that step of blurring. Adding all six levels reconstructs the original because the intermediate Gaussian terms cancel. This follows the difference and reconstruction method in Sections 2.2 and 2.3 of Burt and Adelson.

Laplacian stacks

Apple: Laplacian level 0
Apple: Laplacian level 0
Apple: Laplacian level 1
Apple: Laplacian level 1
Apple: Laplacian level 2
Apple: Laplacian level 2
Apple: Laplacian level 3
Apple: Laplacian level 3
Apple: Laplacian level 4
Apple: Laplacian level 4
Apple: level 5, blurred residual
Apple: level 5, blurred residual
Orange: Laplacian level 0
Orange: Laplacian level 0
Orange: Laplacian level 1
Orange: Laplacian level 1
Orange: Laplacian level 2
Orange: Laplacian level 2
Orange: Laplacian level 3
Orange: Laplacian level 3
Orange: Laplacian level 4
Orange: Laplacian level 4
Orange: level 5, blurred residual
Orange: level 5, blurred residual

Recreating Figure 3.42

I use a mask that selects the apple on the left and the orange on the right, then build a Gaussian stack of that mask. At each level I multiply the apple’s Laplacian level by the mask and the orange’s by one minus the mask. I add the two contributions. Part 2.4 explains this blending step in more detail.

The layout below recreates panels (a) through (l) of Figure 3.42 in Szeliski’s Computer Vision: Algorithms and Applications. The first three rows contain masked Laplacian contributions at levels 0, 2, and 4. The last row sums all six levels, including levels omitted from the display.

(a) Apple contribution, level 0
(a) Apple contribution, level 0
(b) Orange contribution, level 0
(b) Orange contribution, level 0
(c) Combined, level 0
(c) Combined, level 0
(d) Apple contribution, level 2
(d) Apple contribution, level 2
(e) Orange contribution, level 2
(e) Orange contribution, level 2
(f) Combined, level 2
(f) Combined, level 2
(g) Apple contribution, level 4
(g) Apple contribution, level 4
(h) Orange contribution, level 4
(h) Orange contribution, level 4
(i) Combined, level 4
(i) Combined, level 4
(j) Apple contribution, all levels
(j) Apple contribution, all levels
(k) Orange contribution, all levels
(k) Orange contribution, all levels
(l) Final blend
(l) Final blend

Fine edges appear in the first row. Lower rows contain broader detail, and their Gaussian masks spread the transition over more pixels. Each of the first three rows uses a shared symmetric display range across its three panels. The bottom row shows the reconstructed contributions and final blend.

Final oraple
Final oraple

Part 2.4: Multiresolution Blending (a.k.a. the Oraple!)

I use the mask-based blend in Section 3.2 of Burt and Adelson, on page 230. I build Gaussian stacks for the mask and both input images, then obtain Laplacian stacks for the images. At each level, Bi = MiLA,i + (1 − Mi)LB,i.

Here, i is the stack level, and A and B label the two input images. LA,i and LB,i are their Laplacian stack values at that level. Mi is the corresponding Gaussian mask level, and Bi is the blended result at that level. I multiply corresponding pixels: Mi weights image A’s contribution, and 1 − Mi weights image B’s contribution. A mask value of 1 selects A, a value of 0 selects B, and values between them mix both. The final image is the sum of Bi over all six levels.

The mask is sharper at fine levels and more blurred at coarse levels. This keeps small details near the selected boundary while mixing broad brightness changes over a wider region. Since the stacks have matching dimensions, I combine corresponding pixels directly and sum the levels.

Apple and orange

Apple
Apple
Orange
Orange
Mask: apple on left
Mask: apple on left
Mask: Gaussian level 0
Mask: Gaussian level 0
Mask: Gaussian level 1
Mask: Gaussian level 1
Mask: Gaussian level 2
Mask: Gaussian level 2
Mask: Gaussian level 3
Mask: Gaussian level 3
Mask: Gaussian level 4
Mask: Gaussian level 4
Mask: Gaussian level 5
Mask: Gaussian level 5
Final oraple
Final oraple

Isabel and Paige

I replace Paige’s face with mine. I resize the basketball image to 1,000 pixels wide and select corresponding eyes in both images. A similarity transform scales, rotates, and moves my face into position. I then shift the target points 20 pixels right to place my face within Paige’s face area.

Isabel: original
Isabel: original
Paige: original
Paige: original

The mask is an ellipse centered at (x, y) = (470, 168), with horizontal radius 45 and vertical radius 65 pixels. White selects my face; black selects Paige’s image. The black region outside the warped selfie is padding from the transform. The mask restricts the blend to the face area.

Isabel: transformed to Paige’s image
Isabel: transformed to Paige’s image
Elliptical face mask
Elliptical face mask
Isabel + Paige
Isabel + Paige

I use the same six levels and blur widths as the oraple. This keeps Paige’s pose and background while replacing the face. Some shading remains around the jaw and neck where the source faces differ.

Mask and blended levels

Face mask: Gaussian level 0
Face mask: Gaussian level 0
Face mask: Gaussian level 1
Face mask: Gaussian level 1
Face mask: Gaussian level 2
Face mask: Gaussian level 2
Face mask: Gaussian level 3
Face mask: Gaussian level 3
Face mask: Gaussian level 4
Face mask: Gaussian level 4
Face mask: Gaussian level 5
Face mask: Gaussian level 5

For this pair, I show the masked contributions in the same layout as Figure 10 of Burt and Adelson. Rows one through three use levels 0, 2, and 4. The last row adds all six levels. Fine facial details remain localized, while the coarser contributions mix brightness over a larger area.

(a) Isabel contribution, level 0
(a) Isabel contribution, level 0
(b) Paige contribution, level 0
(b) Paige contribution, level 0
(c) Combined, level 0
(c) Combined, level 0
(d) Isabel contribution, level 2
(d) Isabel contribution, level 2
(e) Paige contribution, level 2
(e) Paige contribution, level 2
(f) Combined, level 2
(f) Combined, level 2
(g) Isabel contribution, level 4
(g) Isabel contribution, level 4
(h) Paige contribution, level 4
(h) Paige contribution, level 4
(i) Combined, level 4
(i) Combined, level 4
(j) Isabel contribution, all levels
(j) Isabel contribution, all levels
(k) Paige contribution, all levels
(k) Paige contribution, all levels
(l) Final blend
(l) Final blend

Minion and Snoopy

I resize Snoopy’s image to 1,000 pixels wide and use another similarity transform to place the Minion’s face over Snoopy’s head. I select target points for the Minion’s eyes and use an ellipse centered at (500, 200), with radii 115 and 140 pixels. I blend with the same stack settings.

Minion: original
Minion: original
Snoopy: original
Snoopy: original
Minion: transformed
Minion: transformed
Elliptical face mask
Elliptical face mask
Minion + Snoopy
Minion + Snoopy

The goggles extend above Snoopy’s head, and the blend has a soft boundary around the face. Snoopy’s ears and body remain visible. Both custom blends use curved masks.

What I learned

The most important thing I learned was the difference between Gaussian and Laplacian stacks. A Gaussian level contains the image after repeated blurring. A Laplacian level contains the detail lost between two Gaussian levels. Keeping the final blurred image lets me add the Laplacian levels back together to reconstruct the original. In the blends, I can control how each level crosses the mask boundary.