AI Roadmap LLM Learning Roadmap
✨ Awesome-Anything
Awesome-AI-Tutorials
Awesome-AI-Researchs
Awesome-GitHub-Repos
Awesome-AI-Examples🖼️ Digital Image Processing
DIP I Image Demosaicing and Histogram Manipulation
DIP II Image Denoising
DIP III Edge Detection
DIP IV Digital Half-toning
DIP V Geometric Image Modification🍃 LLM Components
1.1 Tokenization
1.2 Embedding
1.3 Attention
1.4 FFN, Residual Addition, and LN
1.5 Positional Encoding
1.6 Architecture & Decoding Policy🌱 LLM Pre-training
2.1 Pre-training Definition
2.2 Pre-training Data
2.3 Pre-training Workflow
2.4 Continued Pre-training (CPT)
2.5 Pre-training Evaluation☘️ LLM Post-Training
3.1 Supervised Fine-Tuning (SFT)
3.2 Reinforcement Learning
3.3 RLHF: Reinforcement Learning from Human Feedback
3.4 Optimizing PPO-Based Algorithms
3.5 Direct Preference Optimization (DPO)🍀 LLM Popular Models
4.1 BERT & Variants
4.2 PaLM Series
4.3 GPT Series
4.4 LLaMA Series
4.5 GLM Series
4.6 Qwen Series
4.7 DeepSeek Series
4.8 MOE Series
4.9 Other🪴 LLM Applications
5.1 Prompt Engineering
5.2 LLM-based Agents
5.3 Retrieval-Augmented Generation (RAG)
5.4 Enterprise-level RAG Practice (I)
5.5 Enterprise-level RAG Practice (II)🌿 LLM Optimization
6.1 GPU Usage Analysis
6.2 Latency Analysis for Inference
6.3 FlashAttention
6.4 PagedAttention
6.5 Packing for Efficient Training
6.6 PEFT🌾 LLM Compression
7.1 Quantization
7.2 Pruning
7.3 Knowledge Distillation🌵 LLM Hands-on Practice
8.1 nanoGPT: Training a GPT from Scratch
8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
8.4 OpenRLHF: Reward Model Training and Code Analysis
8.5 OpenRLHF: PPO Tutorial and Code Analysis🌴 LLM Must-read Papers
9.1 Technical Reports Explained
9.2 MedLLM
9.3 o1 Techinical Paper
9.4 Prompting
9.5 LLM-as-Judge🌳 LLM Q&A
10.1 Why GRPO training loss starts at 0
10.2 VLM-R1 Reproduction Results
10.3 KL(P||Q) or KL(Q||P)?
10.4 Left Padding or Right Padding?
10.5 What is QK-Norm?
10.6 Why LLM training rarely adopts Dropout?
10.7 Dropout and DropPath🐝 VLM Image Encoders
11.0 Introduction
11.1 Vision Transformer
11.2 CLIP
11.3 BLIP
11.4 EVA
11.6 BLIP-2
11.7 SigLIP📝 MISC.
OpenAI ChatGPT Progression
DIP II Image Denoising
I. Motivation
In the digital image processing field, noises can be caused by a variety of factors, including camera sensor noise, information loss from image compression, or transmission noise from a noisy channel. In both photography and medical scenarios, denoising is important for image analysis and processing tasks. For example, denoising can enhance the visibility of important features such as tumors, blood vessels, and more. To improve the visual quality and signal-to-noise ratio (SNR) of an image, there are many types of denoising techniques; for example, the median filter and Gaussian filter are the most straightforward methods. To determine which method to use, the type of noise should be investigated first. There are two common noises: Additive White Gaussian Noise (AWGN) and impulse noise (pepper/salt noise). The median filter is specifically effective for denoising impulse noise because the pepper and salt noise are valued as either 0 or 255, which makes them evident outliers. For denoising AWGN, low-pass filters and some more advanced algorithms (Bilateral Filtering, Non-local Means, Block Matching 3D, etc.) can be applied.
II. Algorithm
(a) Basic denoising methods
There are a few basic methods for denoising, including low-pass filters, high-pass filters, etc. In this section, low-pass filters are weighted by uniform and Gaussian distribution.
- For uniform-weighted filters, the weight for each surrounding pixel is , where the abbreviation of stands for the chosen kernel size.
- For Gaussian-weighted filters, the weight for each surrounding pixel is calculated as follows
where the (in the axis of width) and (in the axis of height) are the relative locations of surrounding pixels; the specific value for is related to the size of the kernel and defined as
Note that the calculated weights for all surrounding pixels do not sum up to 1, so a division is needed.
After getting the weights for a specific kernel, then pad the image with padding size of (i.e., taking the integer part) and
“reflect” mode as mentioned before. The final step is to traverse the padded image with size of in the original image part (eg. The y-axis is from to ) and do convolution with the kernel. The new calculated value for a pixel is stored in a new image at corresponding location. Finally, the new image would be the denoised image.The convolution operation for 2D space can be formulated as follows.
where the (in the axis of width) and (in the axis of height) are the location of a specific pixel that needs to be convoluted; the and are its surrounding pixels and represents the value for that pixel; the is the weight value for that corresponding pixel and ; the is the final result for the convolution around the pixel . Surely, the kernel calculation mentioned previously already consider the normalization of weights of kernel, so here the denominator part is just 1.
<ins/>
(b) Bilateral Filtering
Usually, the low-pass filter would only keep the low-frequency information, but high-frequency signals like edges are blocked and lost. This is where bilateral filtering can work better because it can preserve the edges. It’s formulated as follows.
where the (in the axis of width) and (in the axis of height) are the location of a specific pixel that needs to be denoised; is the location of neighboring pixels around the pixel at and is the value for that pixel. The weight is calculated as:
where the is the intensity for a pixel; and are two hyperparameters and can be adjusted for better performance; the and are sigma space and sigma intensity respectively, where the former controls the weight of spatial distances and the latter controls the weight of intensity distances. More analysis of these hyperparameters is discussed in the experiment section.
(c) Non-Local Means (NLM) Filtering
Previously mentioned algorithms only utilize the local information within the surrounding window for denoising. But the non-local means filtering utilizes the information from a larger region.
The above calculation is almost the same as in bilateral filtering. is the denoised value for the pixel at ; the is the intensity for the pixel at . The and are the locations of pixels around , which are in a larger search window M. The formula of is as follows.
where the is the filtering parameter and adjustable. The calculation for is formulated as:
where
The is a Gaussian weight for similarity measure; and denote the relative position in the neighborhood window, which is usually a smaller window N than the search window M; and is the standard deviation of the Gaussian kernel and can be adjusted.
Overall steps of this NLM algorithm are:
(1) choosing a pixel to denoise;
(2) calculating weight for each surrounding pixels within the big search window M by comparing the similarity between the two pixels, where the similarity is computed by calculating the difference between the two patches centered at the two pixels;
(3) averaging the values of neighbor pixels with corresponding weights.
<ins/>
(d) Denoising for color images
For denoising color images, we basically applied the denoising techniques to different channels of images. The channel could be RGB or any other color space like HSV. If applied to HSV color space, the conversion between RGB and HSV is needed, which can be done by using OpenCV.
To evaluate the denoising performance, the PSNR metric is used, which is formulated as:
where is the maximum possible pixel intensity and is usually 255; the and are the original noise-free image (reference image) of size (H, W) and noised/denoised image of size (H, W), respectively.
We may have to use color space conversion clipping because when the image is converted from RGB color space to HSV space and is applied with a denoising algorithm on HSV space, the numeric range on HSV space would change. This may cause problems when converting the image from HSV space back to RGB space because the numeric range may exceed 255 or below 0. In such a case, we can clip the range to 0 – 255. Without doing this, the value, for example, of 257, would become 1, which would cause a large area of color artifacts on the denoised image.
Additionally, angel averaging is especially needed when doing an average on the H channel of HSV color space. In HSV color space, the H channel means hue, ranging from 0 – 360 (degree) or 0 – 180 (degree) as implemented by OpenCV C++. This is because OpenCV C++ scales the hue component to fit in an 8-bit unsigned integer (0–255), and 180 provides enough resolution while fitting in this range.
In this case, if we average two values of 1 and 179, which are both near red color as defined by hue, the average value would be 90, which is azure and is near green and blue. Obviously, this behavior will inevitably cause evident color artifacts.
Therefore, a smarter averaging for angles is needed, which is formulated as follows.
where the function is just applying function to . The and are the sum of sin’s and cos’s, as calculated:
where the is the angle that needs to be averaged. The formula is straightforward. To average a few angles, we just
- sum up their length (could think of sin and cos as length in different axes) at the y-axis and x-axis to and , respectively.
- Then divide by , i.e., the value of .
- The last step is to apply to the value of , ending up with the final averaged angle.
Another way to understand this is to think of the average as using the Parallelogram law to the vectors of these angles.
The detailed implementation of the angel averaging is:
(1) for each angle needing to be averaged, multiply them by 2 to map back to 360 degrees scale (because the H channel is from 0-180 as implemented by OpenCV C++) and convert them from degree to radian;
(2) applying both sin and cos functions to these radians, and summing them up to get and , respectively;
(3) passing them to the
atan2 function (implemented by C++ standard library) and getting a value ranging from to ;(4) transforming the part of to by adding , but keeping the part of not changed;
(5) converting the radian (from 0 to ) to angle (from 0 to 360), then dividing it by 2 to map it back to the range of (0,180) because the H channel is ranged within it.
The overall pipeline for denoising color images is:
(1) calculating PSNR of the noisy image for comparison;
(2) (optional) converting the image from RGB color space to HSV color space, or any other color space if needed;
(3) applying the denoising algorithm to each channel of the image, or maybe just a single channel depending on specific needs (this step involves convolution, i.e. weighted averaging, so the angle averaging is needed if the denoising algorithm is applied in HSV);
(4) transforming back to RGB if it’s in other color spaces;
(4) calculating the PSNR for the denoised image and comparing it to that of the noisy image.
<ins/>
III. Experiment
(a) Basic denoising methods
Example Code (C++)
- Define kernels
- Apply kernels
Inputs
Results
According to Figure 11, it’s obvious that when the kernel size increases, the denoised image with a uniform kernel is much blur than that with a Gaussian kernel. Because the uniform kernel does unweighted average within a window, image intensity would become smoother and then result in a blurry appearance.
The Gaussian kernel has more weight on the center pixel within a window, which makes it preserve more information about the center pixel. Therefore, the denoised images with Gaussian kernels look relatively clearer.
To get the best denoising effect, a kernel size of 3 is better than other sizes. The denoised images with a kernel size of 3 are shown in Figure 12. According to the metric of PSNR, the denoised images with both kernels are better than the original noisy image. However, the PSNR of the uniform-denoised image is just slightly higher than the noisy image (less than 0.1); Gaussian denoising improves the image’s PSNR by 1.5 compared to the noisy version.
This is easy to understand because even though a uniform kernel removes some noises, it also removes some high-frequency signals. Instead, the Gaussian kernel keeps relatively more high-frequency information than the uniform kernel.
<ins/>
(b) Bilateral Filtering
Example Code (C++)
- Bilateral Kernel
- Apply the kernel
Inputs
Results
The impacts of sigma space and sigma intensity can be seen in Figure 13.
When the sigma intensity is fixed and the sigma space increases (first two rows), the image becomes clearer, but also more like an oil painting. This is because when sigma space is small, the spatial distance is more important: the farther the surrounding pixels are, the smaller the weight they have. In such cases, the bilateral filtering is prone to be equivalent to Gaussian filtering. When the sigma space is larger, the spatial distance part is gradually ignored; intensity distance is more important, so the intensity of the image will be enhanced, and so make it more like an oil painting.
When the sigma space is fixed and the sigma intensity increases (last two rows), the image becomes clearer first and then blurrier. When sigma intensity is small, the intensity distance is more important: the more different the surrounding pixels are, the smaller the weight they have. Therefore, a properly small sigma intensity is prone to preserve high-frequency signals, leading to increased PSNR while it increases. However, when the sigma intensity becomes larger, it leads to a blurry image because, in such case, the bilateral filtering will be equivalent to Gaussian filtering and produce a blurry image.
Additionally, as shown in Figure 14, when the kernel size increases, the image is denoised as well, but the PSNR decreases, which is because the details are also somewhat removed by a large kernel window. So usually, a smaller kernel size of 3 or 5 will be considered.
As shown in Figure 15, the final best bilateral-filtering-denoised image has a PSNR of 32.1935, which is much qualitatively and quantitatively better than that of basic low-pass filters. The bilateral filter has a better balance between noise filtering and high-frequency signal preservation than both Gaussian filtering and uniform filtering.
In terms of efficiency, in uniform and Gaussian filtering, the kernel weights will be pre-computed and kept fixed in convolution process. However, the kernel weights of bilateral filtering need to be computed during runtime because they are different and depend on pixel intensities. Besides, two hyperparameters of sigma space and sigma intensity also increase complexity and require tuning to acquire the best performance of denoising.
<ins/>
(c) Non-Local Means (NLM) Filtering
Example Code (C++)
- Use OpenCV’s
fastNlMeansDenoising
Inputs
Results
There are four parameters that are adjustable in this NLM algorithm: , neighbor window size , search window size , and standard deviation . Theoretically, with a larger standard deviation, the similarity computed within the neighbor window will rely more on the surrounding pixels and less on center pixels; if the standard deviation is smaller, the similarity will rely more on the center pixel.
As for the parameter of , it is also a smoothing parameter, but for the search window . With a larger value of , the algorithm will pay more attention to the patches with smaller similarities, and the weights for the more similar patches will become lower. Therefore, when there are few similar patterns, a higher value of h may be needed to assign more weight to less similar patches, thereby preventing overfitting to a small number of dissimilar ones. Like in Figure 16 (upper row), a larger of 11 produces better results than smaller ones.
The parameter of neighbor window size directly determines the window size of computing similarity. If the image does not have many similar or repetitive patterns, a smaller neighbor window would be preferable. Because if the neighbor window is large, most pixels within the window (especially the pixels closer to the window edge) cannot be helpful to compute the similarity and will further reduce the weights on actually similar patches. This aligns with the results in Figure 16 (middle row).
As for the parameter of search window size , theoretically, the larger the search window size is, the better performance the NLM is because we have a better chance to find similar patterns in a larger range in the image. But usually, the performance may not increase a lot when the search window size reaches a certain value, or even have no positive effect at all. Besides, a larger search window size will lead to a longer computation time. Therefore, a suitable value of this parameter will suffice. As shown in the last row in Figure 16, the PSNR doesn’t not change a lot with large increase of this search window size.
The tuned result of NLM is shown above, with parameters of , neighbor window size , and search window size . According to the PSNR metric, the NLM generates the best denoised image, considering the best PSNR from bilateral filtering, uniform filtering, and Gaussian filtering are 32.1935, 28.348, and 29.8797, respectively.
However, the computation consumption of the NLM is much higher than the previous three algorithms. Compared to bilateral filtering, the NLM not only computes the kernel weight in real-time, but it also has to compute the similarity between patches. This behavior can be very time-consuming and computationally intensive. Additionally, there are four hyperparameters to adjust for finding the best denoising performance. So maybe bilateral filtering would suffice for most use cases.
<ins/>
(d) Denoising for color images
Example Code (C++)
- Define uniform filtering with angle averaging
- Apply the angle-averaging-based uniform filter
- The RGB and HSV conversion is done with OpenCV’s
cvtColorfunction
Inputs
Results
Let’s start with analyzing the noise type of noisy image 1 by drawing the absolute image difference and its histograms so we can know what algorithms to use, as shown in Figure 18 and Figure 19.
It might be confusing at first glance because the differences in the G and B channels are not uniformly located and even have some shapes of the pepper. The noise in the R channel is clear though: it’s absolutely impulse noise (pepper and salt noise: pepper means 255 and salt means 0), as seen in both Figures 18 and 19. To explore more on the G and B channels, let’s draw them in a different way, i.e., drawing the positive differences and negative differences separately, as seen below.
G and B channels. It’s easy to understand that it’s Gaussian noise by looking at the shape of distributions in Figure 21 (first row) and the noise in Figure 20 (first row). The only question is why the negative differences (in the second row) are not Gaussian.
In both G and B channels, the values are close to 255 on peppers, and other values should be closer to 0. When we add Gaussian noise to the other parts (non-peppers), the negative additions can lead to the near-zero values being completely negative, and these negatives are just clipped. That’s why in G and B channels, the non-pepper parts are black instead of looking like Gaussian noise (Figure 20 second row) and the distribution shapes are not like Gaussian distributions (Figure 21 second row).
For such mixed noise of pepper and salt noise and Gaussian noise, we can use median filtering for denoising the outliers (pepper and salt noise) in the R channel first, then use bilateral filtering or NLM for denoising the Gaussian noise in G and B channels. Normally, we should apply median filtering before the bilateral/Gaussian filtering, i.e., removing outliers before low-pass filtering. But here, we can apply these filters in any order because they are applied to different channels and won’t affect each other.
As shown in Figure 22, the denoised image has a PSNR of 27.7472, with PSNRs on each channel of 30.505, 26.4581, and 26.2784, respectively. The overall pipeline is to (1) split the noisy images into RGB channels, (2) use median filtering on the R channel, (3) use bilateral filtering and NLM on G and B channels, respectively, and (4) combine three denoised channels together. The specific parameter of the median filtering is with kernel size of 3; the kernel size, and for bilateral filtering are 3, 2, and 75, respectively; the , neighbor window size , and search window size for NLM are 27, 3, and 9, respectively.
As shown in Figure 23, the denoised image’s PSNR is 22.7402 for the noisy image 2. This is done in the RGB space. Let’s transform it to HSV space and apply the filtering in the H channel (also using angle averaging).
As shown above, the resulting image’s PSNR is 27.3258 and has much better visual quality than filtering on the RGB space.
IV. Source Code
All inputs and results are in
.raw file form and we provide an online tool to visualize it: raw2image. You have to manually input the height, width, channel, bit-depth, etc. because the raw image does not contain any meta info.Colab link. Warning: the installation of OpenCV on Colab requires more than one hour. If you’d like to run the code, installing locally with for example Docker is recommended.
Prev
DIP I Image Demosaicing and Histogram Manipulation
Next
DIP III Edge Detection
Loading...