Lazy loaded imageDIP I Image Demosaicing and Histogram Manipulation

I. Motivation

Both demosaicing and histogram manipulation are important in digital image processing (DIP), which are applied to improve the visual quality of an image, either converting from gray to RGB space (demosaicing) or adjusting the brightness and contrast of images (histogram manipulation).
In the real world, thanks to digital imaging technology, the images are captured by color filter array (CFA) sensors instead of film, which is an analog medium. However, the acquired signals from sensors are either values for R channel, G channel, or B channel, because Bayer filter only allows one color of light to reach a single pixel. The goal of demosaicing is to reconstruct a full-color image by interpolation from the incomplete color arrays captured by sensors and the Bayer filter.
In the magnetic resonance imaging (MRI) field, the diffusion-weighted MRI generates both anatomical and functional images, where the latter can reflect the diffusion direction of water molecules. However, the functional images are usually in visually bad condition and make it hard to watch. The histogram manipulation can be used for adjusting the brightness and contrast of diffusion-weighted images by modifying the distribution of pixel intensities, so medical practitioners can check the quality of images better without being influenced by seemingly dark images.

II. Algorithm

(a) Bilinear Demosaicing

The basic idea behind bilinear demosaicing is to estimate the missing color values for each pixel (ex. R and G in a B pixel) by using the color information from its neighboring pixels. To implement this algorithm, we assume that the color value for each pixel is known and belongs to either R, G, or B channel.
Conventionally, the upper left pixel is indexed as (0, 0), and the ones on its right and below it are (0, 1) and (1, 0), respectively. The colors for (0, 0), (0, 1), (1, 0), and (1, 1) are G, R, B, and G, respectively, which constitute a 2 by 2 minimal unit of Bayer array as shown in Figure 1.
Figure 1. A simple demonstration of Bayer array.
Figure 1. A simple demonstration of Bayer array.
Obviously, we need to fill in red and blue colors for (0, 0), blue and green colors for (0, 1), and so on. For filling these colors, the neighbor cells for a pixel will contribute to its color values, then we use averaging (for bilinear demosaicing specifically) to compute the estimated values. There are, in total, four cases for estimating colors.
  • For a red pixel, its green and blue values are calculated as follows.
  • Similarly, the green and red values for a blue pixel are calculated as follows.
The red and blue color estimation for a green pixel is split into two sub-cases.
  • If the green pixel is located at an even column (0, 2, 4, …),
  • If the green pixel is located at an odd column (1, 3, 5, …),
For the pixels located at the edge of images, there may be no left or upper neighbors. For handling this situation, a padding technique with a padding size of 1 is applied to images before demosaicing. The method of padding could be either “reflect” or “symmetric”. In this case, the “reflect” one would be more suitable and is adopted. Because the padded values would not be the values on the edge but the values close to the edge, which makes the estimation of the color value on the edges be done by averaging the same color. The padded value can be visualized as follows.
Figure 2. Demonstration of “reflect” padding. The left column is padded pixels.
Figure 2. Demonstration of “reflect” padding. The left column is padded pixels.
Therefore, the overall algorithm steps are:
(1) Pad Bayer array image with “reflect” mode and padding size of 1;
(2) Demosaic image as mentioned above (calculating R and B for G pixel, and the like.);
(3) For each estimated R, G, or B value, put them in a new image (cannot overwrite the original one) at the corresponding location and channel.
<ins/>

(b) Histogram Manipulation

There are two methods for doing this manipulation: (1) the transfer-function-based method and (2) the bucket-filling method.
For the transfer-function-based method, the basic idea behind this is to map the non-uniform distribution of image values to a uniform distribution. So the overall range of pixel values can be distributed throughout the range from 0 – 255, instead of a small range.
  • The key first step for achieving this is to calculate the histogram of the image by counting the number of each unique value.
  • Then, transform the histogram into a probability distribution function (PDF) by dividing each count by the total number of pixels.
  • The next step is to convert PDF to a cumulative distribution function (CDF) by accumulating the count of values from 0 to 255.
  • Now we can start the most key step of this transfer-function-based method:
      • where the is the value of the original image, ranging from 0 – 255; is the CDF of the value ; is the mapping value of . With this formula, we directly map each value in the original low contrast image to corresponding value, and we get the equalized image.
      notion image
For the bucket-filling method, the equalized image could be closer to uniform distribution than the transfer-function-based method. The idea of this algorithm is to put the pixel values in original images into corresponding buckets (in total 256 buckets) evenly, which leads to an overall uniform distribution. This is also a technique for mapping pixel values.
  • The first step is to count the pixel number for each unique pixel value from 0 – 255, where the coordinates of pixels are recorded for later mapping because the same value may be mapped to different values.
  • Then we calculate the number of pixels assigned to each bucket as follows.
      • where the is originally assigned number of pixels to each bucket; the is the remaining number of pixels that are not assigned, which will be assigned to the first few buckets in sequence. Therefore, the first buckets would have  pixels, and the reminding  buckets would have  pixels.
  • After the above calculation and assignment, the final thing is to sequentially map the original pixel values to the bucket, where the index of the bucket is the new value for that pixel.
Both the transfer-function-based method and the bucket-filling method won’t lead to out-of-range mapping, which means that the original 0 – 255 values will also be mapped to a range of 0 – 255 and won’t cause overflow or underflow.
<ins/>

(c) Contrast Limited Adaptive Histogram Equalization

According to the paper Contrast Limited Adaptive Histogram Equalization (CLAHE), it’s a technique used to enhance the contrast of an image for medical images originally, while limiting the amount of noise in uninteresting areas (background) that can occur with traditional histogram equalization. It involves two versions: AHE and CLAHE.
In the original AHE algorithm, it divides the image into small rectangular regions called "contextual regions" and then performs histogram equalization on each of them independently. This helps to prevent the negative effects of global information in the image and creates a more visually pleasing result within local regions. To avoid the visibility of boundaries between regions, a technique of bilinear interpolation is applied. However, the AHE algorithm will also cause even worse results in uninteresting areas like the background because the noises are amplified without the global information.
To handle this situation, the CLAHE algorithm further utilizes “clip limit” (or “contrast factor”) and limits the contrast enhancement in these backgrounds or so-called “homogeneous areas”. The contrast factor is defined as a multiple of the average count of all values in the histogram. By clipping the histogram with the contrast factor and assigning these clipped pixels with other values evenly, the slope of the histogram will be less steep.
Let’s say we have a 256x256 grayscale image where 128x256 pixel values are 0 and another 128x256 pixel values are 1. All other values from 2 to 255 are counted 0. So the average count is (128x256 + 128x256 + 0 + … + 0)/256 = 256. So if the clip limit is set to be 1, then all values that have count more than 256 will be clipped to be 256. If clip limit is set to be 3, then all values that have count more than 768 will be clipped to be 768.
This technique can be effective in homogeneous areas and limits the contrast enhancement of the noises. When the contrast factor is very high like more than 1,000, the histogram will not be clipped and the CLAHE will be equivalent to the AHE algorithm; When the contrast factor is equal to 1, all pixels will be evenly distributed throughout the range of 0 – 255, after which the AHE will not have contrast enhancement effect on the region because the mapping would be self-to-self and nothing changes.
🤔
If the pixels are redistributed, then some pixels in the pure black background are mapped to very bright values, doesn’t this mean the noise is still enhanced?
Well, no. The redistribution operation just aims to generate a redistributed CDF; this new CDF will be used for AHE (i.e., histogram mapping). After this, the acquired mapping will be applied to the original pure black background, which only contains near-zero values. Because this mapping is almost self-to-self with a clip limit of 1, i.e., 0 to 0, 128 to 128, and 255 to 255, it makes the background unmapped and noise unamplified.
The overall steps behind the CLAHE algorithm are:
(1) splitting the original image into several contextual regions;
(2) for each region, clip the histogram according to the pre-designated clip limit hyperparameter;
(3) using a histogram equalization algorithm to map the values in the clipped histogram to new values.
We will use the open-source code for this CLAHE algorithm implemented by OpenCV, where the number of tiles and the clip limit are hyperparameters to be adjusted.
<ins/>

III. Experiment

(a) Bilinear Demosaicing

Example Code (C++)
  • Padding: Here we only define the reflect mode for the bilinear demosaicing
    • Bilinear demosaicing
      Inputs
      notion image
      Results
      Figure 3. Left is the original image in RGB. Right is the demosaiced image with the bilinear demosaicing.
      Figure 3. Left is the original image in RGB. Right is the demosaiced image with the bilinear demosaicing.
      As shown in Figure 3, the result produced by bilinear demosaicing has a noticeable artifact of color distortion, especially at the edges (like on the edge of a house or leaves). This is because the bilinear demosaicing only considers the nearest surrounding pixels for estimating the color value. Additionally, it estimates the color for a pixel with the pixels that have the same color, i.e., calculates the green color value based on only surrounding green pixels. This behavior didn’t take into account the relationship between colors.
      A more advanced algorithm can be applied to improve the demosaicing performance: Malvar-He-Cutler (MHC) Demosaicing. The MHC demosaicing method considers more pixels around and utilizes spatial information. For example, when estimating G value, depending on its location, R value or B value will also be used for interpolation. This can produce much better results than the simple bilinear demosaicing, especially for the problem of color artifacts at edges.
      <ins/>

      (b) Histogram Manipulation

      Example Code (C++)
      • Define functions of calculating histogram, PDF, CDF, transfer function, bucket filling, etc.
        • Apply these functions
          Inputs
          notion image
          Results
          Figure 4. the histograms of the original image.
          Figure 4. the histograms of the original image.
          Figure 5. Left is the enhanced image with the transfer-function-based method; right is the enhanced image with the bucket-filling method.
          Figure 5. Left is the enhanced image with the transfer-function-based method; right is the enhanced image with the bucket-filling method.
          Figure 6. Left is the transfer function of the transfer-function-based method; middle is the cumulative histograms before enhancement with the bucket-filling method; right is the cumulative histograms after enhancement with the bucket-filling method.
          Figure 6. Left is the transfer function of the transfer-function-based method; middle is the cumulative histograms before enhancement with the bucket-filling method; right is the cumulative histograms after enhancement with the bucket-filling method.
          <ins/>

          (c) Contrast Limited Adaptive Histogram Equalization

          Code Example (C++)
          • Here, we mainly define the util functions: RGB2YUV or getXFromYUV (so we can extract luminance) and the like, because, for RGB image, we need to transform it into grayscale first and then apply the algorithms.
            • Apply the CLAHE (from OpenCV C++)
              Inputs
              notion image
              Results
              Figure 7. Left is the transfer-function-enhanced result; right is the bucket-filling-enhanced result.
              Figure 7. Left is the transfer-function-enhanced result; right is the bucket-filling-enhanced result.
              Figure 8. Left is the CLAHE-enhanced image with a grid size of 30 and a clip limit of 2; right is the CLAHE-enhanced image with a grid size of 30 and a clip limit of 3.
              Figure 8. Left is the CLAHE-enhanced image with a grid size of 30 and a clip limit of 2; right is the CLAHE-enhanced image with a grid size of 30 and a clip limit of 3.
              Figure 9. Cumulative histograms of Y channel of input image in YUV color space.
              Figure 9. Cumulative histograms of Y channel of input image in YUV color space.
              Figure 10. The results of CLAHE with fixed clip limit of 2, and grid size of 1, 5, 10, 15, 20, 25, 30, 35, and 40 (in the order from left to right, from top to bottom).
              Figure 10. The results of CLAHE with fixed clip limit of 2, and grid size of 1, 5, 10, 15, 20, 25, 30, 35, and 40 (in the order from left to right, from top to bottom).
              As shown in Figure 7, both the transfer-function-enhanced and bucket-filling-enhanced images look visually bad.
              But technically, why so much dark area? There are not many dark pixels in the original image, and most are closer to the white side, which can be seen from its cumulative histograms on the Y channel (in YUV color space), as shown in Figure 9. After applying the histogram manipulation, many pixel values are mapped to relatively lower values (darker pixels).
              As shown in Figure 8, the result from the CLAHE algorithm is much more visually pleasant. IMO, a relatively lower clip limit of 2 generates better visual quality. Let’s set the clip limit to 2 and test the impact of grid size on the image.
              As observed in Figure 10, a smaller grid size leads to a darker image, and a larger grid size leads to a brighter image. This is easy to understand.
              • When the grid size equals one (i.e. the image is not split at all), CLAHE becomes more like the transfer function method, leading to dark areas;
              • when the grid size becomes larger and larger, the local regions increase, and the enhancement effect will be better by focusing locally and suppressing background noise.
              • But when the grid size exceeds some thresholds, the pixels in each region are very few. In such a case, the CDF will look like a trapezoid, making all local values mapped to the highest 255 values. This is why, in the last image in Figure 10, the homogeneous area (foggy area) is nearly white.

              IV. Source Code

              All inputs and results are in .raw file form and we provide an online tool to visualize it: raw2image. You have to manually input the height, width, channel, bit-depth, etc. because the raw image does not contain any meta info.
              Colab link. Warning: the installation of OpenCV on Colab requires more than one hour. If you’d like to run the code, installing locally with for example Docker is recommended.
               
              Prev
              Awesome-AI-Examples
              Next
              DIP II Image Denoising
              Loading...
              Article List
              LLM Learning Roadmap
              ✨ Awesome-Anything
              Lazy loaded image Awesome-AI-Tutorials
              Lazy loaded image Awesome-AI-Researchs
              Lazy loaded image Awesome-GitHub-Repos
              Lazy loaded image Awesome-AI-Examples
              🖼️ Digital Image Processing
              Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
              Lazy loaded image DIP II Image Denoising
              Lazy loaded image DIP III Edge Detection
              Lazy loaded image DIP IV Digital Half-toning
              Lazy loaded image DIP V Geometric Image Modification
              🍃 LLM Components
              Lazy loaded image 1.1 Tokenization
              Lazy loaded image 1.2 Embedding
              Lazy loaded image 1.3 Attention
              Lazy loaded image 1.4 FFN, Residual Addition, and LN
              Lazy loaded image 1.5 Positional Encoding
              Lazy loaded image 1.6 Architecture & Decoding Policy
              🌱 LLM Pre-training
              Lazy loaded image 2.1 Pre-training Definition
              Lazy loaded image 2.2 Pre-training Data
              Lazy loaded image 2.3 Pre-training Workflow
              Lazy loaded image 2.4 Continued Pre-training (CPT)
              Lazy loaded image 2.5 Pre-training Evaluation
              ☘️ LLM Post-Training
              Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
              Lazy loaded image 3.2 Reinforcement Learning
              Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
              Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
              Lazy loaded image 3.5 Direct Preference Optimization (DPO)
              🍀 LLM Popular Models
              Lazy loaded image 4.1 BERT & Variants
              Lazy loaded image 4.2 PaLM Series
              Lazy loaded image 4.3 GPT Series
              Lazy loaded image 4.4 LLaMA Series
              Lazy loaded image 4.5 GLM Series
              Lazy loaded image 4.6 Qwen Series
              Lazy loaded image 4.7 DeepSeek Series
              Lazy loaded image 4.8 MOE Series
              Lazy loaded image 4.9 Other
              🪴 LLM Applications
              Lazy loaded image 5.1 Prompt Engineering
              Lazy loaded image 5.2 LLM-based Agents
              Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
              Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
              Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
              🌿 LLM Optimization
              Lazy loaded image 6.1 GPU Usage Analysis
              Lazy loaded image 6.2 Latency Analysis for Inference
              Lazy loaded image 6.3 FlashAttention
              Lazy loaded image 6.4 PagedAttention
              Lazy loaded image 6.5 Packing for Efficient Training
              Lazy loaded image 6.6 PEFT
              🌾 LLM Compression
              Lazy loaded image 7.1 Quantization
              Lazy loaded image 7.2 Pruning
              Lazy loaded image 7.3 Knowledge Distillation
              🌵 LLM Hands-on Practice
              Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
              Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
              Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
              Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
              Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
              🌴 LLM Must-read Papers
              Lazy loaded image 9.1 Technical Reports Explained
              Lazy loaded image 9.2 MedLLM
              Lazy loaded image 9.3 o1 Techinical Paper
              Lazy loaded image 9.4 Prompting
              Lazy loaded image 9.5 LLM-as-Judge
              🌳 LLM Q&A
              Lazy loaded image 10.1 Why GRPO training loss starts at 0
              Lazy loaded image 10.2 VLM-R1 Reproduction Results
              Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
              Lazy loaded image 10.4 Left Padding or Right Padding?
              Lazy loaded image 10.5 What is QK-Norm?
              Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
              Lazy loaded image 10.7 Dropout and DropPath
              🐝 VLM Image Encoders
              Lazy loaded image 11.0 Introduction
              Lazy loaded image 11.1 Vision Transformer
              Lazy loaded image 11.2 CLIP
              Lazy loaded image 11.3 BLIP
              Lazy loaded image 11.4 EVA
              Lazy loaded image 11.6 BLIP-2
              Lazy loaded image 11.7 SigLIP
              📝 MISC.
              🤔 Large Language Models Pass the Turing Test.
              Lazy loaded image OpenAI ChatGPT Progression
              ⏳ Claude Sonnet 4 Test with Sandtris
              🔍 What DeepSeek-R1's Surge Teaches Us About SEO
              💰 CivitAI’s Payment Issue
              🎓 arXivSub, an Academia Assistant