Want to know how the image output by the image sensor can be turned into the colorful picture we see through the FPGA? Today we will take you to find out.
01
What are Raw pixels?
The photos we usually see are the result of ISP and a series of image processing, so what does the original pixel output by the image sensor look like? The beautiful picture you see is like this:
figure 1
But in fact, the output of the sensor is similar to this:
figure 2
That's right, this is what we often call the Bayer mode composed of Raw pixels. Many people who play with SLRs know that most SLR cameras support the Bayer format Raw pixel mode to output photos, because Raw pixels are the most primitive image information obtained by the sensor. It is convenient for us to do later processing. Today we are going to talk about how to process Raw pixels with FPGA. Before that, let's have a general understanding of the Raw pixel and Bayer format.
We can think that each pixel is composed of three color components of red, green and blue, commonly known as RGB. For a Raw pixel, you can think of it as a pixel that contains only one color component. For Bayer mode, let's look at a set of images. If an image only displays its even rows and even columns, and only displays its red component, it will look like Figure 3:
image 3
If only the odd rows and odd columns are displayed and only the blue component of it is displayed, it looks like Figure 4:
Figure 4
The remaining pixels show only their green component, as in Figure 5.
Figure 5
We combine these three components to see the picture with color mosaic in Figure 2, which is what we often call the Bayer format. Of course, the raw pixels output by the sensor are not obtained in this way. Here is just a chestnut to explain how the raw pixels are distributed in the Bayer format.
Next, we will talk about how to use FPGA to do De-Bayer in real time.
02
How to do De-Bayer with FPGA?
The so-called De-Bayer here is to convert the Bayer format into a normal RGB format. We employ bilinear interpolation, an algorithm that balances computation and image quality. The bilinear interpolation method uses the color components of its own adjacent 4 pixels to supplement the two missing color components.
The output pixels of the sensor are usually scanned line by line, so firstly, FPGA needs to be used to build a 3*3 template.
Image 6
As shown in Figure 6, a 3*3 template is constructed through a shift register and FIFO. The function of the FIFO is to do the image line buffer, so the size of the FIFO needs to be determined according to the resolution of the image and the pixel width. The Lattice Crosslink chip has 20 9K distributed RAMs on-chip, which can be used for line caching.
The bilinear interpolation algorithm calculates the average value of the left and right upper and lower pixels of the pixel as the fill color. For the sensor output by progressive scanning, the pipeline processing method can be used for addition and division operations, so as to achieve good performance. One thing to pay attention to here is the processing of the boundary, because we cannot build a 3*3 template for the boundary pixel. Here, the adjacent pixel color interpolation method can be used for the boundary pixel.
At present, many sensors are MIPI interfaces, and Crosslink itself has a powerful MIPI bridging function. Adding De-Bayer makes it easy for Crosslink to realize CSI-2 input and DSI output. In addition, Crosslink can handle bilinear De-Bayer at a speed of 148.5 MHz, which means that it can handle CSI-2 to DSI at 1080p 60 frames.
Solar PV Power Monitoring Solution
Solar PV Power Monitoring Solution,DC Multi-Circuits Monitoring Device,DC Dual-circuits Monitoring DIN Rail Energy Meter,Single-phase PV Reflux Monitoring Meter,Three-phase PV Reflux Monitoring Meter,AC Multi-function Smart Energy Meter
Jiangsu Acrel Electrical Manufacturing Co., LTD. , https://www.acrel.com.pk