Color filters for next generation CMOS image sensors (CIS)

Lev Klibanov Ph.D. Independent Consultant and Paul Boldt Ph.D. Founder, ned, maude, todd & rod inc.

Introduction

Consumer and industrial cameras are mainstream components in many products: IoT, surveillance, smartphones, robots, etc., and most of these cameras use an image sensor chip based on CMOS technology or, as it is called, a CMOS Image Sensor (CIS). CISs have seen considerable technological evolution [1] [2], from front to backside illumination, the number of pixels per CIS, pixel design, and pixel size (the current smallest pixel being 0.7μm) [3].

Each pixel has a photodiode that is color blind i.e. it only captures the total intensity of light impinging on its surface. In order to obtain a full-color image, most sensors implement a color filter array (CFA) to record the primary colors of light. After the CIS records the colors, they are combined to create a full-color picture [4].

Today’s preferred approach for CIS color separation is polymer/dyed color filters (CF). The typical pattern of the CFA is Bayer’s color pattern (see image below). A given pixel incorporates 4 sub-pixels such that there are two green components, a blue component and a red component.

Figure 1. Typical CFA with Bayer’s color pattern [5].

Like CMOS technology in general, CISs have undergone an incredible amount of scaling. The resulting reduction in pixel size has allowed a decrease in CIS footprint and increased resolution. However, the reduced pixel size has brought forward some CF challenges. In addition to general photodiode sensitivity the challenges include:

1.????Photoresponse nonuniformity of the CF due to an uneven distribution of dyed particles

2.????Degradation of the spectral properties of the polymer CF under constant UV illumination

3.????Signal noise (up to 43% of all crosstalk in CIS [6]) due to CF color (spectral) crosstalk

Let’s look at color cross-talk a bit more. Color crosstalk originates from a wide optical transmission of the CF, as a function of the wavelength. A typical polymer CF spectral response [7] is presented in Figure 2. Each of the CFs has transmission tails or quantum efficiency in areas of the spectrum outside of the true color regions. Thus, the polymer CF allows the transmission of other colors.

Figure 2. Spectral Response of typical RGB CFs [7].

CF technologies

Moving forward the industry will need to address color crosstalk. To date, few CF designs for mitigating the spectral crosstalk have been discussed in the literature. Here is a brief synopsis of known and proposed CF technologies.

1. Pigment/Dye Filters

These are the currently used technology. An exemplary structure is shown in Figure 1.

Pros: Dye CFs implement a traditional absorptive structure. Lithography technology has been used to fabricate these filters for many generations of CIS technology. Importantly, dye filters are relatively inexpensive.

Cons: Dye CFs are more susceptible to spectral crosstalk as pixel size decreases, they can fade over time and they require at least three steps of the lithography process.

2. Plasmonic Filters [8]:

Plasmonic-based CFs are fabricated from thin metal films, where a given film has an array of holes for each color, where the holes associated with each color have a different diameter. Generally, the diameter increases with increasing wavelength. Typical plasmonic filters are presented in Figure 3.

Pros: Plasmonic color filters can be fabricated from a single metal film, allowing a reduced manufacturing time and cost. Also, plasmonic color filters show improved reliability against ultraviolet illumination.

Cons: The hole array of plasmonic filters shows a low transmission of approximately 40?50%. Further, there is still some cross-talk between colors in the visible light spectrum [9].

Figure 3. Typical plasmonic filters made of metal film for red (d=240 nm), green (d=180 nm) and blue (d=140 nm) [8].

3. Dielectric Subwavelength Grating Filters

Dielectric subwavelength grating CFs are fabricated from either silicon or metal i.e opaque or semitransparent material and mounted on a transparent dielectric [10].

Pros: Dielectric subwavelength grating CFs can be formed with as few as two films. Therefore, manufacturing time and cost are reduced. Further, the dielectric CFs possess better reliability against ultraviolet illumination and at elevated temperatures.

Cons: Dielectric CFs show good transmission of red and green but provide lower transmission of blue (~0.5). However, the cross-talk between the colors for the transmission visible spectrum is still there (for example, see figure 1b in [10]).

Figure 4. Scanning Electron Microscopy (SEM) images of dielectric color filters [10]: Green d=140 nm, Red d=90 nm, blue d=240 nm

4. Multilayer metal/dielectric filters:

Metal/dielectric CFs comprise a multilayer stack forming an optical (interference) cavity or Fabry-Perot structure. Transmitted light is tuned by changing the number of layers, their thickness, or the materials.

Pros: Metal/dielectric multilayer CFs can be manufactured in half the thickness of the color dye filters without an extra infrared cut-off filter.

Cons: The Fabry-Perot structure requires additional lithography steps and fine-tuning of the layer. Both add cost. Further, again, there is a low (~0.5) transmission of blue light.

Figure 5. Schematic cross-section of patterned RGB metal/dielectric filters [11]

5. Pigment/Dye Filters with beyond RGB color pattern:

Adopting a different color pattern for dye CFs is another approach for reducing pixel color cross-talk and increasing the color transmission. Current RGB filters in a Bayer design can be changed to a more color-friendly Cyan, Yellow, and Magenta (CMY) or Red-Yellow-Yellow-Magenta (RYYM) . For instance, a large advantage of CMY filters is improved colour transmission compared to a conventional RGB filter [12].

Figure 6. CMY filter transmission curve [12].

Pros: CMY CFs display a high transmission in visible light wavelengths allowing CIS with a CMY filter array to be more effective in the low-light environment. This mitigates one of the challenges of scaling.

Cons: CMY CF arrays require a more complex de-mosaicking algorithm compared to standard RGB arrays to obtain a natural color image.

Patent Insights

If dye RGB CFs are hitting their limit, the obvious question is what color filter technology will replace it? One way to assess possible technologies is to access the patent “activity” associated with each. What technology shows the most activity?

The first step: determine the number of patents related to CFs over time. For this we used ?“image sensor” and “color filter” as keywords and CPC classification H04N. Searches were conducted at the Lens.org i.e. “www.lens.org” patent database [13]. The start date was set at 1990, estimating when work related to the first color commercial camera, the Kyocera Visual Phone VP-210, released in Japan in May 1999 [14].

The total number of filed, published, and granted patents between 1990 and the time of writing is 6,161. Figure 7 plots the number of granted patents over this timeline.

Figure 7. Patent documents from 1990 to today [13].

Search data was also organized by Applicants. The results are presented in Figure 8. There were no surprises here with the top 10 Applicants for the CIS CFs being known image sensor manufacturers i.e Samsung, OmniVision, Canon, etc. or material suppliers, such as ?Fujifilm.

Figure 8. Top 10 Applicants of CIS CF-related patent documents [13].

The next step: parse the data with respect to CF technology. Where are the highest number of patents i.e. what CF technology is seeing the most research, is of the most interest? The same basic keywords and CPC classification (H04N) criteria were used.?Then, specific technologies were added. These included, for example, “interference filters” “subwavelength gratings” and “plasmonic filters”.?Returns as a function of technology are shown in Figure 9. It is noted that Figure 9’s vertical axis is a “log” scale, so each gradation represents a 10X increase in the number of patents.?CF technologies including “interference filters,” “subwavelength gratings,” and “plasmonic filters,” top out at approximately 100 patents i.e. only a small fraction of all granted patents related to CF technology.

Figure 9. Filter technology vs. number of patents.

The data is telling us more. First, there are approximatly ~4X more patents associated with CMY/RYYM CF technology than plasmonic CFs i.e. the nearest challenger. This suggests CMY/RYYM CFs are of the most interest. Second, CMY/RYYM represents only a fraction of the total CF patents. Since the total CF number spans 30 years, a significant number of current generation i.e. pigment/dye CFs and earlier concepts are likely represented in the total. In the end, the data is telling us to focus on CMY/RYYM for the next generation CFs.

The next question: does industry agree with the patent numbers? Are there any indications within industry that CMY/RYYM CFs will be the next generation CFs. The short answer: there are.

All of the above-mentioned CF technologies offer improved performance, but they require several additional process step, which increases the final product cost. A quick search of CF information in the latest smartphone specifications, or analyzed CISs from Reverse Engineering (RE) houses, indicate industry is turning to CMY/RYYM technology.?This agrees with the patent data and reinforces the importance of cost.

The recent Huawei P30 smartphone [15] now uses a Red-Yellow-Yellow-Magenta (RYYM) CF instead of a dye/RGD CF. Huawei indicates their RYYM-based sensor collects up to 40% more light than the green filters in RGGB. OmniVision has also moved away from dye/RGB. They however turned a CMY CF pattern in their latest CIS (see Figure 10).

Figure 10. Omnivison CIS color filter array - CMY

Conclusion

CMOS Image Sensors for mobile applications (think smartphones) have experienced tremendous evolution over the years. Today they provide incredible resolution, low light intensity operation and improved color resolution. However, today the current CF standard of dye/RGB filters is showing limitations. A number of technologies were proposed to improve the CF wavelength resolution, lifetime and reduce the manufacturing cost. The feasible technology, for now, is still based on dye color filters, but with a different pattern of colors, like RYYM or CYM. These color patterns can easily be adopted in current fabrication process flows, while offering better color separation, higher light transmission and reduced color cross-talk between adjacent pixel cells. Patent data and industry agree on this point.

Bibliography

[1] SK Hynix, "Evolution of Pixel Technology in CMOS Image Sensor," SK Hynix, 28 01 2020. [Online]. Available: https://news.skhynix.com/evolution-of-pixel-technology-in-cmos-image-sensor/. [Accessed 19 06 2020].

[2] M. Lapedus, "Scaling CMOS Image Sensors," Semiconsuctor Engineering, 2020.

[3] Samsung, "Samsung Introduces Industry’s First 0.7μm-pixel Mobile Image Sensor," Samsung, 24 09 2019. [Online]. Available: https://news.samsung.com/global/samsung-introduces-industrys-first-0-7%CE%BCm-pixel-mobile-image-sensor. [Accessed 19 06 2020].

[4] T. V. W. &. G. G. KARIM NICE, "How Digital Cameras Work," howstuffworks, [Online]. Available: https://electronics.howstuffworks.com/cameras-photography/digital/digital-camera4.htm#:~:text=In%20order%20to%20get%20a,to%20create%20the%20full%20spectrum.&text=Another%20method%20is%20to%20rotate,front%20of%20a%20single%20sensor.. [Accessed 12 09 2020].

[5] "Phone Vision 06 – RGB Color Intensities," Azzlsoft, [Online]. Available: https://azzlsoft.com/tag/bayer-filter/. [Accessed 21 June 2022].

[6] C.-H. K. a. etc., "Improvement of Crosstalk on 5M CMOS Image Sensor with 1.7x1.7μm2 pixels," Proceedings of Integrated Optoelectronic Devices 2007, vol. 6471, 2007.

[7] H. R. e. al, "CMOS imager technology shrinks and image performance," IEEE Workshop on Microelectronics and Electron Devices, pp. 7-18, 2004.

[8] Y. e. al., "Plasmonic Color Filters for CMOS Image Sensor Applications," Nano Letters , no. 12, 2012.

[9] Sozo Yokogawa, "Plasmonic Color Filters for CMOS Image Sensor Applications," Nano letters, vol. 12, no. 8, p. 4349–4354, 2012.

[10] Yu Horie, "Visible Wavelength Color Filters using Dielectric Subwavelength Gratings for Backside-illuminated CMOS Image Sensor Technologies," Nano Letters, Caltech Library, 2017.

[11] L. Frey, "Color filters including infrared cut-off integrated on CMOS image sensor," OPTICS EXPRESS, vol. Vol. 19, no. No. 14, p. 13073, 4 July 2011 .

[12] Fijifilm, [Online]. Available: https://www.fujifilm.com/us/en/business/semiconductor-materials/image-sensor-color-mosaic/cmy/applications. [Accessed 03 11 2021].

[13] https://www.lens.org/.

[14] "Camera phone," Wikipedia, [Online]. Available: https://en.wikipedia.org/wiki/Camera_phone. [Accessed 26 11 2021].

[15] Prasad, "How the RYYB sensor on the Huawei P30 Pro works - GSMArena.com news," GSMArena.com, 13 May 2019. [Online]. Available: https://www.gsmarena.com/how_the_ryyb_sensor_on_the_huawei_p30_pro_works-news-37007.php. [Accessed 16 11 2021].