WO2014132096A1

WO2014132096A1 - Block color conversion

Info

Publication number: WO2014132096A1
Application number: PCT/IB2013/000602
Authority: WO
Inventors: Alexandre Delattre; Janos BOUDET
Original assignee: Nintendo European Research And Development
Priority date: 2013-02-26
Filing date: 2013-02-26
Publication date: 2014-09-04
Also published as: EP2962449A1

Abstract

Improved RGB to YUV conversion with chroma downsample process that happens at the first stage of many (most) image and video compression techniques provides a measurably better quality than the standard conversion by exploiting the fact that decompressors saturate the output RGB values back into their allowed ranges to thereby optimize the intermediate lower resolution color values. Example non- limiting implementations output results equal or nearly equal to the standard conversion when the latter performs well, has a computation cost that is less than a few times the standard conversion's computation cost so as to allow its implementation in similar hardware, reaches a quality level close to that of the full brute force optimization, and applies to all quasi affine colorspace conversions with chroma downsampling.

Description

Block color conversion

This technology relates to digital image processing, and more particularly to image block color conversion systems and methods. Still more particularly, the technology herein relates to improving RGB to YUV or other color standard conversion with chroma downsample. In more detail, the technology herein provides improved RGB to YUV conversion with chroma downsampling that happens at the first stage of many (most) image and video compression techniques. It has applicability in computer graphics, video processing and image processing among others. BACKGROUND AND SUMMARY

RGB Color Space

We learned in grade school that we could make different colors by mixing together different amounts of the three primary colors (red, green and blue or "RGB"). Adding red and green light together produces yellow, orange or brown. Mixing green and blue light produces cyan. Mixing red and blue light produces purple or magenta. Mixing the same amount of all three results in white or grey.

The RGB color model is widely used for electronic color display and image representation technology in computers, digital video, smart phones, high definition television and many other applications. Typically, the intensity of each color component is represented by a digital number or value. The commonly-used sRGB color standard for consumer electronics represents each color component with 8 digital bits. The gamut of chromaticities that can be represented in sRGB is the color space defined by the Red/Green/Blue primary colors. For non-negative values, this sRGB representation is unable to represent colors outside of this color space, which is well inside the range of colors visible to a human.

YUV Color Space

But there are other ways to represent color. Historically, when commercial black-and-white television was first developed, only brightness (luma or "Y") was transmitted. Television engineers wanted to add color while maintaining backwards compatibility with the millions of already-existing black-and-white television receivers all over the world. The solution they came up with: supply two additional chrominance "difference" signals ("U" and "V") that would complement the existing Y luma signal to supply additional color information. The resulting "YUV" color representation has been used for many years for conventional television broadcast standards including NTSC, SECAM and PAL. It turns out that there are psychovisual advantages to separating luma from color components, and YUV has turned out to have advantages beyond backward compatibility.

A benefit of representing color in YUV is that by separating the color and brightness channels, it becomes possible to easily process those color channels individually— for example to apply certain kinds of corrections or compressions. Generally speaking, YUV format allows for higher compression rates without a proportionately high loss of data, as the U and V portions can be highly compressed and computed from the non- or lowly-compressed Y portion. It is therefore now common to convert between RGB format (used by many hardware manufacturers) and YUV pixel formats (used by the JPEG and MPEG compression methods).

Convert Between RGB and YUV

The basic equations for converting between YUV and RGB are well known and very straightforward: Y=0.299R'+ 0.587C+ 0.114B'

U= - 0.147R'- 0.289G' + 0.436B' = 0.492(B' - Y)

V= 0.615R' - 0.515G' - 0.100B' = 0.877(R' - Y)

R'= Y + 1.140V

G'= Y - 0.395U - 0.581V

B'= Y + 2.032U

See e.g., Jack, Video Demystified: A Handbook for the Digital Engineer (Elsevier 2007). Here is another way to express the conversion from RGB888 to YUV is:

Y = ((66 * R + 129 * G + 25 * B + 128) » 8) + 16

U = ((-38 * R - 74 * G + 112 * B + 128) » 8) + 128

V = ((112 * R - 94 * G - 18 * B + 128) » 8) + 128

To convert from YUV to RGB888, the following equations can be used:

R = clip(( 298 * C + 409 * E + 128) » 8)

G = clip(( 298 * C - 100 * D - 208 * E + 128) » 8)

B = clip(( 298 * C + 516 * D + 128) » 8)

where C = Y - 16, D = U - 128 and E = V - 128 and clip() denotes clipping a value to the range of 0 to 255.

Generally speaking, converting from RGB to YUV involves converting one RGB image into three separate images: a Y image, a U (difference) image and a V (difference) image. The Y image in many contexts is the same size and resolution as the original color image but contains only luma (brightness) information and no color information. The color information is contained in the additional U and V images. See Figure 1.

As indicated by the "clip()" functionality above, standard decoders often saturate the decoded YUV->RGB values back into the valid RGB range. They do this because even the standard conversion process sometimes produces YUV values that fall outside of the RGB range when decoded. Thus, many common techniques truncate such "out of range" values so they don't cause problems and result in RGB representations display equipment can ultimately use and process.

Downsampling

As alluded to above, especially when the color conversion is for compression purposes, it is often desirable to downsample the U and V components - or the components for which our eyes have lower spatial frequency response. "Downsample" in this context reduces the number of bits required to represent the U and V color information. Often, the RGB color information may be downsampled to provide a YUV4:2:0 image or any other form of downsampling. YUV420 is a planar format, meaning that the Y, U, and V values are grouped together instead of being interspersed. The U and V are generally both downsampled, whereas in many applications the Y component will not be downsampled. By grouping the downsampled U and V values together, the image becomes more compressible in many practical implementations - and the higher resolution Y luma value continues to provide higher resolution image detail whereas the downsampled U, V values supply color information with less resolution than provided by the original RGB representation.

Hence, such downsampling or sub-sampling is a lossy process that decreases the color resolution of the U and V color information and thus the amount of information that needs to be transmitted and/or stored. This loss of color information allows higher compression ratios to be achieved while usually maintaining acceptable image color quality. In YUV4:2:0 (a common representation used in MPEG), the U and V color values are resampled to provide only ¼ the number of color values in each of U and V as compared to the Y luma values. See prior art Fig. 2. The resulting YUV values are then upsampled after storage or transmission to reconstruct RGB values for display or other processing. Meanwhile, the lossy UV compression described above gives a 2x compression ratio (in this particular non-limiting example) that will be a starting point for any subsequent compression/decompression process to achieve further compactness for data storage and/or transmission.

Downsampling Reduces Resolution of Intermediate Color Space Representation Because of the downsampling process in an scRGB or sRGB 2x2 color block to

YUV420 color conversion, the YUV420 color space has lower resolution than the original RGB 2x2 color space. This means some valid colors in the original RGB 2x2 color space cannot be represented in the lower-resolution YUV420 color space. If the YUV240 representation of a particular 2x2 color block is a close approximation of the scRGB representation, a human who perceives the result will not notice the difference. However, there can be some cases where the difference is quite noticeable. For example, if the original RGB 2x2 color block was being used to encode contrasting colors (e.g., bright red on a black background), then the loss of color resolution due to the conversion may generate a very noticeable artifact of the color conversion process. This is undesirable.

It is possible to use Downsampled Reduced Color Resolution to Achieve Higher Fidelity

Because of the particular nature of YUV420, we realize that even though the lower-resolution YUV420 color space has fewer potential values than scRGB or sRGB 2x2 for example, there are nevertheless some values in the YUV420 color space that do not exist (i.e., are undefined) in the higher-resolution RGB 2x2 color space. We can take advantage of this by intentionally converting certain RGB block color values into YUV420 values that would otherwise have no direct mapping when reconverted from YUV back to RGB, but which when further processed by conventional existing YUV-to-RGB conversion hardware and software, will provide resulting RGB values that are closer to the original image RGB values than would be obtained if only the subset of the YUV color space representing direct mapping from RGB to YUV were used initially.

Thus, in contrast to conventional wisdom, the example non-limiting technology herein exploits the characteristic that color block conversion can produce YUV values that fall outside of the RGB range when decoded in order to provide a more optimized and higher fidelity overall block color conversion.

Such improved conversion can be applied selectively, or it can be applied to all blocks in the original image. Even if applied to all blocks, the non-limiting example improved conversion will not degrade results as compared to conventional conversion techniques, and will provide noticeable improvements especially for certain image blocks such as for example high contrast situations (e.g., red on black).

The example non-limiting implementation works on block+color spaces and helps wherever a system will perform a colorspace conversion followed by or combined with an upsample. The example non-limiting technique creates YUV420 block values that do not lie in the traditional output range of standard downsamplers and exploits the fact that standard decoders project out of range reconstructed RGB values back into their normal range. While some out of range values may already get produced by standard encoders (and this is good, because existing decoders are therefore already capable of handling it), the example non-limiting technology herein produces values that are even more systematically out of range to improve and optimize image quality. More Detailed Example

In more detail, converting a w x h RGB image with downsampling into YUV₄₂₀ thus results in three images: aw x h Y image and two ~ ~ images, that can be independent in memory, or interlaced. Figure 1 thus shows a w x h RGB image on the left-hand side being converted into a w x h Y image on the right-hand side plus two additional ¼-sized U & V color difference images. A full range of colors (within the limitation of the size of the YUV₄₂₀ color space) can be obtained when these three apparently-monochromatic images are recombined in a decoding process.

The non-limiting example color conversion task global across the image is broken up into smaller independent (local) tasks based on blocks or subsets of the color image. As one non-limiting example, a local task considers a 2 x 2 RGB block of the original image, and converts it into six values (four Y values, one U value and one V value):

\rgb₀ rgb Bloc convert

The canonical way to convert a four pixel (2 x 2 RGB values £ RGB ) RGB block into a YUV output with downsampled UV is to apply the quasi affine transform (note that we also discuss below the case of bilinear upsampling).

The standard RGB →YUV4?o matrix conversion is

Note that matrix M₄ has many zeroes and symmetries, so the actual full matrix multiplication is not carried out in efficient real world implementations. Also note that this matrix is the orthogonal projection matrix that projects RGB⁴ onto M₄-R₄-YUV₄₂o as vector spaces using the orthogonality defined by the following YUV dot product applied to RGB vectors:

(RGB x RGB) -→ M

(α,β) = α^τ -Q-β

where Q = (M^T■ M)

( y + V + V_gV_r + UgU_r

g r + v_gv_r + u_gu_r + 1¾² + u_g ²

YbYr + v_bv_r + u_bu_r y_by_g + v_bv_g + u_bu_g

extended to the 2x2 blocks as:

(RGB⁴ x RGB⁴) -→ M

with the block diagram matrix

Q ^3X3 ^3X3 ^3X3

3X3 Q ^3X3 ^3X3

3X3 ^3X3 Q ^3X3

3X3 ^3X3 ^3X3 Q

where M is the matrix defined above.

That dot product defines what we will call the YUV norm:

RGB -→ M

or RGB₄ i→ R

It should be further noted that the example non-limiting color conversion process described above does not operate on vector spaces, but rather on (bounded) subregions of vector spaces. In particular, for example, even though YUV420 (as a vector space) is a sub vector space of RGB4 (as a vector space), YUV420 (as a bounded block color space) is not entirely contained in RGB4 (as a bounded block color space). We therefore recognize that orthogonal projection is not always the optimal way to perform the color conversion processes.

However, when it is desirable to rapidly convert a large number of images (e.g., in the context of digital video), computation efficiency becomes a concern. A quasi affine transform (QAT) for RGB→ YUV color space conversion may use matrix M in the form:

where k

The term "quasi affine" is used instead of affine because after the transform matrix is applied, the output color vector is clamped, using the saturate (sat) operator, to a certain range (usually [0..1.0] or [0..255]) so as to avoid overflows in processors used to perform the computation.

Similarly, the inverse YUV RGB color space conversion may be in the form

where sat is the operator that clamps the color vector components to the output range. Examples of these affine transforms are found for example in:

ITU-R Recommendation BT.601

ITU-R Recommendation BT.709

CIE 1931.

Compressed images are decompressed or decoded before use. Let's recall what the standard YUV₄₂o→RGB⁴ decode process is. Since there is no downsampling involved on the decompression/decoding side, and therefore no information loss, it is quite straightforward:

The conventional conversion (both compression and decompression) described gives the optimal results with respect to the YUV norm so long as the conversion of a vector in RGB⁴ does not give a value outside of the [0..1] range of YUV420. One aspect of the example illustrative non-limiting technology herein aims at improving those cases— which generally occur in strongly saturated and contrasted 2 x 2 blocks.

Finding the absolute optimal solution when we consider the saturation is the following non-linear constrained optimization problem: given a 2 x 2 RGB block rgb4 e RGB4, find yuv₄2o eYUV₄2o such that the following is minimal:

It is nonlinear because of the saturation operator, and it is constrained since YUV420 is not the 6-dimensional vector space, but its [0..1]⁶ cube (see discussion of Figure 3D-E below). Solving this full-fledged problem is more complex than constrained quadratic programming, and is orders of magnitude more expensive than the standard conversion. Using current technology, it may be not desirable to try to implement such a more comprehensive solution on limited-resource computing devices such as are often found in consumer electronic devices.

Example Non-Limiting Practical Solutions

The example non-limiting technology herein uses a less complex, more practical method that aims at achieving most of the gain of the full method, but having a cost closer to the standard method.

In one example implementation, we use an optimized method for converting RGB to Y values that is slightly more expensive than conventional techniques but which provides improved performance in high-contrast situations. Our exemplary illustrative non-limiting example implementation takes advantage of the fact that while generally speaking in a downsampled situation, target YUV color space has less resolution than the original RGB color space representing the original image, the YUV color space is capable of representing certain colors that are not representable in the original RGB color space. Meanwhile, conventional video decoding hardware and software will project those certain colors back into the RGB color space. We take advantage of this by choosing a conversion from RGB to YUV that in some cases produces YUV coordinate values that lie outside of the RGB color space but which, when projected back into the RGB color space as part of decoding, will produce RGB values that are close to the original image RGB values than can be achieved by conventional transforms.

In more detail, one example non-limiting implementation assumes that we have a known decode process called D, and we want to find the encode process E+ that improves upon E, the standard encode process, such that, for each input pixel block x, |D(E+(x))-x| is minimal (ideally) or at least smaller than |D(E(x))-x|. (That is, without math symbols, find the pixel block encode process that yields, when chained with the standard decode process, a result as close as possible to the pixel block we are trying to encode).

It would be possible to use constrained piecewise quadratic programming to solve this. This approach would be the canonical and tractable and would provide the best possible results for any quadratic distance function over color blocks. However, for many applications (e.g., especially consumer electronics) it a full such solution may be overly expensive in terms of processing time and resources. For example, such color conversion is often performed on a stream of images, and may need to be performed in real time or close to real time by relatively lower power embedded processors (e.g., such as is found in many consumer electronic devices, smart phones, etc.). Accordingly, in many non-limiting implementations, it may be desirable to trade off such a "best possible" solution for one that is more practically efficient.

One non-limiting example implementation provides an individual fast Y optimization method that yields the best possible results if we only allow ourselves to have the Y's modified compared to the standard method. It is safe, because it always yields results at least as good as the standard process.

Further Example Improvement Optimizing UV

The UV can also be optimized. If we choose to optimize the UV as compared to the standard method, we can subsequently use the aforementioned Y optimization to further improve the results. This in turns provides two example non-limiting possible venues:

First, one non-limiting approach optimizes the UV based on a heuristic. One proposed non-limiting heuristic is based on the observation that saturated pixels are able to recover quality by optimizing their Ys, whereas non saturated pixels cannot. Therefore, it is sensible to shift the common UV towards the unsaturated pixels, since the saturated pixels can cover their losses by changing their Ys. Of course, in such example non-limiting implementations, the Y range is bounded (so it cannot compensate too large distortions), and single dimensional (so it cannot fix both U and V at the same time), so it is to mildly shift the UV towards the unsaturated pixel components instead of trying to shoot straight for the optimum point. Indeed, this non- limiting example heuristic relies on the fact that the saturated pixel set remains constant when we move the UV value around. This is only true in a small neighborhood around the initial (standard converted) UV, and that's why it's better to underestimate the UV shift, so as to not introduce undesirable errors.

It is also possible to attack the UV optimization with a full-fledged optimization that assumes the use of a subsequent Y optimization process. This problem is also a continuous piecewise quadratic programming problem, but it is of lower dimensionality than the full optimization process, because we are exploiting the fact that the Ys are independent, and so can be optimized after the UVs have been optimized.

Additional Non-limiting features and advantages include:

measurably better quality than the standard conversion results equal or nearly equal to the standard conversion when the latter performs well

computation cost that is less than a few times the standard conversion's computation cost so as to allow its implementation in similar hardware

reaches a quality level close to that of the full brute force optimization applies at least to all quasi affine colorspace conversions with chroma downsampling.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

These and other features and advantages will be better and more completely understood by referring to the following detailed description of example non-limiting embodiments and/or implementations in conjunction with the drawings, of which:

Figure 1 shows an example conversion from RGB to YUV color space;

Figure 2 shows example conventional sRGB to YUV420 downsampling;

Figure 3 shows an example non-limiting computer processing architecture for compressing a color image for transmission and/or storage;

Figures 3A-3C show example non-limiting simplified color conversions for purposes of illustration;

Figures 3D-3E are example non-limiting illustrations of conversion of color blocks between an RGB color space and a YUV color space;

Figure 4 shows an example more detailed processing flowchart for compressing an RGB image; and

Figure 5 shows an example non-limiting computer processing architecture for decompressing and using a color image.

DETAILED DESCRIPTION Example Overall Block Diagram

Figure 3 shows an example non-limiting computer processing architecture for encoding a color image 10 into a file 50 for storage in non-transitory storage 30 and/or into a stream for transmission over a network 40 or other transmission path. Color image 10 can be derived from any source such as a graphics processor, a camera, or any other source capable of providing a color image or sequence of color images. In one embodiment, a processor 20 executes program code stored in non-transitory storage 30 to compress the color image while optimizing RGB to YUV conversion. The processor may comprise a state machine, a gate array, digital logic, a digital signal processor, a microcomputer, or any other desired implementation.

Figure 4 shows one non-limiting example sequence of separate stages used to compress and encode color image data. Note that some of these steps are aggregated in real implementations, but are separated here so as to give a better view of the logical blocks.

In this non-limiting example, an RGB color image 10 (ex. JPEG) is first converted to a YUV444 color image.

The YUV420 image is then compressed by a sequence of additional steps (block DCT or "Discrete Cosine Transform", quantization, zigzag RLE or "Run Length Encoding", and entropy coding) to provide a compressed color image signal for storage and/or transmission. The example non-limiting technology herein provides improved conversion techniques between 10 and 200, 300.

At the other end of the storage or transmission channel, the processing may be reversed (see Figure 5) to recover the original RGB color image 10 or to convert the decompressed image into some other format for display or other usage. Such decoding can be performed by a processor comprising a state machine, a gate array, digital logic, a digital signal processor , a microcomputer, or any other desired imp lementation.

In one example non-limiting implementation, the decoding side can use conventional hardware and software such as found in many existing prior art devices to decompress and convert the signal from YUV420 to RGB for display or other use. Thus, the example non-limiting technology herein produces a compressed output that can be decompressed and otherwise processed by existing prior art devices without any changes whatsoever in the decompression/decoding device.

When such prior art devices are presented with a compressed format produced by the compressor/encoder of Figure 3, they are caused to perform a different decoding than they would perform if presented with a prior art conventional compressed format - namely they will perform a YUV to RGB block conversion that converts from values in the YUV color space that has no corresponding coordinates in the RGB space, and they will, by the conventional YUV to RGB conversion process, convert such YUV values into RGB values that more closely approximate the original image file RGB color values before initial conversion on the encoding side than can be obtained by conventional technology. This is thus a new process not previously performed and yet is backward compatible with and in fact takes advantage of existing prior art hardware and software decoders and decompressers.

Conventional Wisdom

Thus, while some prior art block color conversion schemes convert RGB values into YUV values that cannot be represented in the original RGB color space, at least some conventional wisdom might lead one skilled in the art to minimize the error between the initial RGB values and the converted YUV values. In contrast, in example non-limiting implementations, the RGB to YUV block conversion is structured so intentionally produce YUV values that are non-representable in the original RGB color space but which will provide reduced color representation errors when block reconverted back from YUV into target RGB color space using conventional block conversion technology.

Example Non-Limiting Simplified Illustrative Example

Figures 3A-3C show a simplified (2D), but more easily represented version/example non-limiting implementation. In the particular non-limiting case illustrated here, the goal is to convert a 2x1 pixel block from RGB to YUV. In this simplified example for purposes of illustration, each pixel has only 2 components (R and G) instead of the typical three components (R,G,B) but in a preferred implementation all three components would be converted and the conversion would be from 3D RGB color space as opposed to 2D RG color space. Note that even in the 2D simplified example, the output of the conversion is a 3-component value

In Figure 3 A, original pixels A and B are illustrated on the RG gamut boundary, but anywhere close would work too. In this particular illustrative example, "-" (Figure 3B) denotes the standard (conventional) conversion and "+" (Figure 3C) denotes the enhanced conversion provided by example non-limiting technology herein. In the diagram, x means that x is saturated.

We can see from Fig. 3A, 3B and 3C that ¾⁺ Y V/_B yields pixels {A'⁺B'⁺} that are a better match for {AB} than {A'^~B'^~} (the conventional approach) would be. In particular, in Fig. 3B one can see that with ^l'⁼ [¾¾_B] and B'⁼ [Y_BV_AB], B' and A' are the destination pixels decoded by the standard decode process and are therefore forced back into the standard RG range. In contrast, in Fig. 3C, Y_B is the optimized Y_B . We can see that in this case, the Y optimization is able to produce a B' that exactly matches B.

The discussion of Figures 3A-C have been in connection with A and B representing single pixel values for purposes of simplification, but since the example illustrative non-limiting technology herein operates based on blocks, in Figures 3D (cut version) & 3E (3D version), points P0 .. . PN represent color blocks (e.g., 2x2 pixels).

Since this simplified problem shown in Figures 3D-3E is 4D->3D instead of say 12D→6D (in the case of RGB₄→YUV₄2o), it does not capture all the issues that the full problem has but is nevertheless illustrative. Thus, the cube (projected as a square) should not be thought as the RGB cube (RGB₄), b u t a s t h e R1G1B1R2G2B2R3G3B3R4G4B4 hyper (duper) cube, and the 2D plane should really be thought of as the YUV₄₂₀ 6-dimensional space. The dotted line represents the overlap between the spaces (note that in actuality, YUV₄₂₀ doesn't extend this far out of the RGB₄ hypercube but is exaggerated this way for purposes of illustration).

In Figures 3D-3E, P0 represents the case where the conventional process already provides optimal results. PI shows a conventional block color conversion where the conventional approach yields less than optimal results, and P2 represents an example non-limiting block color conversion the technology herein provides. As one can see from Figure 3D, the conventional block color conversion transform projects the RGB value into the YUV block color space in a very different way than the transform provided by the example non-limiting optimized implementations described herein.

In more detail, Figure 3D uses notation that matches Figs. A, B and C. The RGB4 gamut of Fig. 3D looks skewed because the YUV norm has been chosen so that color block distances are proportional to the real distances (on paper), and orthogonal projections using that norm are indeed right angles. In Fig. 3D, P0 shows the case where the standard method is indeed optimal; PI shows the case where the optimization can provide optimal results ; and P2 shows the case where the optimization improves upon the standard method, but is limited in effect by the fact that the YUV420 boundary is reached, (the ideal optimization is shown next to it). Note that in this Fig. 3D section drawing, the sat arrows do not necessarily lie within the 3D drawing plane. The dotted lines do however, which is a reason for choosing the YUV norm for this illustrative drawing.

More Detailed Illustrative Example

The following avoids simplification and provides a more rigorous mathematical description of an example non- limiting implementation.

Let's suppose that we have a source 2 x 2 RGB block rgb₄ G RGB₄, that converts to yuv₄₂₀ = [y₀ y- y₂ y- u v]^T eYUV₄₂₀ using the standard process described above. The initial back and forth conversion between YUV and RGB would be:

rgb = sat(rgb₄)

As mentioned above, if rgb₄ = rgb₄ then yuv₄₂o is the optimal solution for Example Non-Limiting Y optimization

When yuv₄₂o is not the optimal solution, we can find new optimal values for the

Y's, that we will call [y₀' y y₂' y' ]^T'. Since changing any value yi only modifies the matching output RGB value rgbj, we can optimize these values separately. Let's consider the pixek in the block.

We are going to find the scalar value 5_y . that minimizes the error e_t■

rgbi'' iSyi) = rgb[ + S_y.e_y

)

Note that any norm can be chosen here, but to remain consistent with the standard solution, it is best in a non- limiting example to choose the YUV norm. Let's define the notsati operator:

S_y^ notsat^_y.) ^^{1 1}^ ^^ ^^' ]

V^{l Vl J} UO otherwise J _fe={1..₃

Now assuming that we are in a range [yi_im.i_n,

where we know that the components of rgb" are saturated and are constant

(3K_S \ V5_y. G [y_iimin - y_t , y _max - y^ notsat^ _y^) = K_s)

we can transform e_t into a second degree polynomial in 5_y.. Note that this constantness of nonsat, happens in a neighborhood of 0. e_y[ = notsati(0)e_y (componentwise multiplication)

rgb"(5_y.) = sat(rgb[ + 5_y.e_y)

= rgb + Sy.e^

^Εί (A*) = ¥ i + ¾¾ί - ^r9^hi 'yuv

|2

= ||5_y.e^ + (¾-r^_M^

= <Δ_έ ,Δ_ί> + 25_y.(e_y^_l,A_i) + <¾<e^,e^>

ei(6_y )'s quadratic part is positive so there is only one extremum, a minimum, that is the root of 6; (6_y. ) 's derivative:

a_ei(6_yi)

0

Yi

=> 26_y. <e_y>1 , Δ;) + 2<e_y>1 , Δ;) = 0

<= Oy. =— , assuming e_y is not 0

¹ \^ey,i '^ey,i'

Now, provided that y_i + 5_y.e[y_min,y_max holds, we have found a minimum value for δ_ν ..

Example Non-Limiting UV Optimization

An optional and more complex step is choosing the optimal UV value of the yuv 420 block. This step should happen before the Y optimization step, since it assumes that the Y optimization step will be performed afterward. It yields value as the standard transform provided that there is not saturation. Otherwise, its aim is to provide better results. It works by assuming that on a given block, the previous Y optimization method will be applied, and optimizes the block's UV taking this into account. Let (u,v) be the variables we solve for.

Pi = t>i + ae^

e.(a) = IIPi(a)llyui;

= ||M-_Pi(a)||

= <t>i ,bi) + 2a( e_y^,bi> + a²{e^,e^)

Let's first find the a_{min i} that minimizes the Y for a given block.

9δ_ί(α)

0 =— = 2a( e_yi ,e_yi ) + 2< e_yi ,bi>

σα <e_yilJbi> <e_yil,bi>

<= ct

y.i -y.H

τηίη,ί)

{Έ _γ , ue_u + ve_v - A)

ue_u + ve_v

-y,i

<e_y>1 , e_u) + v{e_yi , e_v) - <e_y>1 , Δ;)

ue_u + ve_v - -= e i

-y.H yuv

(assuming ||e_yi|| = 1)

ue_u + ve_v - Δ; - (u<e^ , e_u) + v(e^ , e_v) - (e^ , Δ;))

yuv

's compute the distance over all the pixels in the block.

(e_u (*-y,i J *-u)*-y,i ' Pi (^min)

, Δ;)) e^~

+v(l - <e^ , e_v}²)

Finding fw,v such that both— ^- and — are zero is equivalent to solving the system:

0 = H^■ x—

where

>]

Note that this provides the optimal UV for a YUV420 block only if the subsequently optimized Y values do not fall outside of their allowed ranges, and if the RGB saturations remain constant between the original conversion, and the optimized one (we use the approximation that the active constraint set will remain the same in some neighborhood of the initial value). If additional RGB saturations appear with the optimized value, this solution will not be optimal, but will still be better than the one that optimizes only the Ys. If RGB constraints disappear at the optimized value, then the solution will not be optimal, but in benchmarks, it still remains generally better than the raw Y optimized one. Finally, if the Ys do saturate, then the UV optimized solution is generally not good at all. In this case, we can compute, along the [original UV, optimized UV] 2D segment, at which point the first Y saturates, and keep that point (thus forcing us to stay within the constant active constraint region).

Example Non-Limiting Heuristic UV Optimization

The UV optimization described in the previous section is not very fast (especially because the Y saturation handling cannot be omitted). So here is a "tame" heuristic way of finding a better UV for the block. The rationale behind this choice is that since some RGB components get saturated, going further in the saturation direction does not make it worse for them, but it goes toward giving more weight to the values that are not saturated, thus increasing the accuracy of the conversion.

rgbo' ut.i = ΪΊΪΚ - rgbi'

uv_out = (M₄ - rgb₄) \_uv

better approximation : yuv'₂₀ = yuv₄₂₀ + k^■ [u_out , v_out , 0]^T

In non saturated cases, rgb_o'_{ut i} = 0, so that the standard solution is already optimal. Since this is a heuristic, the constant k is calibrated in the example non- limiting implementation. A value of 2 almost systematically improves the quality of a block.

Example Non-Limiting Y Range Extension

A remarkable fact about the non- limiting example Y optimization method described above is that, if the output Y is allowed to extend out of the [0..1] range, it can do perfect encoding of pure R (resp. G, B) against a dark background or Cyan (resp. Magenta, Yellow) against a bright background. This happens often with colored font rendering, and is a pathological case for the standard conversion process. There are at least two ways to take advantage of an extended Y range:

A compatible way: The fact that the 'studio swing' does not really exploit the full [0..255] range, but a limited range (typically [16..240]), the disclosed non- limiting conversion process could output Y values under 16 or above 240. A standard decoder will clamp and push back all Y values below 16 up to 16, so it won't benefit from the full advantage of optimized Y values, but, on the other hand, they won't suffer from it. A modified decoder that just takes the value as-is will benefit from the example out of 'studio swing' out of range optimized Y values.

A custom way: if the encode/decode is fully under control of the implementer, the Y range can be chosen so as to improve the tradeoff between Y optimization quality and global Y quantization quality.

The Example Non-Limiting Case When The Decoding is Bilinear

The methods we have described consider that the YUV₄₂o→RGB⁴ conversion uses the R₄ matrix (i.e., the UV values are broadcasted identically to the four RGB values), but in some implementations, the uv upscale is done bilinearly. The Y optimization we suggested works perfectly fine, provided that the yuv₄₂o→ Tgb[ uses a bilinear upscale too. On the other hand, the UV optimization becomes a global problem not just limited to single 2 x 2 blocks. Notations used above:

scalars have the standard typeface, (eg : x)

colors or vectors are boldface, (eg : v)

matrices are capital boldface, (eg : M)

R stands for red.

G stands for green.

B stands for blue.

Y stands for luma.

U stands for chroma dimension 1.

V stands for chroma dimension 2.

RGB represents the. [0.. 1]_r [0.. 1]_g ^χ [0.. 1]& standard RGB cube.

YUV represents the [0.. 1]j ^χ [0.. 1]_M ^χ [0.. 1]_v YUV cube.

RGB⁴ represents the [0.. 1]^{3 '4} ≡ RGBQ ^χ RGB\ ^χ RGB2 x RGB3 12-dimension hypercube of all possible 2 2 RGB pixel blocks.

YUV₄₂₀ represents the output [0.. 1]6 6-dimension hypercube [O.- lL^ x [0.- 1]^ ^x 10.. 1 1,, x | 0.. 1 ] , _s x 0.. 1]_a x [0.. 1]_v.

u-v = u^Tv is used to show a canonical component dot product, and <u, v> = u ( v) = U^JMY is used to show the YUV space dot product, when u and v are expressed in RGB space, where M is the conversion matrix from RGB to YUV as vector spaces.

The example non-limiting technology herein thus provides a process for improving the RGB->YUV BLOCK conversion, for some YUV like colorspace, with some form of downsampling, by exploiting the fact that the decoder will saturate back the values of the decoded rgb values into their allowed ranges. Separately improving the Y's and the UV's is an efficient but non-limiting approach to that problem. If a process is to use both Y and UV optimization, optimizing the UV first is often the right way to go. The example non-limiting technology herein can also perform YUV444→YUV420 conversions (among others) e.g.,: the source color space may well be the same as the destination color space, the destination only being downsampled with respect to the source. The decode process could then revert to some other color space, RGB as one non-limiting example. While the example non- limiting technology can be used to improve image luma for high contrast image blocks, the non-limiting disclosed technology also provides a method for optimizing the chroma. The combined luma+chroma optimization process non-limiting example disclosed here is only one optimizing method exploiting the fact that decompressors saturate the output RGB values back into their allowed ranges.

While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not to be limited to the disclosed embodiment. For example, the given description covers the RGB to YUV₄₂₀ downsampling, but the technique applies just as easily to different UV downscale ratios or to other representations. Similarly, while the detailed description above uses particular downsampling modes as examples, the technology herein is not limited to such particular downsampling modes but applies to any downsampling. Thus, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the claims.

Claims

1. A compressor comprising:

a downsampler that maps color blocks defined in a first block color space into a second block color space different from the first color space,

the mapping comprising a processor using a heuristic and a constrained piecewise quadratic calculation to map the color blocks into points in the second block color space that lay outside of the first color space.

2. For use with a decode process D, an improved encode process E+ that encodes input pixel blocks such that, for each input pixel block x encoded, |D(E+(x))- x| is smaller than |D(E(x))-x| and the pixel block encode process yields, when chained with the decode process, a result close to the pixel block to be encoded.

3. The process of claim 2 including using a processor to perform a constrained piecewise quadratic programming to optimize luma values Y alone, and an estimation of constraints at the standard conversion point.

4. An implementation of claim 2 comprising a full constrained quadratic optimization that finds the best E+(x).

5. The process of claim 2 wherein the result comprises Y and the process further comprises also optimizing UV.

6. The process of claim 2 where the result comprises YUV and only the

Y's are optimized using a constrained quadratic optimization where the constraints can be approximated.

7. The process of claim 2 comprising

first optimizing UV values, and

subsequently applying Y optimization where only the Y's are optimized using a constrained quadratic optimization where the constraints can be approximated to further improve the results.

8. The process of claim 2 wherein optimizing UV comprises optimizing based on a heuristic.

9. The process of claim 8 wherein optimizing UV based on a heuristic comprises enabling saturated pixels to recover quality by optimizing their Ys, whereas non saturated pixels cannot recover quality by shifting common UV towards unsaturated pixels.

10. The process of claim 9 wherein the shifting comprises mildly shifting the UV towards the unsaturated pixel components instead of trying to shoot straight for the optimum point.

11. The process of claim 2 wherein optimizing UV comprises performing a continuous piecewise quadratic programming problem of lower dimensionality and optimizing the independent Y's after the UV.

12. A process for block-converting a color block for upsampling by an upsampling decoder that will saturate decoded values into an allowable range, comprising:

with a processor, downsampling the color block to provide lower resolution color values; and

with the processor, exploiting the behavior of the decoder to saturate decoded values into an allowable range to thereby optimize the lower resolution color values for subsequent decoding by the upsampling decoder.

13. The process of claim 12 where the lower resolution color values comprise

luma and color difference signals, and

the exploiting comprises separately improving luma and/or the color difference values.

14. The process of claim 13 wherein the exploiting comprises optimizing both luma and color difference values.

15. The process of claim 14 wherein the exploiting comprises

optimizing the color difference values first and then

optimizing the luma value.

16. The process of claim 12 wherein the color block comprises an RGB representation.

17. The process of claim 12 wherein the upsampling decoder decodes to provide an RGB representation.

18. A method of compressing a digital image using a processor to automatically block convert image blocks from a first color space to a second color space, including:

using the processor to automatically compress the block converted image blocks to provide a compressed image format by exploiting the behavior of a decoder to saturate decoded values into an allowable range to convert at least some of said image blocks to coordinates in the second color space that are undefined in the first color space in order to optimize luma for high contrast image blocks.

19. The method of claim 18 wherein the first color space comprises RGB and the second color space comprises YUV.

20. The method of claim 18 wherein the conversion separately optimizes the values [y₀' y y₂' y₃' Y' .

21. The method of claim 18 wherein the converting optimizes generation of a luma value based on transforming e_t into a second degree polynomial in 5_y. and then finds a minimizer for that polynomial.

22. The method of claim 18 wherein for each luma value yi, the processor minimizes the error.

23. The method of claim 18 wherein the processor computes based on a minimum of the root of (5_y .) 's derivative: e_{y [} = notsati(0)e_y (componentwise multiplication)

rgb" (5_y.) = sat(rgb[ + 5_y.e_y)

let A_f = (rgbl - rgbt)

= (Ai , Ai) + I _y.ie^ . Ai) + 5_y£<e^ , e^>

24. The method of claim 18 further including the processor optimizing UV values based on a subsequent Y optimization.

25. The method of claim 24 wherein the UV value optimization is based on heuristics.

26. The method of claim 24 wherein the UV value optimization is based on calculating so that minimal values will be provided for ^3δ™^ιη·¹ and ^3δ™^ιη·¹

^{& 1} du dv ^■

27. The method of claim 18 further including converting, with a decoder, the compressed format from yuv₄2o→ Tgb[ using a bilinear upscale and measures and optimizes the error assuming that the decoder uses a bilinear or any other upscale filter to reconstruct the full resolution U and V.

28. A method of decompressing an image comprising:

using a processor to automatically recover compressed block converted image blocks; and

using the processor to automatically block convert recovered image blocks from a second color space to a first color space, including converting at least some of said recovered image blocks from second color space coordinates that do not exist in the first color space while exploiting the behavior of block convert to optimally saturate decoded values into an allowable range.

29. The method of claim 28 wherein the first color space comprises RGB and the second color space comprises YUV.

30. A method of compressing an image while exploiting the behavior of a decode process to saturate decoded values into an allowable range comprising:

using a processor to automatically block convert image blocks from a first color space to a second color space, including converting at least some of said image blocks to coordinates in the second color space that do not exist in the first color space in order to optimize luma values for high contrast image blocks; and

using the processor to automatically compress the block converted image blocks.

31. Apparatus for compressing an image comprising:

a processor being configured to automatically block converts image blocks from a first color space to a second color space, including converting at least some of said image blocks to coordinates in the second color space that do not exist in the first color space in order to optimize luma values, the processor being further configured to automatically compress the block converted image blocks to provide a compressed image in a color space + block space.

32. Apparatus for decompressing an image comprising:

a processor connected to receive a compressed image format,

the processor being configured to automatically decompress to recover color- converted image blocks,

the processor being further configured to automatically block convert image blocks from YUV420 color space to RGB color space, including converting at least some of said image blocks to coordinates from the YUV color space + block space that do not exist in the RGB color space, to thereby improve image luma for high contrast image blocks.

33. Non-transitory storage medium storing information comprising instructions that when executed by the processor cause the processor to:

automatically block convert image blocks from a first color space to a second color space, including converting at least some of said image blocks to coordinates in the second color space that do not exist in the first color space in order to optimize luma values, and

automatically compress the block converted image blocks to provide a compressed image.

34. Non-transitory storage medium storing information comprising instructions that when executed by the processor cause the processor to:

receive a compressed image format,

automatically decompress the compressed image format to recover color- converted image blocks, and

automatically block-convert image blocks from YUV420 color space to RGB color space, including converting at least some of said image blocks to coordinates from the YUV color space that do not exist in the RGB color space, to thereby improve image luma for high contrast image blocks.

35. A decode process D for use with an encode process E+ that encodes input pixel blocks such that, for each input pixel block x encoded, |D(E+(x))-x| is smaller than |D(E(x))-x|, the decode process yielding a result close to the pixel block to be encoded.

36. A block color converter for converting intermediate color pixel blocks developed by an encode process E+ that encodes the color pixel blocks into intermediate YUV color values, the block color converter comprising:

a data director that directs intermediate YUV color values having the characteristic that, for each color pixel block x encoded, |D(E+(x))-x| is smaller than |D(E(x))-x|, where D is performed by the below-referenced upsampling decoder;

the upsampling decoder saturating the directed intermediate YUV color values into an allowable RGB range to thereby provide pixel block color conversion.