WO2015144563A1

WO2015144563A1 - Image processing system and method

Info

Publication number: WO2015144563A1
Application number: PCT/EP2015/055813
Authority: WO
Inventors: Robert Godlieb; Jasper Zuidervaart
Original assignee: Koninklijke Philips N.V.
Priority date: 2014-03-28
Filing date: 2015-03-19
Publication date: 2015-10-01

Abstract

An image processing system (100) is provided, which is arranged to produce an image of a human with an overlay. The system comprises a user image analyzing unit (130) comprising a noise obtainment unit (132) arranged to obtain a noise level, the noise level representing noise in a user image. The user image represents a human, and in an embodiment a human face. The system comprises an overlay transformation unit (140) comprising a noise application unit (142) arranged to apply noise to the overlay image according to the noise level. In an embodiment the overlay image represents facial hair. The system comprises an overlay unit (150) arranged to overlay a transformed overlay image produced by the overlay transformation unit on top of the user image.

Description

Image processing system and method

FIELD OF THE INVENTION

The invention relates to an image processing system comprising an overlay unit arranged to overlay an overlay image on top of a user image. The invention further relates to an image processing method, computer program, and a computer readable medium.

BACKGROUND

When considering a new facial hair style, e.g. a beard and/or moustache-style, there is a desire to get an impression how this would look. In the art computer programs are known that take an image the face and overlay a beard-style over this picture.

The beard overlay is sometime a ready-made facial hair style; sometimes the computer program enables the user to draw a beard over the picture.

Although these known approaches allow a user to get some impression how he would look with a given facial hair style, in the known methods the produced images of a user with an overlaid facial hair style look artificial. This distracts from the aim of the preview method. Instead of getting a realistic impression of how a given facial hair style would really look, the pictures are not taken seriously.

An imaging system described in WO 98/21695, with title "Imaging system for simulating hair styles", included herein by reference, addresses the fake appearance that is obtained when overlaying a hair style on an image of a human face to a certain extent. The system aims to simulate achievable hair replacement surgery results.

The known imaging system is arranged to superimpose, i.e., overlay, a hair style on an image of a user. The hair style is selected from an index of hair styles and displayed to a system operator on the client's image. The displayed hair style is

semitransparent so that the system operator may manually position the hair style at a desired location on the client's image. The imaging system further allows the operator to select a color matching region in the image of the user. The color matching region is used to copy the color of the client's existing hair to the overlaid hair style.

The article "Multi-scale Image Harmonization" by K. Sunkavalli et al, ACM Transactions on Graphics, Vol. 29, No. 4, Article 125 (2010), addresses the problem that applying traditional image compositing techniques, such as alpha matting and gradient domain compositing, to images taken from different sources or shot under different conditions can produce unrealistic results. A framework is presented that explicitly matches the Visual appearance of images before blending them. At the heart of this framework is a multi-scale technique that allows to transfer the appearance of one image to another. By carefully manipulating the scales of a pyramid decomposition of an image, contrast, texture, noise, and blur are matched, while avoiding image artifacts. The output composite can then be reconstructed from the modified pyramid coefficients while enforcing both alpha-based and seamless boundary constraints. The proposed framework can be used to produce realistic composites with minimal user interaction in a number of different scenarios.

US7227976B1 discloses a system and method for detecting facial features of humans in a continuous video and superimposing Virtual objects onto the features automatically and dynamically in real-time. The suggested system is named Facial

Enhancement Technology (FET). The FET system consists of three major modules, initialization module, facial feature detection module, and superimposition module. Each module requires demanding processing time and resources by nature, but the FET system integrates these modules in such a way that real time processing is possible. The users can interact with the system and select the objects on the screen. The superimposed image moves along with the user's random motion dynamically. The FET system enables the user to experience something that was not possible before by augmenting the person's facial images. The hardware of the FET system comprises the continuous image-capturing device, image processing and Controlling system, and output display system.

The article "Noise Brush: Interactive High Quality Image-Noise Separation" by J. Chen et al, ACM Transactions on Graphics, Vol. 28, No. 5, Article 146 (2009), proposes an interactive approach using joint image noise filtering for achieving high quality image-noise separation. The core of the system is a joint image-noise filter which operates in both image and noise domain, and can effectively separate noise from both high and low frequency image structures. A user interface is introduced, which allows the user to interact with both the image and the noise layer, and apply the filter adaptively and locally to achieve optimal results. The interactive system improves the initial image-noise separation results and can also be deployed in various noise-consistent image editing tasks, where preserving the noise characteristics inherent in the input image is a desired feature. SUMMARY OF THE INVENTION

It would be advantageous to have an improved image processing system that is arranged to produce an image of a human with an overlay. The invention is defined by the independent claims; the dependent claims define advantageous embodiments.

An image processing system is provided, arranged to produce an image of a human with an overlay, the system comprising an image input arranged to receive a user image, the user image representing a human, a storage arranged to store an overlay image, a user image analyzing unit comprising a noise obtainment unit arranged to obtain a noise level, the noise level representing noise in the user image, an overlay transformation unit comprising a noise application unit arranged to apply noise to the overlay image according to the noise level, and an overlay unit arranged to overlay a transformed overlay image produced by the overlay transformation unit on top of the user image. The user image analyzing unit comprises a lighting obtainment unit arranged to determine lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, and the overlay transformation unit comprises a lighting application unit arranged to apply said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions.

Users are sensitive to visual changes, like differing noise level in an image. An image in which noise levels suddenly change has a fake appearance. As a result such an image is less suitable for decision making since focus is detracted from the image by the noise shift. The combined image obtained by overlaying the transformed overlay image on top of the user image suffers less from this defect. Since noise levels are determined in the user image and then matched in the overlay image the resulting image has a more realistic appearance.

In an embodiment, the user image represents a human face and the overlay image represents hair, e.g., a man's facial hair like beard and/or moustache. The system is particularly suitable for overlaying facial hair on a human, since noise levels may be well established on the relatively smooth region of human cheeks without facial hair. The invention may alternatively be used for previewing hair styles (e.g. short, curly or straight) or eyebrow styles.

Next to noise levels, humans are also sensitive to changing lighting conditions. By determining and matching light levels the combined image becomes also more realistic. In both cases, there is a mismatch in fidelity between the picture taken of the person's face and the overlay, say beard, which is drawn onto it.

The system obtains a user acceptable preview of a facial hair style based on a picture of a face, overcoming the perception of reality problem for interpreting the preview image. The fidelity and visual style appearance of the beard-overlay is matched with the fidelity and visual style appearance of the user image.

An aspect of the invention concerns an image processing method producing an image of a human with an overlay.

It is noted that known system cited above only uses the technique of alpha- blending to improve coherence. Alpha-blending cannot achieve coherence between a user image and an overlay with respect to noise or lighting. Moreover, alpha-blending is not suitable for facial hair since beard hairs do not fade.

The image processing system is an electronic device. The system may be implemented on a mobile electronic device, say a mobile phone, tablet, and the like. For example, the system may be embodied as a so-called downloadable 'app' . The app may use a camera of the mobile phone to capture the user image. The app may have one or more overlay images stored with the app. The system may also comprise a drawing unit arranged to enable a user to draw an overlay image.

The image processing system may be embodied in a computer, in particular a server. For example, the system may receive the user image over a computer network, say the internet.

The image processing system and method described herein may be applied in a wide range of practical applications. Such practical applications include: internet shopping comprising previewing clothing overlays on user images, razor devices with corresponding computer application for previewing a beard style.

A method according to the invention may be implemented on a computer as a computer implemented method, or in dedicated hardware, or in a combination of both.

Executable code for a method according to the invention may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product comprises non-transitory program code means stored on a computer readable medium for performing a method according to the invention when said program product is executed on a computer. In a preferred embodiment, the computer program comprises computer program code to perform all the steps of a method according to the invention when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium. Herein, a computer is any device capable of processing images, e.g. a smartphone, a tablet, a laptop or a desktop computer, and a computer program is any software that runs on a device capable of processing images. The invention can be used with any kind of operating system.

Another aspect of the invention provides a method of making the computer program available for downloading. This aspect is e.g. used when the computer program is uploaded into e.g. Apple's App Store, Google's Play Store, or Microsoft's Windows Store, and when the computer program is available for downloading from such a store.

An image processing system is provided, which is arranged to produce an image of a human with an overlay. The system comprises a user image analyzing unit comprising a noise obtainment unit arranged to obtain a noise level, the noise level representing noise in a user image. The user image represents a human, and in an

embodiment a human face. The system comprises an overlay transformation unit comprising a noise application unit arranged to apply noise to an overlay image according to the noise level. In an embodiment the overlay image represents facial hair. The system comprises an overlay unit arranged to overlay a transformed overlay image produced by the overlay transformation unit on top of the user image.

Another aspect of the invention provides an image processing method producing an image of a human with an overlay, the method comprising: receiving a user image, the user image representing a human, storing an overlay image, determining lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, including determining brightness levels for respectively left and right cheeks of a face represented in the user image, applying said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions to obtain a transformed overlay image, including determining a multiplier for a brightness of pixels in the overlay image based on a weighted average between the determined brightness levels for respectively the left and right cheeks, and overlaying the transformed overlay image on top of the user image. BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,

Fig. 1 is schematic block diagram of an image processing system,

Fig. 2 is a user image showing facial landmarks,

Fig. 3 shows a user image, overlay images, and images of a human with an overlay,

Fig. 4a shows a region of a user image,

Fig. 4b shows a region of a user image on which a smoothing filter has been applied,

Fig. 4c illustrates a noise map,

Fig. 4d illustrates a thresholded noise map,

Fig. 5a illustrates a user image with left and right region,

Fig. 5b illustrates a weighted average between determined brightness levels, Fig. 6 shows a schematic flowchart of an image processing method,

Fig. 7a shows a computer readable medium having a writable part comprising a computer program according to an embodiment,

Fig. 7b shows a schematic representation of a processor system according to an embodiment.

Items which have the same reference numbers in different figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.

List of Reference Numerals:

100 image processing system

1 10 image input

120 storage

130 user image analyzing unit

132 noise obtainment unit

134 facial landmark unit

136 lighting obtainment unit

140 overlay transformation unit

142 noise application unit 144 geometric transformation unit

146 lighting application unit

150 overlay unit

160 first artistic filter

162 second artistic filter

200 user image including facial landmarks

210 facial landmark

310 user image

320 overlay image

321 redacted overlay image 320

322 redacted overlay image 320

324 overlay image

330 image of a human with an overlay

340 user image with artistic filter

350 image of a human with an overlay

360 image 350 with an artistic filter

430 region of a user image

432 region 430 with smoothing filter

434 noise map

436 noise map with thresholding

520 user image

522, 524 left and right region

600 image processing method

610 initialization phase

620 analyzing phase

622 noise level analyzing step

624 facial landmarks analyzing step

626 lighting conditions analyzing step

630 application phase

632 noise applying step

634 geometric transformation applying step

636 lighting conditions applying step

640 overlay phase

1000 computer readable medium 1010 writable part

1020 computer program

1 100 processor system

1 1 10 integrated circuits

1 120 processing unit

1 122 memory

1 124 dedicated integrated circuit

1 126 communication element

1 130 interconnect

DETAILED DESCRIPTION OF EMBODIMENTS

While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail one or more specific embodiments, with the understanding that the present disclosure is to be considered as exemplary of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described.

Fig. 1 is schematic block diagram of an image processing system 100. Image processing unit 100 is arranged to produce an image of a human with an overlay.

Below a number of detailed embodiments are described of an image processing system, wherein the user image represents a human face and the overlay image represents hair, e.g. facial hair like beard and/or moustache. A user image showing a human face may be referred to as a face image for short.

There are other application areas for which it is useful to produce images of humans with an overlay, e.g. a method or a system for displaying clothing. In such a system the overlay image represents an article of clothing. The image of the article of clothing, say a sweater is overlaid on an image of a user. Landmark points may be identified in the user image, e.g., shoulder points, hip points, and the like. The overlay image of clothing is geometrically transformed to match the user image. The overlay image is adjusted to match a noise or light level of the user image.

The embodiments below all assume an overlay representing facial hair is superimposed on a user image representing, i.e., showing, a human face; it has to be kept in mind that other applications are possible.

Image processing unit 100 comprises an image input 1 10 and storage 120. Image input 1 10 is arranged to receive a user image. For example, image input 1 10 may receive the user image over a computer network, say the internet. Image input 1 10 may receive the user image from an image capturing device, such as a camera, e.g. a web-cam. The camera may be comprised in system 100. In this embodiment the user image represents a human face. Fig. 3 shows a user image 310. User image 310 has been processed to remove the background; this is not necessary.

Storage 120 is arranged to store an overlay image. For example storage 120 may be an electronic memory. Storage 120 may also be a file server, a hard disk, and the like. In this embodiment, the overlay image represents facial hair. Storage 120 may be cloud storage, i.e., comprising an interface to retrieve information over a computer network, say the internet. System 100 may be geographically distributed, however, all of system 100 including the information stored in storage 120 may alternatively be local. Fig. 3 also shows an overlay 320.

User and overlay image may be represented in any image format. In an embodiment, the user image and/or overlay image may be represented in a raster format, e.g., jpg, gif, tiff, and the like. The user and/or overlay image may be represented in a vector format, either 2D or 3D, and may be transformed (by unit 144) as vector graphics. In an embodiment, noise and light transformations happen on a raster image format.

Image processing unit 100 comprises a user image analyzing unit 130, an overlay transformation unit 140, and an overlay unit 150.

Analyzing unit 130 is arranged to analyze the user image. There are a number of properties of the user image that may be analyzed. Interestingly, the architecture that is described below is modular. Analyzing unit 130 may comprise one or multiple analysis units arranged to analyze the user image for one or more properties. Transformation unit 140 may then comprise one or multiple corresponding transformation units to transform the overlay image. Overlay unit 150 is arranged to overlay the transformed overlay image produced by the overlay transformation unit on top of the user image. The output of overlay unit 150 may be displayed to a user, e.g., on an electronic display. Overlaying may be done by

superimposing. The overlay may comprise an alpha channel. The alpha channel indicates where the overlay is to be overlaid, and where it is not. The alpha channel may be

geometrically transformed together with the image information in the overlay. However, noise or lighting transformation will typically not affect the alpha channel.

Overlay unit 150 may be arranged to blur the edge of the transformed overlay and/or apply progressive transparency towards the edge of the transformed overlay image before merging the transformed overlay and the user image. This further improves the realism of the combined image.

In a simple embodiment, the alpha channel contains only values of either 1 or 0, causing either display of the overlay or not. A more advanced embodiment uses

progressive transparency towards the edges of the overlay image. For example, in an embodiment the alpha channel of the overlay image contains alpha-values, e.g., between 1 and 0, wherein at least a center part of the overlay image has alpha values indicating no transparency, e.g. alpha-values of 0, and at least part of a border region of the overlay image has alpha value indicating at least partial transparency, e.g. alpha-values less than 1. Herein an alpha- value of 1 indicates that a corresponding part of the overlay image will be fully superimposed on the user image, i.e., no transparency, an alpha-value of 0 indicates that the overlay that a corresponding part of the overlay image will not be displayed, and alpha- value in between 0 and 1 indicate partial transparency; e.g. 0.5 would half-draw, 0.1 would draw only 10 percent weight of the overlay onto the background image.

It is not necessary for image processing unit 100 to have a modular architecture. In an embodiment, the one or more units of analyzing unit 130 and

transformation unit 140 may be distributed around image processing system 100 if so desired.

In an embodiment analyzing unit 130 comprises a facial landmark unit 134 to determine facial landmarks in the user image. Facial landmarks may include points like eye corners, mouth corners, and nose tip. Fig. 2 shows a user image 200 in which detected facial landmarks are indicated with a white dot. For example, facial landmark 210 indicates a boundary point of the face. In the art, a facial landmark unit is also referred to as a facial landmark detection engine.

In addition to facial landmarks the facial landmark unit 134 may also determine bounds of the face in addition to the location in the image of relevant features such as e.g. the eyes and the mouth. Facial landmark engines are available on systems like iOS, Android, in the OpenCV library, and the like. Using a Facial Landmark Detection (FLD) engine, the location of the face in the picture that was taken is known.

Transformation unit 140 comprises a geometric transformation unit 144 arranged to apply a geometric transformation to the overlay image to match the determined facial landmarks. The geometric transformation may be an affine transformation. For example, based on the bounds as detected in the user image and the bounds of the pre-loaded beard-style overlay, determine the affine transformation may be determined so that the overlay to match the user image. In an embodiment, multiple points in the overlay image correspond to multiple facial landmarks. For example, points in the overlay image may be a tagged with

corresponding facial landmark of the multiple facial landmarks. Facial landmark unit 134 is configured to detect at least this set of multiple facial landmarks. Geometric transformation unit 144 is configured to determine a transformation so that the tagged points lie close to the corresponding facial landmark detected in the user image.

If the number of points in the overlay image is 3 then an affine transformation exists so that the transformed points exactly match the 3 facial landmarks. It may be found by solving corresponding linear equations. If the number of points in the overlay image is more than 3 an exact match may not exist, in this case a transformation is determined that minimizes the difference between the transformed points of the overlay and the facial landmarks; this closest may be determined according to a closeness measure, e.g., according to least summed square errors.

Facial landmark unit 134 and geometric transformation unit 144 are not strictly necessary. For example, if the user image and overlay image both adhere to precise standards regarding the position of facial features.

In an embodiment image processing unit 100 comprises an optional first artistic filter 160 and/or an optional second artistic filter 162. Overlay unit 150 is arranged to apply artistic filter 160 to the user image before overlaying the transformed overlay image. Fig. 3 also shows a user image 340 on which first artistic filter 160 has been applied.

Optionally, after the overlay has been superimposed on the user image the second artistic filter 162 may be applied to the combined image. Fig. 3 shown in image 330 the result of overlaying an overlay image without using artistic filters. In image 350 the result of overlaying the overlay image on image 340 is shown. First artistic filter 160 or equivalent has previously been applied to the overlay image, say in storage, so that the artistic filtering matching in overlay and filtered user image. Image 360 is obtained by applying second artistic filter 162 to image 350. Artistic filtering further reduces the artificial impression because they mask the different origins of the two images.

A look is created that is visually coherent with the beard renderings for the human observer. Although applying an artistic filter reduces the fidelity of the photographic image it improves the match with the impression of the rendering. Artistic filters, as they are known in the art, degrade the photographic fidelity of an image to match an artistic style. Artistic filters are known per se in image processing. Examples of artistic filters include cubism, oil painting, canvas, brush stroke, pointillism, and the like. One may also apply the process disclosed in US 8,437,514 to generate a 'cartoon-like' appearance from a photo or video-frame.

The photographic user image is processed to achieve an appearance that is consistent with the realism and style of a rendered overlay, say a facial hair overlay such as a beard overlay.

Second artistic filter 162 further enhances the consistency, by performing another filtering step on the combined preview to meld the elements in a single preview image that is consistent and a credible preview of the facial style. Applying a process to the combined image creates a more coherent simulated facial hair style image where the distinction between actual captured user image and rendered beard is less clearly and disturbingly visible.

Interestingly, although one can clearly see that images 350 or 360 do not have a high photographic fidelity, they credibly reduce the impression that the overlay is fake.

Generally speaking, the image analyzing unit 130 is arranged to analyze a property of the user image, the overlay transformation unit 140 is arranged to align a property of the overlay image with the analyzed property of the user image to produce a transformed overlay image. Overlay unit 150 is arranged to overlay the transformed overlay image produced by the overlay transformation unit on top of the user image. Because the property of the overlay image matches the property of the user image, the mismatch is reduced, at least in so far as the property is concerned. Thus the artificial impression that the image with overlay would give without transforming the overlay is reduced.

In an embodiment, analyzing unit 130 comprises a noise obtainment unit 132 arranged to obtain a noise level. The noise level represents that amount of noise in the user image. Transformation unit 140 comprises a corresponding noise application unit 142 arranged to apply noise to the overlay image according to the noise level.

Noise obtainment unit 132 may be arranged to determine the noise level from properties of the camera that was used to take the use image. For example, facial landmark unit 134 may be configured to obtain the camera type from a tag of the user image, say a manufacture and/or model field of a tag system in the user image, say, in the Exchangeable Image File Format (EXIF). For example, noise obtainment unit 132 may use the camera type to look up noise level in a database that associates camera type to noise level. Optionally, noise obtainment unit 132 also takes lighting conditions into account, e.g., the database may link camera type and lighting conditions to noise level. For example, a tag associated and/or embedded in the user image may indicate light conditions. For example, aperture and speed in the EXIF data may be used to estimate the light conditions when the user image was taken. For example, the noise obtainment unit may be arranged to represent the noise level as one or more noise parameters of a noise generation algorithm.

Alternatively, the noise levels may be directly obtained from the user image. For example, noise obtainment unit 132 may be arranged to apply a noise level estimation algorithm to the user image.

The analysis may be applied to the whole of the user image, but may also be applied to a region in the user image. For example, noise level may be estimated from the region in the user image on which the overlay will be overlaid. The noise level may be obtained from a region within the user image.

In an embodiment, analyzing unit 130 determines a region within a cheek of the face in the user image. Analyzing unit 130 may determine the regions from facial landmarks obtained by facial landmark unit 134. For example, a center of the region may be obtained by adding a cheek vector to a facial landmark, say a mouth-corner. Fig. 5a shows two regions: a left region 522 and a right region 524 determined from facial landmarks.

Various noise estimation algorithms are known in the art to estimate noise from an image. Below one noise estimation algorithms is disclosed. Interestingly, the noise estimation algorithm is applied to a region.

The noise level may be determined by subtracting a smoothed image from the original image. The smoothing may be done by applying a smoothing filter to the user image; for example using a filter with a limited spatial effect. An example of a smoothing filter is box-blur smoothing. The range of the smoothing may be taken, as e.g. 4 pixels.

Fig. 4a shows a detail of a user image, in this case a region of the cheek. On this region a smoothing filter is applied, the result of which is shown in Fig. 4b.

The inventors had the insight that the cheeks of humans without a beard have a reasonably smooth structure. In that case, the difference set, e.g., the original un-smoothed user image minus the smoothed image is indicative of the noise levels. For example, the noise obtainment unit may comprise a smoothing unit arranged to obtain a smoothed user image by applying a smoothing filter to the user image, obtaining the noise map comprises subtracting the smoothed user image from the user image.

Fig. 4c shows the result of subtracting a smoothed user image from an original user image: noise map 434. The user image of Fig. 2 was used to prepare Figs. 4c and 4d. A noise map may have positive and negative values. To produce image 434 the absolute values in the noise map have been used. In an embodiment, a noise map may use similar data structures as an image, say a two-dimensional array. The noise map may be maintained as the noise pattern for the particular image, say as delta values in brightness-deviation for pixels of an area.

A noise map may be thresholded. For example, all values in the noise map that have an absolute value less than a noise threshold are set to zero. Thresholding captures a practical noise map that may be applied to an overlay to achieve a coherent realistic combined image. Fig. 4d shows thresholded noise map 436 obtained from noise map 434. When creating a noise map by taking the difference between an original and a smoothed image there is a risk that some remnants of the original image may be visible in the noise map. This effect is significantly reduced whilst retaining the improved blending in by thresholding the noise map. As an example threshold one may remove the smallest 10% of the noise values. Larger or smaller threshold values are possible. Threshold values may be expressed relative to the dynamic range of the threshold or as absolute values, etc..

The noise map may be used in at least two different ways, to obtain noise parameters from or to directly apply to the overlay image. For example, the noise obtainment unit may be arranged to represent the noise level as one or more noise parameters of a noise generation algorithm, the noise parameters are obtained from the noise map. For example, the noise level may be represented by magnitude of noise and the frequency of noise.

Representing noise level as parameters has the additional advantage that no remnants of the user image are left in the noise applied to the overlay.

Thus analyzing unit 130 determines the noise level of the user image. The noise level may be represented in different ways, e.g., as one or more parameters or as a noise map.

Transformation unit 140 comprises noise application unit 142 which is arranged to apply noise to the overlay image according to the noise level. The noise in a rendered or professionally captured overlay, say a facial hair overlay is typically very low, whereas the noise in a user image may be quite high, e.g. due to non-professional capturing equipment or low-light conditions. Noise application unit 142 is arranged to apply the noise generation algorithm with the noise parameters the overlay image. Methods of adding noise patterns are known per se in image processing.

For example, if the noise level is expressed as one or more parameters, the noise application unit may be arranged to apply the noise generation algorithm with the noise parameters to the overlay image. The noise application unit may directly apply a noise generation algorithm to the overlay image, thus avoiding an intermediate noise image. The noise application unit may also generate a noise map or noise image and apply that to the overlay image.

The noise application unit may also be arranged to apply noise according to the noise map to the overlay image, e.g., by adding the noise map the overlay image.

In an embodiment, geometric transformation unit 144 applies a geometric transformation to the overlay image to map it to the face in the user image, before noise application unit 142 applies the noise to the overlay image according to the noise level. In this situation noise obtainment unit 132 may be arranged to determine the part of the user image over which the overlay image will be superimposed. Noise application unit 142 can then apply the noise level to the overlay image only for this part of the user image.

In an embodiment, the noise obtainment unit may be arranged to obtain a noise map indicating location of noise in the user image to represent the noise level, and the noise application unit is arranged to apply noise according to the noise map to the overlay image. For example, one may obtain a noise map for the whole image, e.g., by obtaining a difference between the original user image and a smoothed user image. The resulting noise map may be thresholded. The geometric transformation may be applied to the overlay image, next the noise map may be restricted to the parts in the user image where the overlay will be applied. For example, one may obtain a restricted noise map by performing an 'and' operation between the geometrically transformed alpha-channel of the overlay and the noise map. Next the restricted noise map may be applied to the overlay, e.g., by adding the noise map to the overlay image.

Accordingly, the noise level is applied to the overlay image, e.g., by generating random noise onto the image from computed parameters representing noise, say density and magnitude, or the rendered beard overlay may be changed by pixel-mapping of the deviations in the determined noise map, etc. Matching noise levels ensures a matching fidelity in appearance of the shown simulated face style.

In an embodiment, analyzing unit 130 comprises a camera property unit (not separately shown) to determine the image resolution and/or dynamic range of the user image. For example, these may be determined from the image data itself or from information attached to the user image. Transformation unit 140 may comprise a property application unit (not shown) to apply the determined image resolution and/or dynamic range to the overlay image. For example, resolution may be matched by a scaling operation. For example, dynamic range may be matched by mapping the dynamic range in the overlay image to the larger or smaller ranger obtained from the user image. In particular, the dynamic range in the overlay may be compressed to match a smaller dynamic range in the user image.

In an embodiment, analyzing unit 130 comprises a lighting obtainment unit 136 arranged to determine lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, and transformation unit 140 comprises a lighting application unit 146 arranged to apply said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions. Lighting differences are readily apparent in an image. For example, light may predominantly come from the left, the right, front, or back. If the lighting conditions on the overlay image are different, say a diffused even illumination, it will be readily apparent in the combined image that the overlay is fake.

For example, the lighting obtainment unit 136 may be arranged to determine the brightness level for respectively the left and right cheek of the face represented in the user image. In an embodiment, analyzing unit 130 determines a left and right region within the left and right cheek of the face in the user image. Analyzing unit 130 may determine the regions from facial landmarks obtained by facial landmark unit 134. For example, a center of the region may be obtained by adding a vector the left and right mouth-corner respectively. Fig. 5a shows a left region 522 and a right region 524 determined from facial landmarks.

To determine the lighting distribution, the illumination of the relevant areas on the face is measured (and the beard overlay is adjusted correspondingly). In a simple, fast method the average color value of a set of pixels is measured on the left cheek and on the right cheek. Using the average of pixels instead of individual pixels avoids noise of small scale skin artifacts.

From the average color value a brightness value is obtained. For example, the color may be expressed in, e.g., the HSV color model. The V values are then used to modify the rendered beard overlay image to match the V parameter for the sides of the face.

In an embodiment, the lighting application unit may be arranged to determine a multiplier for the brightness of pixels in the overlay image as a weighted average between the determined brightness level for respectively the left and right cheek. For example, acceptable results are obtained by defining a multiplier for V by the ratio of the left and right average V to the overall average V of both areas. The beard overlay is then adapted by multiplying the V of the pixels by those ratio values, using linear interpolation. This means the overall lighting distribution is adapted to match that on the picture of the face (e.g. the horizontal light/dark gradient).

Applying a gradient-varying multiplier to a pixel may be done by modifying the RGB values of the pixels of the image or more conveniently in HSV space.

A higher quality may be obtained by determining a V ratio for every point of the face where a beard rendering will be applied and adapting the beard overlay with those values. An alternative to adapting the light levels across the rendered beard overlay is to do a brightness correction on the original picture as taken. This however leads to a clear reduction in fidelity and credibility of the overall combined picture.

Fig. 5b illustrates a brightness gradient. In this image V_L denotes the average brightness in a left region of the user image, V_R denotes the average brightness in a right region of the user image, and V_AVG denotes the overall average brightness within both the left and right image. Brightness values (e.g., V values) of pixels on the left side of the overlay image are multiplied by (V_L/V_AVG) and brightness values of pixels on the right side of the overlay image are multiplied by (V_R/V_AVG). Pixels in between the left and right side may multiplied by a multiplier that is linearly interpolated between (V_L/V_AVG) and (V_R/V_AVG). That is, the multiplier is a weighted average of (V_L/V_AVG) and (V_R/V_AVG).

The lighting analysis and application has merit independent from the noise units. For example, the following image processing system is considered particularly advantageous even without a noise obtainment or application unit: An image processing system arranged to produce an image of a human with an overlay, the system comprising an image input arranged to receive a user image, the user image representing a human, a storage arranged to store an overlay image, a user image analyzing unit comprising a lighting obtainment unit arranged to determine lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, wherein the lighting obtainment unit is arranged to determine the brightness level for respectively the left and right cheek of the face represented in the user image, an overlay transformation unit comprising a lighting application unit arranged to apply said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions, and an overlay unit arranged to overlay a transformed overlay image produced by the overlay transformation unit on top of the user image, wherein the lighting application unit is arranged to determine a multiplier for the brightness of pixels in the overlay image based on the weighted average between the determined brightness level for respectively the left and right cheek. Lighting application unit 146 may be arranged to apply the following lighting application method:

Parameters RatioLeft and RatioRight are assumed to have been determined, e.g., by lighting obtainment unit 136. RatioLeft and RatioRight are indicative of the relative brightness of the left and right side respectively of the user image. For example, RatioLeft and RatioRight may be determined be selecting a left and right region in the cheek of the user as indicated above. Example values are RatioLeft = 0.6 and RatioRight = 1. 4. Lower values correspond to a darker region and higher value to a brighter region.

Then for each pixel (x,y) in the overlay image a multiplier is computed indicating how the brightness of said pixel should be adjusted. For efficiency, one may check if pixel (x,y) is fully transparent, and if so, the lighting adjustment for this pixel may be skipped.

For example, the multiplier (RatioX) may be computed as follows: RatioX = RatioLeft + (x/Overlay Width) * (RatioRight- RatioLeft). In this example, x is the horizontal coordinate, increasing from left to right, from the direction of the left region to the right region. OverlayWidth is the width of the overlay in pixels, i.e., in the x-direction. This formula computes the multiplier RatioX as the weighted average between RatioLeft and RatioRight.

The multiplier may be used directly in an HSV representation. Other representations are equally possible, but require accordingly adjusted formulas. For example, one may set PixelColor as the HSV representation of the pixel that is currently adjusted. If HSV is used, then PixelColor is a triple: (PixelColor.Hue, PixelColor. Saturation,

PixelColor. Value). The adjusted color for the pixel is now given by the HSV triple:

(PixelColor.Hue, PixelColor. Saturation, PixelColor.Value * RatioX).

In an embodiment, light adjustment is done before noise adjustment. In that embodiment noise is applied to an overlay that has been adjusted for lighting. In an embodiment, and overlay is first geometrically transformed, then adjusted for lighting, then adjusted for noise. This avoids applying light adjustment on the noise, which is unrealistic.

Applying lighting to an overlay based on a user image may also be implemented as a method of making available for downloading. For example: A method of making available for downloading image processing software arranged to produce an image of a human with an overlay, the image processing software being arranged to: receive a user image, the user image representing a human, store an overlay image, determine lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, including determining brightness levels for respectively left and right cheeks of a face represented in the user image, apply said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions to obtain a transformed overlay image, including determining a multiplier for a brightness of pixels in the overlay image based on a weighted average between the determined brightness levels for respectively the left and right cheeks, and overlay the transformed overlay image on top of the user image.

Also the artistic filter may be applied independent of noise or lighting. An example embodiment is the following: An image processing system arranged to produce an image of a human with an overlay, the system comprising a storage arranged to store an overlay image, and an overlay unit arranged to overlay an overlay image on top of the user image, wherein the overlay unit is arranged to apply a first artistic filter to the user image before overlaying the transformed overlay image, and optionally is arranged to apply a second artistic filter to the result of said overlaying.

In an embodiment, the user image is received in a raster format and the overlay image is stored in a vector format. The geometric transformation is applied to the overlay image in vector format to match the human in the user image, in particular a face in the user image. After the geometric transformation the overlay image is converted to a raster format. The lighting and/or noise transformation are then applied on said geometrically transformed overlay image in raster format.

Typically, the device 100 comprises a microprocessor (not shown) which executes appropriate software stored at device 100 and; for example, that software may have been downloaded and/or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as Flash (not shown). The devices 130 and 140 may also be equipped with microprocessors and memories (not shown). Alternatively, the devices 100, 130 and 140 may, in whole or in part, be implemented in programmable logic, e.g., as field-programmable gate array (FPGA); or may be implemented, in whole or in part, as a so- called application-specific integrated circuit (ASIC), i.e. an integrated circuit (IC) customized for their particular use.

In an embodiment, the image processing system comprises an image input circuit arranged to receive a user image, the user image representing a human, a storage circuit arranged to store an overlay image, a user image analyzing circuit comprising a noise obtainment circuit arranged to obtain a noise level, the noise level representing noise in the user image, an overlay transformation circuit comprising a noise application circuit arranged to apply noise to the overlay image according to the noise level, and an overlay circuit arranged to overlay a transformed overlay image produced by the overlay transformation circuit on top of the user image. The circuits may be a processor circuit and storage circuit, the processor circuit executing instructions represented electronically in the storage circuits. The circuits may also be, FPGA, ASIC or the like. The system may comprise additional circuits, e.g., a facial landmark circuit, a geometric transformation circuit, a lighting obtainment circuit, a lighting application circuit, etc.; the circuits implementing the corresponding units described herein. A facial landmark circuit may also be referred to as a facial landmark detection circuit.

Fig. 3 shows a user image, overlay images, and images of a human with an overlay. Image 310 is an example of a user image. In this case the background has been removed for clarity. Although removing the background is possible, this need not be done in embodiments. Image 310 shows a human face. Image 320 is an example of an overlay image. Overlay image 320 shows facial hair. Image 322 shows a redacted overlay image 320. A 'redacted' overlay image is an image in which a property has been matched with a corresponding property of the user image. An example of a property is noise, lighting, dynamic range, resolution, and the like. Note that the difference between images 320 and 322 is subtle though clearly visible when viewed side by side. Redacted overlay image 322 blends in better with user image 310. Image 330 shows the result of overlaying image 322 in user image 310.

Image 322 has been adjusted for noise and lighting in user image 310. Image 321 has only been adjusted for lighting. Image 321 shows adjustment for a different lighting than image 322. Note that images 322 and 321 differ in the apparent lighting.

Image 340 shows user image 310 after processing with a first artistic filter. Artistic filters reduce photographic fidelity of an image while remaining recognizable. Image 324 show overlay image 320 which has been processed with the same artistic filter used for user image 340. To improve efficiency, overlay images may be stored in filtered form, i.e., after being processed with the first artistic filter. Image 350 shows the result of overlaying image 324 on image 340. Note that as a result of a reduction in photographic fidelity, boundaries between user image and overlay are less visible; yet the image remains recognizable. Image 360 shows the result of applying a second artistic filter. The second artistic filter further improves blending between user image and overlay. The second artistic filter may or may not be the same as the first artistic filter. Fig. 6 shows a schematic flowchart of an image processing method 600. The method comprises an initialization phase 610, an analyzing phase 620, an application phase 630, and an overlay phase 640. Note, that these phases need not be discrete a certain phases may start before others have finished. Especially phases 620 and 630 are distinguished mainly for expository reasons, as these phases may be integrated to a large extent.

In the initialization phase 610 a user image is received and an overlay image is obtained. The user image represents a human. The overlay image may be present in the form of a storage.

The analyzing phase 620 may comprise one or more of the following group of analyses:

analyzing 622 the user image, the analyzing comprising obtaining a noise level, the noise level representing noise in the user image,

analyzing 624 the user image unit comprises determining facial landmarks in the user image, and

- analyzing 626 the user image comprises determining lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image.

The transformation phase 630 may comprise one or more corresponding transformations:

- transforming 632 the overlay image, the transforming comprising applying noise to the overlay image according to the noise level, and

transforming 634 the overlay image comprises applying a geometric transformation to the overlay image to match the determined facial landmarks.

transforming 636 the overlay image comprises applying said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions.

In the overlaying phase 640 the transformed overlay image is overlaid on top of the user image.

Many different ways of executing the method are possible, as will be apparent to a person skilled in the art. For example, the order of the steps can be varied or some steps may be executed in parallel. Moreover, in between steps other method steps may be inserted. The inserted steps may represent refinements of the method such as described herein, or may be unrelated to the method. For example, analysis 620 and transformation 630 and may be executed, at least partially, in parallel. Moreover, a given step may not have finished completely before a next step is started.

A method according to the invention may be executed using software, which comprises instructions for causing a processor system to perform image processing method 600. Software may only include those steps taken by a particular sub-entity of the system. The software may be stored in a suitable storage medium, such as a hard disk, a floppy, a memory etc. The software may be sent as a signal along a wire, or wireless, or using a data network, e.g., the Internet. The software may be made available for download and/or for remote usage on a server. A method according to the invention may be executed using a bitstream arranged to configure programmable logic, e.g., a field-programmable gate array (FPGA), to perform the method.

Fig. 7a shows a computer readable medium 1000 having a writable part 1010 comprising a computer program 1020, the computer program 1020 comprising instructions for causing a processor system to perform an image processing method, according to an embodiment. The computer program 1020 may be embodied on the computer readable medium 1000 as physical marks or by means of magnetization of the computer readable medium 1000. However, any other suitable embodiment is conceivable as well. Furthermore, it will be appreciated that, although the computer readable medium 1000 is shown in Fig. 7a as an optical disc, the computer readable medium 1000 may be any suitable computer readable medium, such as a hard disk, solid state memory, flash memory, etc., and may be non-recordable or recordable. The computer program 1020 comprises instructions for causing a processor system to perform said image processing method.

Fig. 7b shows in a schematic representation of a processor system 1 100 according to an embodiment. The processor system comprises one or more integrated circuits 1 1 10. The architecture of the one or more integrated circuits 1 1 10 is schematically shown in Fig. 7b. Circuit 1 1 10 comprises a processing unit 1 120, e.g. a CPU, for running computer program components to execute a method according to the invention and/or implement its modules or units. Circuit 1 1 10 comprises a memory 1 122 for storing programming code, data, etc. Part of memory 1 122 may be read-only. Circuit 1 1 10 may comprise a

communication element 1 126, e.g., an antenna, connectors or both, and the like. Circuit 1 1 10 may comprise a dedicated integrated circuit 1 124 for performing part or all of the processing defined in the method. Processor 1120, memory 1 122, dedicated IC 1 124 and communication element 1 126 may be connected to each other via an interconnect 1 130, say a bus. The processor system 1 100 may be arranged for contact and/or contact-less communication, using an antenna and/or connectors, respectively.

It will be appreciated that the invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice. The program may be in the form of source code, object code, a code intermediate source and object code such as partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention. An embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the processing steps of at least one of the methods set forth. These instructions may be subdivided into subroutines and/or be stored in one or more files that may be linked statically or dynamically. Another embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the means of at least one of the systems and/or products set forth.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. An image processing system (100) arranged to produce an image (330) of a human with an overlay, the system comprising:

an image input (1 10) arranged to receive a user image (210), the user image representing a human,

a storage (120) arranged to store an overlay image (320),

a user image analyzing unit (130) comprising

a noise obtainment unit (132) arranged to obtain a noise level, the noise level representing noise in the user image,

an overlay transformation unit (140) comprising

a noise application unit (142) arranged to apply noise to the overlay image according to the noise level, and

an overlay unit (150) arranged to overlay a transformed overlay image produced by the overlay transformation unit on top of the user image; wherein

the user image analyzing unit (130) comprises

a lighting obtainment unit (136) arranged to determine lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, and

the overlay transformation unit comprises

a lighting application unit (146) arranged to apply said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions.

An image processing system as in Claim 1 , wherein the user image represents human face, and the overlay image represents hair.

An image processing system as in Claim 2, wherein

the user image analyzing unit comprises

a facial landmark unit (134) to determine facial landmarks in the user image, and the overlay transformation unit comprises

a geometric transformation unit (144) arranged to apply a geometric transformation to the overlay image to match the determined facial landmarks.

4. An image processing system as in Claim 1, wherein

the noise obtainment unit is arranged to represent the noise level as one or more noise parameters of a noise generation algorithm, and

the noise application unit is arranged to apply the noise generation algorithm with the noise parameters to the overlay image.

5. An image processing system as in Claim 1, wherein

the noise obtainment unit is arranged to obtain a noise map indicating location of noise in the user image to represent the noise level, and

the noise application unit is arranged to apply noise according to the noise map to the overlay image.

6. An image processing system as in Claim 5, wherein

the noise obtainment unit comprises

a smoothing unit arranged to obtain a smoothed user image by applying a smoothing filter to the user image, obtaining the noise map as a difference between the smoothed user image from the user image.

7. An image processing system as in Claim 2, wherein

the lighting obtainment unit is arranged to determine the brightness level for respectively the left and right cheek of the face represented in the user image, and

the lighting application unit is arranged to determine a multiplier for the brightness of pixels in the overlay image based on the weighted average between the determined brightness level for respectively the left and right cheek.

8. An image processing system as in Claim 1, wherein

the overlay unit is arranged to apply an artistic filter to the user image before overlaying the transformed overlay image.

9. An image processing method producing an image of a human with an overlay, the method comprising:

receiving a user image, the user image representing a human,

storing an overlay image,

- analyzing the user image, the analyzing comprising obtaining a noise level, the noise level representing noise in the user image,

transforming the overlay image, the transforming comprising applying noise to the overlay image according to the noise level, and

overlaying said transformed overlay image on top of the user image; wherein - analyzing the user image comprises determining lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, and

transforming the overlay image comprises applying said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions.

10. An image processing method as in Claim 9, wherein

analyzing the user image unit comprises determining facial landmarks in the user image, and

- transforming the overlay image comprises applying a geometric

transformation to the overlay image to match the determined facial landmarks.

1 1. An image processing method as in Claim 9, the method further comprising:

determining lighting conditions in the user image by measuring a brightness level of at least two different regions of the human represented in the user image, including determining brightness levels for respectively left and right cheeks of a face represented in the user image,

applying said determined lighting conditions to the overlay image by changing brightness of pixels in the overlay image according to the determined lighting conditions, including determining a multiplier for a brightness of pixels in the overlay image based on a weighted average between the determined brightness levels for respectively the left and right cheeks.

12. A computer program comprising computer program code to perform all the steps of any one of claims 9-1 1 when the computer program is run on a computer.

13. A method for making available for downloading the computer program as claimed in claim 12.