WO2001077976A2

WO2001077976A2 - Image segmenting to enable electronic shopping for wearable goods and cosmetic services

Info

Publication number: WO2001077976A2
Application number: PCT/US2001/009729
Authority: WO
Inventors: Pierre N. Fay; Joshua Flachsbart; David Franklin
Original assignee: Eyeweb, Inc.
Priority date: 2000-03-28
Filing date: 2001-03-27
Publication date: 2001-10-18
Also published as: WO2001077976A3; AU2001249481A1

Abstract

A system and corresponding method for segmenting into regions a digital image derived from a photograph of part of a body, such as a photograph of the face of a person. The segmented regions can then be used as the basis for showing for example the face of the person wearing different cosmetic materials, since particular kinds of cosmetic materials are applied to particular regions of the face. The image segmenting is performed using a segment locator, responsive to a digital image of some part of a body and also to generic locating guidance. The segment locator provides segment location information that is approximate, and is used by a segment delineator to provide a precisely segmented image, in the form of at least one binary mask identifying a particular region along with the original image.

Description

IMAGE SEGMENTING TO ENABLE ELECTRONIC SHOPPING FOR WEARABLE GOODS AND COSMETIC SERVICES

FIELD OF THE INVENTION

The present invention pertains to the field of shopping for goods that are worn by a customer (including for example clothes, jewelry and cosmetics) or cosmetic services (including for example hair styles and hair treatments) , using a display device to allow a shopper to electronically (virtually) try on the goods or (virtually) try out the cosmetic services. More specifically, the present invention is concerned with image segmenting technology used to identify segments of an image of a customer where goods would be worn or applied (in the case of cosmetics) , or where cosmetic services would be directed, thus making it possible for a computer to use the segmented image to show the customer with different goods worn or applied or to show the customer as the customer would appear as a result of cosmetic services .

BACKGROUND OF THE INVENTION

In purchasing goods to be worn or applied or in purchasing cosmetic services, a customer would like to see how the customer would look to others when wearing the goods, or for example after a makeover (including the use of cosmetic materials, hair treatments and hair styling, and jewelry) . For example, in purchasing a cosmetic material, it is very often useful to try on the cosmetic material, such as lipstick, eyeshadow, or blush, before purchasing it. In trying on first one cosmetic material and then another, it is necessary for the customer to remove the first cosmetic material. In addition, to actually try on a cosmetic material, a customer must visit a store selling the cosmetic material .

Today, because of the increased computing capabilities of readily available computers, including so-called personal computers, it is reasonable to attempt to have a computer provide a customer with an image of the customer' s own face or an image of more of the customer, modified to illustrate how the customer would appear wearing a cosmetic material or other wearable goods, including even a hairstyle. To do so, however, requires a precise identification in the image of those regions of the customer's face or other parts of the customer where the goods would be worn (or, in the case of cosmetics, where the cosmetic materials would be applied) .

If this could be done, then a customer could visit a retail store and use a computer at the retail store to do a virtual "trying on" of the wearable goods, or a virtual "trying out" of cosmetic services, and could even do the virtual "trying on" or "trying out" from home, or from some other, convenient location.

SUMMARY OF THE INVENTION

Accordingly, the present invention provides an image segmenter and a corresponding method for segmenting an image, for segmenting into predetermined regions a digital image derived from a photograph of a person or animal, including : a segment locator, responsive to the digital image and to generic locating guidance, for providing segment location information; and a segment delineator, responsive to the digital image and to the segment location information, for providing a segmented image including a binary mask.

In a further aspect of the invention, the segment locator is a template deforming module, and the segment locator is a steerable filter module. In such an embodiment, the generic locating guidance is a deformable template (a kind of generic representation of the part of the body whose image is being segmented) and associated constraints (indicating how the part of the body whose image is being segmented may change from individual to individual) , and the segment locating information is a deformed template.

In other further aspects of the invention, either the segment locator or the segment delineator is based on a neural network, or either the segment locator or the segment delineator is based on principal component analysis, or either the segment locator or the segment delineator is based on wavelet graph analysis.

In yet other further aspects of the invention, the predetermined regions include a lipstick region, an eyeshadow region, and a blush region. The pre-determined regions can in general be any regions of a body, including regions where eyeglasses are worn, or where a shirt or blouse is worn, or where pants are worn, or the individual's hair (for helping show a hair style or hair treatment) .

In yet another aspect of the invention, the image segmenter is part of a shopping system allowing a customer to electronically try on wearable goods (including cosmetic materials) or electronically try out cosmetic services. The shopping system according to the invention includes: a means for providing the digital image derived from a photograph of a person; and a remote electronic store, responsive to the digital image, the remote electronic store including the segmenting system, the remote electronic store for providing the customer via a display device with madeup images showing how the customer would appear wearing the wearable goods or as a result of the cosmetic services.

In another aspect of the invention, a method is provided for enabling a customer to electronically try on wearable goods (including cosmetic materials) or electronically try out cosmetic services, including the steps of: taking a photograph of the customer and providing a digital encoding of the photograph; and performing a two-part process to precisely identify pre-determined regions where the wearable goods would be worn or applied or where the cosmetic services would be directed, a first part for approximately locating each of the regions, and a second part to delineate the approximately located regions .

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the invention will become apparent from a consideration of the subsequent detailed description presented in connection with accompanying drawings, in which:

Fig. 1 is a flowchart/ block diagram showing the elements of a system for electronically trying on cosmetics, as an example of wearable goods, allowing for the purchase of the cosmetics tried on, according to the present invention;

Fig. 2 is an illustration of a human face, indicating regions identified by the present invention where different types of cosmetics (eyeshadow, blush, and lipstick) would be applied;

Fig. 3 is a flow chart showing the steps according to the present invention for electronically trying on and ordering cosmetic materials;

Fig. 4 is a flowchart/ block diagram showing the principal components of a face segmenter (one kind of image segmenter) , and in particular a face segmenter for identifying where on a human face to apply different types of cosmetic materials;

Fig. 5 is a flow chart showing one embodiment of a method for providing a segmented image, according to the present invention; and

Fig. 6 is a flowchart/ block diagram showing the principal components of an image segmenting system according to the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

The image segmenting invention of the present invention will first be described with respect to identifying segments (regions) in a photograph of a customer showing where cosmetic materials would be applied. Such image segmenting is called here face segmenting in case of a photograph of only a customer's face. Then the preferred embodiment of the image segmenting invention itself will be described, again in particular with respect to face segmenting allowing a customer to electronically try on cosmetics. Finally, other embodiments of the image segmenting of the present invention will be described, and also other applications, besides electronically trying on cosmetics.

Face (Image) Segmenting to Allow Electronically Trying On Cosmetics

Referring now to Fig. 1, a system enabling a customer 12' to electronically try on and possibly order cosmetic materials, such as eyeshadow, lipstick, or blush, includes one or more kiosks 10, which are located in convenient locations, such as in retail stores, shopping malls or in specialty stores, or in even as standalone facilities in parking lots and on street corners. In addition, the system includes a remote electronic store 20, which interfaces with the kiosk 10 via respective interfaces 24 and 13. The system optionally includes a personal computer 14 located in a cosmetics retail location 18, or even in the kiosk 10. Alternatively, a customer 12' can use any personal computer 14, such as a home personal computer, to interface with the system of the present invention. In the preferred embodiment, a customer 12' uses a personal computer from a convenient location, such as the customer's home, to interface with the system of the present invention.

Still referring to Fig. 1, according to the present invention, a customer 12 visits the kiosk 10 where a photograph of the customer is taken, possibly by an assistant, but preferably in a purely automated process (for example by a machine like those that automatically take photographs of tourists at pavilions or other locations visited mostly by tourists) . The photograph is provided to the remote electronic store 20 as a digitized image, along with the customer's password.

It is of course not necessary for a customer to visit a kiosk to have the customer's photograph taken. If the customer has an appropriate camera (high-resolution, color) and equipment needed to provide a photograph as a digital image (equipment that may be integral with a camera, such as a digital camera) , the customer could load a digital image of a photograph of the customer onto a home computer and provide the digital image to the remote electronic store 20 over the Internet .

The remote electronic store includes an image maker 21 that examines the image, as will be described below, to determine what regions of the image correspond to regions of the customer's face where different types of cosmetics would be applied and to produce a set of (binary) masks (one for each region) that can overlay the image and show the customer how the customer would appear wearing different cosmetic materials. The combination of the set of binary masks along with the original image is here called a segmented image. The image maker 21 saves the segmented image and password in a customer database 25. Later, the customer 12' uses a personal computer 14 to access the remote electronic store 20 so as to be able to electronically try on different cosmetic materials, such as eye shadow, lipstick, and blush. The customer provides the customer's password and indicates through choice indicators what cosmetic materials the customer would like to "try on." (The customer would also use choice indicators to place an order.) The image maker then retrieves from the customer database the segmented image based on the customer password and retrieves from a cosmetic database 22 information about the cosmetics selected for electronically ^' (virtually) "trying on" by the customer, and provides to the customer an image, called here a made-up image, of a customer wearing the selected cosmetic materials. If the customer 12' then chooses to purchase the cosmetic material, the customer can provide a credit card number or other billing information to the remote electronic store 20, and the selected cosmetic materials and credit card number are then stored in a cosmetics retail database 16 as an order to be processed. An interface 17 with a retail location then provides the order to an appropriate cosmetics retail location 18, a location where the customer might conveniently pick up and possibly pay for the order. Alternatively, instead of having the customer visit the cosmetics retail location, the cosmetics retail location 18 may ship the ordered cosmetic materials to the customer.

Still referring to Fig. 1, the kiosk 10 includes a turnkey photo system 11, which includes a camera 15 capable of taking a high resolution color photograph of a customer's face. A customer 12 then uses the interface 13 with the remote electronic store 20 to obtain a password. In addition the customer can provide for the first time an indication of what types of cosmetics the customer is interested in. This preference information is called here cosmetics selection data. It can indicate particular preferred cosmetics (by brand name) , or can indicate information useful to the remote electronic store for suggesting particular cosmetics. For example, the customer can be asked to indicate whether the customer plays sports, or is often outdoors, which would indicate that some kinds of cosmetics are more suitable then others .

The kiosk 10 provides to the electronic store, and in particular to the image maker 21 through the interface with the kiosk, the customer's password, image data and cosmetic selection data. Later, when the customer 12' uses the personal computer 14 to electronically try on cosmetic materials, the remote electronic store 20 suggests particular cosmetic materials to the customer. To suggest cosmetic materials, the customer interface 23 pulls' out of the customer database 25 the cosmetics selections data provided by the customer at the kiosk, analyzes the cosmetics selection data and refers to the cosmetics data base 22 for cosmetics product information, indicated as cosmetics data in Fig. 1, based on the cosmetics selection data. In the preferred embodiment, cosmetics data includes information not only about the product itself such as its ingredients, but also data useful to the image maker in determining how to display cosmetic material when it is applied to a human face. The cosmetics data is retrieved from the cosmetics database 22 by the image maker 21 in preparing a made-up image. It is of course also sometimes advantageous to have a customer input further cosmetics selection data through the personal computer 14 while in the course of electronically trying-on cosmetic materials.

In the preferred embodiment, the customer interface 23 also suggests cosmetics materials based on the original (color) image of customer, stored with the segmented image in the customer database 25, whether or not the customer ever provides cosmetics selection data.

Fig. 2 illustrates different regions of a face the image maker 21 (Fig. 1) must identify, regions where different kinds of cosmetic materials would be applied. For example, the image (digitized version of the original photograph taken at the kiosk) will include a region 31 where lipstick would be applied. The image maker might also identify a region

(not shown) where lipliner would be applied. In addition, the image would include regions 32a-b where blush would be applied. Finally, an image would include regions where eye cosmetics would be applied, including for example eyeliner

(in regions 33a-b and 35a-b) and eyeshadow (in regions 34a- b) . In addition, the image maker would identify regions (not shown) where mascara would be applied (on the eyelashes) .

Referring now to Fig. 3, a flow chart is shown indicating the overall process beginning with the customer visiting a kiosk and terminating in a customer ordering cosmetics .

Face (Image) Segmenting Using a Deformable Template and a Steerable Filter Module

The image segmenting of the present invention, referred to as face segmenting in case of identifying only segments (regions) in an image of a face, will now be described in the particular case of identifying segments of a human face where different cosmetic materials would be applied.

Referring now to Fig. 4, a face segmenter 40, a subsystem of the image maker 21 used to identify where in an image (digitized photograph of the customer's face) different cosmetic materials would be applied, is shown in the preferred embodiment as including a template deforming module 41 and a steerable filter module 42. The template deforming module 41 uses as an input the image of the customer acquired at the kiosk 10 and also uses a deformable template for a human face, along with constraints on how the human face can change, from one person to another. For example, a human face will have a typical range of aspect ratios of height to width, so that in varying for example the height, the width can be assumed to be known to within some typical range of values. The template deforming module 41 produces a deformed template which provides an approximate location of the target regions, i.e. the regions where the different cosmetics materials would be applied.

In the preferred embodiment, in producing the deformed template, the template deforming module 41 uses as input not only the original image, but also key points marked on the image by a technician or even by the customer. The key points provide the precise locations of some particular features of the face in the image, such as the center of the pupils of the customer. The deformable template deforms the starting template by fixing the corresponding locations in the deformable template to the precisely located key points in the image, and as the template is deformed in one dimension to fit the key points, it also deforms in the orthogonal direction according to the constraint information, absent some overriding information.

The steerable filter module 42 then uses the deformed template and the original image to more precisely locate the target regions. It does so by essentially identifying for each particular region edges in the image that delineate the region, using the deformed template merely as a guide to locating the different regions. The final product of the steerable filter module is a segmented image, i.e. the original image and in addition a set of binary masks (one for each region) that are to be overlaid on the original image and that include information about where on the image different cosmetic materials would be applied.

In the preferred embodiment, the original image is marked (either by a technician or by the customer) to locate the key points either at the kiosk 10 (see Fig. 1) or at the remote electronic store. Instead of manually locating key points, however, it is also possible to use other technology, such as neural networks, to automatically examine the image provided by the high resolution camera 15 (Fig. 1) so as to precisely locate the key points.

Referring now to Fig. 5, a flow chart is provided indicating the process performed by the image maker 21 (Fig. 1) . In the preferred embodiment the image that is acquired at the kiosk and used as an input to the image maker is a 24- bit color, high-resolution image, along with a set of coordinates locating on the image the key points, which in the preferred embodiment include the center of the iris of each eye, the points making up a vertical line drawn down the center of the face, and the points making up a horizontal line drawn across the corners of the mouth.

Other Image Segmenting and Other Applications

Besides using a deformable template coupled with a steerable filter module, it is sometimes advantageous to use various other methods for providing a segmented image, including principal component analysis (based on modeling images as reducible matrices in a high-dimensional space) , wavelet graph analysis (similar to deformable template matching, except that Gabor wavelet transform localization is included in every node in the deformable template while the deformable template is being adjusted to fit the image) , and neural network based recognition system (for example those using neural networks trained with a series of customer images using back propagation to determine the weights for connections between radial basis functions in a network of such functions) . According to the present invention, whatever particular methods are used, image segmenting is a task bifurcated into two subtasks: finding the general location of a segment (or equivalently approximately determining a region), and then delineating the segment (i.e. more precisely determining the region) .

Referring now to Fig. 6, an image segmenter 60 is shown as including a segment locator module 61 for locating a segment generally and a segment delineator module 62 for delineating the approximately defined and located segment. The segment locator 61 uses as input the image derived from the photograph of a person (or even a non-human animal) , as well as generic guidance on locating one or more segments in the image, i.e. guidance that is useful for an image of the same part of any body, not guidance associated with a particular body. Thus, for example, in the case of the image segmenter of Fig. 4 (called a face segmenter when applied to segmenting the image of a face, compared to other parts of a body) , the generic locating guidance includes a (generic) deformable template and (generic) constraints.

The segment locator 61 may optionally use key points to assist it in its subtask, as in the embodiment of a face segmenter shown in Fig. 3. As mentioned above, however, the need for key points can be eliminated by adjusting the segment locator to accomplish its subtask without reliance on key points. Also, if key points are used, they can be provided automatically by a pre-processor stage (not shown) , such as a neural network based system for identifying in an unmarked image key points of use to the segment locator 61.

The output of the segment locator 61 is segment location information, such as is provided by the deformed template in the face segmenting embodiment of Fig. 4. The segment delineator 62 uses the segment location information along with the original image, unmarked with key points, to more precisely define the segment, ultimately providing a segmented image, i.e. a set of binary masks along with the original unmarked image .

Besides using a deformable template for the segment locator module 61 and a steerable filter module for the segment delineator module 62, as in the face segmenting embodiment of Fig. 4, each module can be based on a different neural network, one trained to recognize the general location of predetermined target regions (such as regions associated with the eyes or lips) , and one trained to precisely discriminate between a target region and its surroundings.

As has been mentioned above, the present invention is intended to comprehend applications involving the segmenting into regions of the image of any part of a body, and not necessarily the body of a human. In addition, an image segmented according to the present invention can be used not only to display how cosmetics would appear on a wearer, but also how other items would appear, including jewelry, clothes (including a hat) , eyeglasses, a hairstyle and treated hair, or even a tattoo or other form of wearable art . The present invention allows electronically trying on any wearable goods and trying out cosmetic services, allowing shopping for such goods and services even from home over the Internet .

It is to be understood that the above-described arrangements are only illustrative of the application of the principles of the present invention. Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present invention, and the appended claims are intended to cover such modifications and arrangements.

Claims

What is claimed is:

1. An image segmenter for segmenting into predetermined regions a digital image derived from a photograph of a person or animal, comprising: a) a segment locator, responsive to the digital image and to generic locating guidance, for providing segment location information; and b) a segment delineator, responsive to the digital image and to the segment location information, for providing a segmented image including a binary mask.

2. An image segmenter as in claim 1, wherein the segment locator is a template deforming module, wherein the segment locator is a steerable filter module, wherein the generic locating guidance is a deformable template and associated constraints, and further wherein the segment locating information is a deformed template.

3. An image segmenter as in claim 1, wherein either the segment locator or the segment delineator is based on a neural network.

4. An image segmenter as in claim 1, wherein either the segment locator or the segment delineator is based on principal component analysis .

5. An image segmenter as in claim 1, wherein either the segment locator or the segment delineator is based on wavelet graph analysis.

6. An image segmenter as in claim 1, wherein the predetermined regions include a lipstick region, an eyeshadow region, and a blush region.

7. An image segmenter as in claim 1, wherein the image segmenter is part of a shopping system allowing a customer to electronically try on wearable goods or electronically try out cosmetic services, the shopping system comprising: a) a means for providing the digital image derived from a photograph of a person; and b) a remote electronic store, responsive to the digital image, the remote electronic store including the segmenting system, the remote electronic store for providing the customer via a display device with madeup images showing how the customer would appear wearing the wearable goods or as a result of the cosmetic services.

8. An image segmenter as in claim 7 , wherein the remote electronic store suggests particular wearable goods or cosmetic services based on the image or based on selection data provided by the customer.

9. A method for enabling a customer to electronically try on wearable goods or electronically try out cosmetic services, comprising the steps of: a) taking a photograph of the customer and providing a digital encoding of the photograph; and b) performing a two-part process to precisely identify pre-determined regions where the wearable goods would be worn or applied or where the cosmetic services would be directed, a first part for approximately locating each of the regions, and a second part to delineate the approximately located regions.

10. The method of claim 9, wherein the first part uses a template deforming module and the second part uses a steerable filter module.

11. The method of claim 9, wherein at least one of the two parts uses a neural network.

12. The method of claim 9, wherein at least one of the two parts uses principal component analysis.

13. The method of claim 9, wherein at least one of the two parts uses wavelet ' graph analysis.

14. A method as in claim 9, further comprising the steps of: a) obtaining from the customer at least some wearable goods or cosmetics services selection data; and b) recommending to the customer particular wearable goods or cosmetics services based on the wearable goods or cosmetics services selection data or based on the customer image .

15. A method as in claim 14, wherein the wearable goods or cosmetics services selection data includes lifestyle information, such as whether the customer often participates in sporting activities.