WO2011121484A1

WO2011121484A1 - Head-pose tracking system

Info

Publication number: WO2011121484A1
Application number: PCT/IB2011/051198
Authority: WO
Inventors: Martinus Hermanus Wilhelmus Maria Van Delden; Igor Berezhnyy
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2010-03-31
Filing date: 2011-03-22
Publication date: 2011-10-06

Abstract

A head-pose tracking system (1) for tracking the headpose of an observer (400) behind a window (100) is provided. The system (1) comprises an IR light source (200) for providing IR light (201) into a waveguide (110). The window (100) comprises this waveguide (110). The window (100) has a sensor side (SS) and an observer side (OS). The waveguide (110) comprises a grid (150) of light outcoupling structures (151). The system further comprises an IR sensor (300) on the sensor side (SS) and a control unit (500). The control unit (500) is arranged to define the headpose of said observer (400) on the observer side (OS) of the window (100), based on (i) IR light reflected by the observer (400) and sensed by the IR sensor (300) and on (ii) IR light coupled out by the grid (150) in the direction of the sensor side (SS) and sensed by the IR sensor (300).

Description

Head-pose tracking system

FIELD OF THE INVENTION

The invention relates to a head-pose tracking system, to applications thereof and to a specific window unit. The invention especially relates to head-pose tracking behind a shop window or behind the window of a showcase, such as in a shop or museum.

BACKGROUND OF THE INVENTION

Tracking of items or persons and movements thereof is known in the art. US 2005201613, for instance, describes a method and system for determining the position and orientation of an object. A set of markers attached to or associated with the object is optically tracked and a geometric translation is performed to use the coordinates of the set of markers to determine the location and orientation of their associated object.

Amongst others, this document describes a method of determining a location of a point on a device, comprising: determining a relationship between a plurality of reference points and at least one point on the device; determining coordinates of the plurality of reference points; and determining a coordinate of the at least one point, based at least in part on the determined relationship and the coordinates of the plurality of reference points.

US 2009115721 describes a system and method for gesture recognition. The system comprises a projector configured to project colorless light and visible images onto a background surface. The projection of the colorless light can be interleaved with the projection of the visible images. The system also comprises at least one camera configured to receive a plurality of images based on a reflected light contrast difference between the background surface and a sensorless input object during projection of the colorless light. The system further comprises a controller configured to determine a given input gesture based on changes in relative locations of the sensorless input object in the plurality of images, and being further configured to initiate a device input associated with the given input gesture.

WO 2007141675 describes a highlighting method and an interaction system which includes at least one controllable light emitting source linked to an item; and a processor configured to turn on the controllable light emitting source in response to user selection of the item. The controllable light emitting source may be embedded in a mat or a strip. The mat may include a matrix of photo detectors or pressure sensors configured to detect the base or footprint of the item when placed on the mat. The periphery of the product or the footprint may be illuminated upon selection of the product. Alternatively or additionally, a background surface behind the product may be illuminated upon selection thereof.

SUMMARY OF THE INVENTION

Head-pose tracking of observers behind a window (i.e. assuming a sensor on one side of the window and an observer or user on the other side of the window), such as behind a shop window has two fundamental issues: (1) the head-pose tracking system may presume that the observer is standing exactly in the middle of the shop window; thus, when an observer (user) moves to one side or the other, the system still presumes the observer to be in the middle and reacts accordingly, resulting in malfunctioning; and (2) when more than one observer is present, the system may not be able to discriminate between the observers. Hence, a disadvantage that may have to be overcome is head-pose tracking based on the assumption that the observer is in front of the window. Alternatively or additionally, another disadvantage that may have to be overcome is that only one observer may be tracked.

Hence, it is an aspect of the invention to provide an alternative head-pose tracking system, which preferably further at least partly obviates one or more of the above- described drawbacks. It is further an aspect of the invention to provide an alternative display unit, comprising an alternative head-pose tracking system, which preferably further at least partly obviates one or more of the above-described drawbacks. It is further an aspect of the invention to provide an alterative window unit that may be used in such an alternative head- pose tracking system.

The proposed invention may solve the two above-mentioned issues by means of introducing a preferably substantially invisible (to the human eye) grid. Such a grid may, in an embodiment, be imprinted into the (shop) window (when the window is used as a waveguide or lightguide) or waveguide surface (when a separate waveguide or lightguide is applied).

In a first aspect, the invention provides a head-pose tracking system for tracking the head pose of an observer behind a window, the head-pose tracking system comprising:

a. a first light source ("IR source"), arranged to provide IR light into a waveguide; b. said window comprising said waveguide, wherein the window comprises a sensor side and an opposite observer side; wherein the waveguide comprises a grid of light outcoupling structures;

c. an IR sensor ("sensor"), arranged on the sensor side of the window;

d. a control unit connected to the IR sensor, the control unit being arranged to define the head pose of said observer on the observer side of the window, based on (i) IR light reflected by the observer and sensed by the IR sensor and on (ii) IR light coupled out by the grid in the direction of the sensor side and sensed by the IR sensor.

This head-pose tracking system may be able to track an observer behind a window, substantially irrespective of the position of the observer along the window. Such a head-pose tracking system may also be able to track more than one observer behind a window.

The system may comprise a single IR sensor which allows estimation of a position of a human head in 3D space. Specifically the system may analyze a video stream (measured by the sensor) and may provide three angular parameters of the head pose: yaw

("shaking" movement), pitch ("nodding" movement) and roll angles (moving of the head to a shoulder). In non-smart systems (earlier versions), all three parameters may be measured relative to the focal axis of the camera. In other words - such non-smart system presumes that a user's head is located exactly on the camera's focal axis. With the present invention, wherein the grid is used as reference, the head may be located also elsewhere, without substantially jeopardizing the performance of the system.

The technology of facial recognition or three-dimensional face recognition is known in the art. In the present invention, the control unit may especially be arranged to define the nose vector based on the head pose of the observer. The "nose vector" is a vector which originates from the nose and is perpendicular to the face plane defined by the three points: middle of the upper lip and two centers of eyebrows) of a person (observer).

Determining and following the position of the nose vector is a specific example of tracking the head pose. The observer is especially a human, and may be an adult or a child. Hence, head-pose tracking may include tracking the nose vector. By following the nose vector, especially the end of the nose vector, the attention of the observer may be tracked (followed).

The observer is tracked behind a window (on the observer side). Herein, the terms sensor side and observer side are applied. As viewed from the sensor, the space between the sensor and the window is indicated as sensor side. This space is upstream of the window. Viewed from the sensor, the space behind the window is indicated as "observer side". This space is downstream from the window (when viewed from the sensor).

The window may for instance be a shop window or a window of a showcase. The window including the head-pose tracking system is herein also indicated as intelligent window (IW), or intelligent shop window (ISW) in the case of application for shop windows.

The window may be a plate ("window pane"), which may be substantially planar but may also comprise one or more convex or concave parts. The plate may be made of transparent material. Herein, the term "window" is to be understood in its general meaning as an item of a transparent material through which an observer can look and see an object (if present) behind the transparent material. The transparent material will in general comprise a material selected from the group consisting of glass and transparent plastic, or other transparent materials, such as selected from the group consisting of glass, (fused) quartz, ceramics, silicones, PET (polyethylene terephthalate), PE (polyethylene), PP

(polypropylene), PC (polycarbonate), P(M)MA (poly(methyl)metacrylate), PEN

(polyethylene naphthalate), PDMS (polydimethylsiloxane), and COC (cyclo olefin copolymer). Preferably, the transparent material of the window comprises glass. Herein, the term "transparent" is used as known in the art, and especially refers to transparency to at least part of the visible wavelength range (about 380-780 nm) and at least part of the IR

wavelength range (about 780 nm to 300 μιη, here especially in the range of 800- 3000 nm, such as for instance up to about 2.2 μιη, or up to about 1.5 μιη).

The intelligence derived from sensing the observer(s) by means of the head- pose tracking system may be used for all kinds of purposes. For instance, one may determine the popularity of a product. This information can be used for marketing and stock control purposes. Optionally, since the head-pose tracking system allows defining x, y and z positions in a 3D space, based on z, the length (or height) of the observer can be estimated, and information may further be differentiated as a function of age. Hence, the control unit is further arranged to define a location of interest on the sensor side of the window, based on the head pose of the observer (more precisely, based on the (location of the end of the) nose vector).

The control unit may in an embodiment be arranged to define a mean location of interest on the sensor side of the window, based on the head poses of a plurality of observers. The mean location may for instance be illuminated (see also below). The control unit may further be arranged to define a plurality of locations of interest on the sensor side of the window, based on the head poses of a plurality of observers. This may be applied when more than one observer is present on the observer side of the window.

The intelligence derived from measuring may also be used for lighting purposes. For instance, if a product ("object") behind the window (on the sensor side) is viewed by an observer (end of the nose vector pointing at the object), it may be illuminated with target lighting. When the observer moves his/her head and observes another object, the head pose tracking system may detect that and change to target lighting of the other object. Hence, the head-pose tracking system may further comprise a second light source, arranged to provide visible light, and the control unit may further be arranged to control lighting of the location of interest by the second light source in dependence on one or more of the head poses of the observer and predefined instructions. The term "location" may refer to part of the space on the sensor side of the window, where for instance one or more objects are arranged. By lighting such a location, the observer may be assisted in observing the location, especially the one or more objects at that location, and/or the attention of the observer may be drawn to observing the location, especially the one or more objects at that location.

The phrase "in dependence on one or more of the head poses of the observer and predefined instructions" may for instance indicate that lighting may be dependent on the head pose of the observer or that further predefined instructions may define the lighting. For instance, predefined instructions may define lighting (of the location) dependent upon the height of the observer (indication of observer being adult or child), upon the time length the observer pays attention to the location, etc. The predefined instructions may define lighting as a function of a mean location of interest on the sensor side of the window, based on the head poses of a plurality of observers. The predefined instructions may for instance also define that the lighting semi-continuously follows the change in head pose (i.e. movement of the (end of the) nose vector). Further, the predefined instructions may include discrimination between a closer observer and a more remote observer. In this way, the closer observer may for instance be followed or predominately followed, and the remote observer may not be followed or may obtain less attention.

Further, the head-pose tracking system may comprise a plurality of second light sources, arranged to provide visible light. The control unit may further be arranged to control lighting of one or more of the plurality of locations of interest by the second light sources in dependence on one or more of the head poses of the plurality of observers and predefined instructions. In this way, a plurality of observers may be "served" by the head- pose tracking system. The term "second light source" may in an embodiment also refer to a plurality of light sources. Such a plurality of second light sources may be applied to illuminate a plurality of locations. The term visible light herein refers to light having a wavelength selected from the range of about 380-780 nm.

The first light source is arranged to provide IR light into a waveguide. The first light source is herein also indicated as "IR source". In general, edge lighting will be applied to provide the IR light of the IR light source into the waveguide. IR light of the first light source is especially coupled into the waveguide under total internal reflection (TIR) conditions.

The IR light source may provide IR light with a narrow bandwidth, such as light generated by an IR LED. However, the IR light source may also provide broadband IR light. In an embodiment, the first light source is arranged to provide modulated IR light. When using modulated light, the control unit may be able to filter out background IR light. In this way, the signal-to-noise ratio of the signal that is used to define the head pose(s) increases. In another embodiment, the first light source is arranged to provide IR light with a tunable wavelength. Applying multiple known wavelengths may also enable background IR light to be filtered out more easily. Hence, also in this way the signal-to-noise ratio may be increased. Further, optionally the control unit may ascribe a specific wavelength (range) to the head pose of a specific observer. This may facilitate discrimination between observers in the case that more than one observer is detected by the head-pose tracking system.

The term "first light source" may also refer to a plurality of first light sources.

Applying more than one IR light source may improve the illumination of the observer(s) with IR light and/or may allow the use of different IR wavelengths (see also above). By using IR light sources at different edges of the waveguide and by using modulated light, IR shadows may be created at the face of an observer, which may again facilitate head-pose tracking. Hence, the head-pose tracking system may comprise a plurality of first light sources arranged to provide IR light into the waveguide.

The control unit is thus connected (in a wireless or wired manner) to the sensor, but may also be connected (in a wired or wireless manner) to the IR light source(s), for instance to control modulation of the IR light, and/or be connected (in a wired or wireless manner) to the optional second light sources, for instance to control lighting of a location in dependence on one or more of the head poses of the observer and predefined instructions.

The waveguide comprises a grid of light outcoupling structures. Preferably, the outcoupling structures are substantially invisible to the human eye, especially at a distance of 0.5 m or larger. The grid may be a non-intrusive grid. The grid may comprise a series of substantially IR light reflecting and/or scattering (translucent) markers, thereby providing a (known) (dot-to-dot distance) reference grid (of known dimensions). Examples of shapes of outcoupling structures include a circle, triangle, additive symbol (+), cross (X), square, hexagon, and combinations thereof, and so on. In fact, any shape will do as long as it can be clearly distinguished from more common structures at the window's surface, such as for example due to staining, and be processed fast by recognition/identification/vision-based systems. Such outcoupling structures will allow IR light to be coupled out in two directions: the sensor side direction and the observer side direction. In an embodiment, outcoupling at one or more of the outcoupling structures is preferably to one of both sides, but a minor part is coupled out at the one or more outcoupling structures to the other side. Further, different outcoupling structures may be used to couple IR light out to both sides of the window, respectively.

The grid is preferably regular, although in principle also an irregular grid may be applied. Preferably, the grid is regular, or comprises a combination of a regular and irregular grid.

The advantage of the IR visible markers is that these can be easily recognized and processed by the used imaging software, and enable on-the-fly calibration and

subsequent distance-to-user estimation. In addition, computation times are reduced whilst the robustness increases. The head-pose tracking system recognizes the grid by the outcoupling of IR light at those structures in the direction of the IR sensor. The grid may then be used as reference for defining the head pose(s) (such as by the nose vector) of one or more observers. The grid will in general be a regular arrangement of outcoupling structures. The number of outcoupling structures may be in the range of 25-100 /m² window.

The window may have a certain area. Preferably, the waveguide (or light guide) is arranged over at least 50% of the window area, such as at least 80%>. In an embodiment, the window is the waveguide. In such an embodiment, the waveguide is thus arranged over substantially the entire area of the window.

As mentioned above, the window may comprise a transparent plate. In the embodiment in which the window is the waveguide, the transparent plate is thus the waveguide: the window is used as window and as waveguide.

In another embodiment, the window comprises a transparent plate and waveguide foil laminated to the transparent plate. Here, the foil is used as waveguide. As mentioned above, the foil is preferably applied over at least 50%> of the window area, such as at least 80%. Preferably, substantially the entire transparent plate comprises such waveguide foil.

Note that in embodiments in which the window comprises a separate waveguide, such a waveguide will in general only be applied to one face of the window (more precisely one face of the transparent plate).

Hence, the window may in an embodiment be a window pane, wherein the window pane is (also) used as waveguide, and the window may in an embodiment be a window pane, wherein the waveguide is arranged to the window pane (especially on one of the two opposite surfaces of the window pane).

The IR sensor on the sensor side receives IR light coupled out at the outcoupling structures in the direction of the sensor side, and may receive IR light reflected from (the face) of an observer. Especially eyes may reflect IR well. The term "IR sensor" may in an embodiment also refer to a plurality of IR sensors. Hence, optionally, the system may comprise a plurality of IR sensors.

Hence, in a further aspect the invention also provides a display unit comprising a display window and the head-pose tracking system according to any one of the preceding embodiments, wherein the display window is the window of the head-pose tracking system. The display unit is selected from the group consisting of a shop window and a showcase. Showcases may for instance be used in shops, museums, etc.

In yet a further aspect, the invention provides a window unit comprising: a. a window comprising a waveguide, wherein the window comprises a sensor side and an opposite observer side; wherein the waveguide comprises a grid of light outcoupling structures enabling IR light in the waveguide to be coupled out in directions from the sensor side and the observer side; and

b. a first light source, arranged at an edge of a window, and arranged to provide

IR light into a waveguide.

Such a window unit may be installed in a display unit to form the ("smart") window thereof. When combining this with the IR sensor and a control unit, one may be able to provide the herein described head-pose tracking system.

In yet a further aspect, the invention also relates to a method of head-pose tracking of an observer behind a window, wherein the method comprises providing a head- pose tracking system as described herein, coupling IR light out of the window by the outcoupling structures comprised by the waveguide, sensing with the IR sensor the head pose of the observer behind the window, based on (i) IR light reflected by the observer and sensed by the IR sensor and on (ii) IR light coupled out by the grid in the direction of the sensor side and sensed by the IR sensor. In an embodiment, the method may further comprise

illumination of a location on the sensor side of the window, based on a location of interest of the observer as determined from the head pose.

For use of the head-pose tracking system it may be necessary to first perform an initialization procedure, wherein the control unit calibrates the head-pose tracking system and learns the grid.

The above described embodiments all relate to a first light source arranged to provide IR light, outcoupling structures arranged to couple IR light out of the waveguide, an IR sensor, IR light reflected by an observer, IR light coupled out at the grid, etc. However, in a further embodiment, visible light is applied. Hence, in yet a further aspect, the invention provides a head-pose tracking system for tracking the head pose of an observer behind a window, the head-pose tracking system comprising: (a) a first light source, arranged to provide visible light into a waveguide; (b) said window comprising said waveguide, wherein the window comprises a sensor side and an opposite observer side; wherein the waveguide comprises a grid of light outcoupling structures; (c) an visible light sensor, arranged on the sensor side of the window; and (d) a control unit connected to the visible light sensor, the control unit being arranged to define the head pose of said observer on the observer side of the window, based on (i) visible light reflected by the observer and sensed by the visible light sensor and on (ii) visible light coupled out by the grid in the direction of the sensor and sensed by the visible light sensor, including all other herein described embodiments and aspects, but here in relation to visible light of the first light source and for the sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying schematic drawings in which corresponding reference symbols indicate corresponding parts, and in which:

Figs, la- If schematically depict embodiments of the head-pose tracking system and some principles and options;

Figs. 2a-2b schematically depict some embodiments of the

window/waveguide; and

Figs. 3a-3b schematically describe an embodiment in which lighting semi- continuously follows the head pose. DESCRIPTION OF PREFERRED EMBODIMENTS

Fig. la (top view) schematically depicts an embodiment of a head-pose tracking system. The head-pose tracking system is indicated with reference 1. The head-pose tracking system 1 is used for tracking the head pose of an observer 400 behind a window 100. Head poses may be defined by using the nose vector, indicated with reference 401.

The head-pose tracking system 1 comprises a first light source 200, arranged to provide IR light 201 into a waveguide 110. The head-pose tracking system 1 comprises window 100, which window comprises the waveguide 110. Here, in this schematically depicted embodiment, the window 100 is the waveguide 110; or the other way around, the waveguide 110 is window 100. The observer is on the observer side OS of the window 100. The window comprises a sensor side SS and, on the opposite side of the window 100, an observer side OS.

The waveguide 110 comprises a grid 150 of light outcoupling structures 151. IR light 201, coupled into the waveguide 110, is coupled out at the light outcoupling structures 151. The light is coupled out to the observer side OS and the sensor side SS. IR light coupled out to the observer side OS is indicated with reference 211; IR light coupled out to the sensor side SS is indicated with reference 221. IR light reaching the head of the observer 400 may be reflected. This reflected IR light is indicated with reference 231. Hence, reference 201 refers to IR light in general, whereas reference 211 refers to IR light coupled out to the OS side, reference 231 refers to such outcoupled light 211 reflected by the observer 400, and reference 221 refers to IR light 201 coupled into the waveguide 110, but coupled out to the SS side.

The head-pose tracking system 1 further comprises an IR sensor 300, arranged on the sensor side SS of the window 100. This sensor 300 is arranged to sense IR light 221, i.e. IR light 201 coupled out at the outcoupling structures 151 to the sensor side SS and is arranged to sense IR light 231, i.e. IR light 201 coupled out at the outcoupling structures 151 to the observer side OS as IR light 211 and reflected at the observer's head) as IR light 231 (which is then transmitted through the window 100 and sensed by the sensor 300).

The head-pose tracking system 1 further comprises a control unit 500 connected (in a wired or wireless manner) to the IR sensor 300. The control unit 500 is especially arranged to define the head pose of said observer on the observer side OS of the window 100 (such as by the nose vector 401), based on (i) IR light reflected by the observer and sensed by the IR sensor and on (ii) IR light coupled out by the grid 150 in the direction of the sensor side SS and sensed by the IR sensor 300. The schematically depicted head-pose tracking system may be part of a display unit, which is indicated with reference 800. The combination of window 100, waveguide 110 and first light source(s) 200 is indicated as window unit 700.

The head-pose tracking system 1 may allow observers to move freely within the perimeter of the head-pose tracking system while controlling it, without compromising the performance of the system, and may allow solving multi-user problems (by both taking and handing over control).

In indoor environments (shopping malls) lighting conditions may be relatively static and, hence, the influence of disturbing secondary sources (sun, cars, other light sources) may be small, and/or constant within limits. Thus, the system may function as intended. In "outdoor" (street) environments, the dynamics of disturbing light sources may be much larger. In order to cope with these disturbing conditions, during which the system may fail to detect a person in front of the shop windows, several strategies may be employed to improve detection.

In an embodiment, to enhance observer detection by still using only one IR sensor, the IR light source(s) may be modulated, thereby emitting modulated IR-light into the window pane and consequently also the illumination of the observer in front of the window is modulated.

Since the modulating signal (having a waveform; and comprising frequency, duration, pattern, pulse-width, amplitude modulation etc.) is known, the IR sensor can be setup to synchronize with this modulation, thereby enhancing nose vector detection and tracking of the observer (in front of the window). For example, by subtracting two images, one taken with modulation and one without modulation, the observer in front of the shop window is enhanced/revealed.

In another embodiment, as a second improvement to enhance observer detection, and more precisely, to enhance accurate detection of the nose vector (symmetric shadow), for example disrupted by a directional (parasitic) illumination source (a low sun, or a spot light), two (or more) IR light sources may be used, each modulated at their own intensity, thereby yielding a "symmetric" shadow.

In this case, the IR illumination sources may for example be embedded IR-

LEDs, located at opposing ends of one and the same light guide, thereby each being capable of casting their own part of the nose shadow. In addition, when grooves are used to couple out the IR-light, the shape of the markers (more accurately the perpendicular angle to the window pane and/or the top angle of the V-groove itself) may be optimized to substantially couple-out light arriving from one direction/ or couple out light into a given direction. Thus, for two or more IR light sources, one or more known (directional) grids may be used, or just one known mixed grid (top and bottom half of a cross-hair optimized for the one or the other illumination source).

In a further embodiment, again intended as an improvement to enhance person detection, and more precisely to enhance accurate detection of the nose vector (symmetric shadow), multiple wavelength IR sources may be used. These sources may be operated alternately, at the same time, modulated or not, or may be positioned at opposing sides of the light guide. Detection strategies as described above may be deployed, or when two or more wavelengths are driven simultaneously, intensity ratio detection (to reduce the disturbing impact of parasitic illumination sources) may be carried out.

A still further embodiment, again being an improvement to enhance system performance, has as a beneficial effect of a multiple wavelengths set-up, that it may be used to enable multi-user operation, in that one wavelength is assigned to one person, and the second to a second person, and so on, while ISW control can still be done by using only one IR-camera. Also the use of multiple wavelengths is possible to distinguish between an actual observer and an "accidental" person passing by.

Of course, to further increase system integrity and robustness (each of) the IR light sources may be modulated (waveform and intensity etc. also deployed to reduce power consumption), as in previous embodiments and/or a combination of previous embodiments. And, in addition, in this particular case of multiple wavelengths, each light source may be driven individually and independently of another light source, or be driven as an

interdependent ensemble of light-sources, thereby allowing for the digital modulation of the emitted light spectrum, which may for example also be used to exchange information with an electronic device located outside the shop window, or with a device (intentionally) carried around by the observer outside.

Fig. lb schematically depicts in more detail and in a top view the window 100, first light source(s) 200 and sensor 300 of the head-pose tracking system 1. IR light 201 of light source(s) (200) is coupled into the waveguide (here again the window 100 is the waveguide 110) under TIR conditions and is coupled out at outcoupling structures as IR light 221 to the sensor side SS and as IR light 211 to the observer side OS. Here the window 100 is also waveguide; the window pane is used as waveguide.

Fig. lc schematically depicts in more detail and in a front view (seen from the observer's side), window 110 with the grid 150 of outcoupling structures 151. In reality, those outcoupling structures may be invisible to the observer. By way of example, the sensor 300 is seen through the window 100.

Fig. Id schematically depicts in more detail a side view of an embodiment of the outcoupling structure 151. IR light 201 is reflected at the outcoupling structure 151 and leaves the waveguide 110 in a direction (here in the sensor side SS direction as IR light 221), but part of the reflected IR light is reflected back into the waveguide and leaves the waveguide 110 in an opposite direction (here in the observer side OS direction as IR light 211). Hence, the outcoupling structures are used to couple IR light 201 out to both sides of the window 100.

With respect to the grid 150, more precisely the outcoupling structures 151, examples of shapes may include: circle, triangle, additive symbol (+), cross (X), square, hexagon, and combinations thereof, etc. In fact, any shape will do as long as it can be clearly distinguished from more common structures at the window's surface, such as for example due to staining, and be processed fast by recognition/identification/vision-based systems.

In order to only couple IR light towards the sensor 300 at the markers, a so- called "patterned" light guide may be used. It may consist of a sheet of IR light-guiding material, such as a pane of glass, side-lit by one or more IR LEDs. Thus IR light may be coupled into the window pane at narrow angles such that it remains trapped within the pane due to total internal reflection (TIR). Thus, substantially no light is coupled out until the IR light rays encounter a marker at which the angle of reflection is altered beyond the critical angle, leading to coupling out the IR light, which is in turn detected by the IR camera.

Another advantage of this set-up is that the intensity of the outcoupled light can be adjusted to the ambient light conditions, thereby allowing better and dynamic recognition of the markers, and, if required, can even be reduced to zero after calibration. Alternatively, the IR- LEDs may be modulated to be synchronous with the frame rate of the camera, every frame or every other frame, or any other sequence of choice, such that easy computation and cross- comparison is facilitated.

Methods of applying suitable markers at the surfaces of materials transparent to IR and visible light are numerous, and may include spraying, painting, printing, selective coating and dipping. Other (less favorable) methods for marker generation comprise hot, cold and thermal embossing, groove cutting, indenting, which may (in part) be visible to the human eye. More importantly, those (less favored) techniques may cause diffractive patterns, in particular in combination with light originating from narrow wavelength light-sources such as TL tubes and LEDs. Note that some elements of both methods can be applied to existing window surfaces, either directly or by add-on (adhesive) foils. Alternatively, the markers may be applied, during a dedicated manufacturing process, to the inner pane of double glazing, together with an LED.

Preferably, the material enabling the markers substantially reflects/scatters IR light, said material for example being (translucent) (retro-reflective) paints and glass beads, suspended in an oil-based solvent additionally comprising a fixing adhesive (for example see US4916014, and Yuki Nota, Yasuyuki Kono, Augmenting Real-world Objects by detecting "invisible" visual markers, UIST '08, Oct 19-22, 2008, Monterey CA, USA), with the dried/cured adhesive/lacquer no longer dissolving in most water and hydrocarbon-based fluids being used for cleaning.

Fig. le schematically depicts an embodiment similar to that schematically depicted in Fig. la, but now with a plurality of observers 400. Further, the head-pose tracking system 1 further comprises a plurality of second light sources 600, which are arranged to provide visible light 601. This schematic drawing is used to further illustrate the use of the head-pose tracking system 1 to illuminate specific locations based on (amongst others) the head pose of one or more observers 400. Here, the observers 400 look at different objects 10, which objects are indicated with references 10(1) and 10(2). The objects are at different locations. The head-pose tracking system 1 is able to determine the head pose of one or more observers 400 (for instance by their nose vector(s)) and can define on the sensor side SS of the window 100 the location(s) the observer(s) are looking at. Based on this information, the control unit 500 controls the illumination of those locations (here especially the objects) by light 601 of the one or more second light sources 600.

Fig. If schematically shows (in a top view) how the grid 150 can be used. This Figure illustrates the math of computing the horizontal coordinate of the nose vector pointing at the product 10(2) positioned behind the shop window's glass. The general formula for that is as follows: x = d2 * tan(a) + AS^* (1) Where:

1. x is the horizontal distance between the focal axis of the camera 300 and the position of the product.

2. dl = d - d\ , where d\ is the distance between the two axes passing through respectively the camera 300 and the product 10(2) and parallel to the window 100, and d is the distance from the observer's face to the camera 300 and is computed as a function of camera sensor hardware properties and detected observer's face bounding box size in pixels. One way of obtaining d is via a one-time calibration procedure during which an observer is asked to stand at known distances and the system computes the area of the face of the observer. What can also be used is the typical distance from the glass window 100 that people use to observe products and the average adult face area in pixels for a given camera and lens mount.

3. a is the yaw angle determined by software from the image taken by the camera. This angle is an angle between the focal axis of the camera 300 and the nose vector.

4. AS is the shift between an observer's current position and the focal axis of the camera. The AS of the observer is determined using above mentioned reference grid.

Two cases of the observer's position are presented in the Figure:

1. The observer is standing right opposite the camera (position A). In this case AS = 0 thus x = d2 * tan(al) .

2. The observer is standing to the right of the camera's focal axis (position B). In this case formula (1) is used as is, thus x = d2 * tan(a2) + AS .

When the reference grid is used for determining the AS of the observer, generally the following formula is employed:

AS Ag *— (2) d3

Where:

1. Ag is the distance obtained from a distance measurement at the grid 150 (not shown) of window 100, and more specifically the distance between the position of the observer as it appears at the window 100 and the focal axis of the camera 300.

2. d3 is the distance between the axis passing through the camera 300 and the window 100.

Figs. 2a-2b schematically depict some embodiments of the window 100 for use in the head-pose tracking system 1. Fig. 2a schematically shows the window 100 comprising a transparent plate 120 (often indicated as "window pane") and the

waveguide 110 arranged to the window pane, for instance as transparent foil attached to the window's transparent plate 120. In Fig. 2b however, the transparent plate 120 (window pane) also has the function of waveguide 110. Figs. 3a and 3b schematically depict an embodiment of how light may semi- continuously follow the head pose of an observer (i.e. follow the nose vector, or more accurately, follow the end of the nose vector of the observer).

In sensing ubiquitous systems, user interaction controls are not always obvious, since the intention of such systems is to unobtrusively detect what the user is doing and provide an adequate system reaction. However, to ensure a dialogue with the user, feedback that clearly signifies system status may be essential. A solution is used which is simple but which also allows sufficient quality of the desired user-system interaction feedback. The solution offers concrete advantages for interactive shopping windows (ISW), such as flexibility in shopping window arrangements (free choice of items' location as long as an item can be highlighted by means of either a controllable light tile or a spot light beam). In general, this solution allows sufficient quality feedback by using only discrete controllable light sources/reflections. This allows simplicity and flexibility in system arrangement. The light sources are used essentially for user-system interaction feedback but could combine this function with general or ambient lighting of the shop window.

The solution may essentially be a way of controlling RGB values or intensity (or both) of the distributed-in-space (both 2D and 3D) light sources (or light reflections) which represent system feedback output elements. For the case of ISW, the systems' status is to be manifested by means of adjusting color or intensity (or both) of the distributed light sources in accordance with the distance from a user's focus of attention to the system output elements (light tiles) combined with products.

As an illustration, an embodiment is shown dedicated to ISW, wherein each product is highlighted with a separately controllable light tile placed behind or under each product. Head-pose tracking is used to derive the head-pose vector, of which the coordinates are relative to those of the shop window and products placed in it, and consequently tell which product a user is looking at. Further, since the system knows which product the user is looking at it can display product details on the shopping window screen.

However, without appropriate feedback the link between the dynamic head vector change and information displayed on the shop window is unclear to the user. A discrete feedback when a light tile corresponding to the recent ly/currently observed product changes its color, proved to be unnoticeable. Therefore, since due to obvious reasons continuous feedback is impossible to achieve by means of light tiles, we propose to apply a semi-continuous feedback. In an embodiment, the lighting intensity gradually changes when the head pose changes. Referring to Fig. 3a, the observer's attention may be focused on the top left part. Lighting around the observer's location of interest is dimmed, and positions further away are not illuminated. When the attention changes to the centre part, see Fig. 3b, lighting follows. Hence, illumination may follow semi-continuously the movement in attention of the observer.

The term "substantially" herein, such as in "substantially all emission" or in "substantially consists", will be understood by the person skilled in the art. The term

"substantially" may also include embodiments with "entirely", "completely", "all", etc.

Hence, in embodiments the adjective" substantially" may also be removed. Where applicable, the term "substantially" may also relate to 90% or higher, such as 95% or higher, especially 99% or higher, even more especially 99.5% or higher, including 100%. The term "comprise" includes also embodiments wherein the term "comprises" means "consists of. The term "plurality" refers to two or more. A phrase like "a plurality of light sources to illuminate a plurality of locations" does not imply that the "pluralities" refer to identical numbers. For instance, four light sources may be applied to illuminate two locations.

Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein.

The devices referred to herein are amongst others described during operation. As will be clear to the person skilled in the art, the invention is not limited to methods of operation or devices in operation.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "to comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. A head-pose tracking system (1) for tracking the head pose of an observer

(400) behind a window (100), the head-pose tracking system (1) comprising:

a. a first light source (200), arranged to provide IR light (201) into a waveguide

(110);

b. said window (100) comprising said waveguide (110), wherein the window

(100) comprises a sensor side (SS) and an opposite observer side (OS); wherein the waveguide (11) comprises a grid (150) of light outcoupling structures (151);

c. an IR sensor (300), arranged on the sensor side (SS) of the window (100); d. a control unit (500) connected to the IR sensor (300), the control unit (500) being arranged to define the head pose of said observer (400) on the observer side (OS) of the window (100), based on (i) IR light reflected by the observer (400) and sensed by the IR sensor (300) and on (ii) IR light coupled out by the grid (150) in the direction of the sensor side (SS) and sensed by the IR sensor (300).

2. The head-pose tracking system according to claim 1, wherein the control unit is arranged to define the head pose of the observer, based on a nose vector (401).

3. The head-pose tracking system according to claim 1, wherein the control unit is further arranged to define a location of interest on the sensor side of the window, based on the head pose of the observer.

4. The head-pose tracking system according to claim 3, further comprising a second light source (600), arranged to provide visible light (602), and wherein the control unit is arranged to control lighting of the location of interest by the second light source (600) in dependence on one or more of the head poses of the observer and predefined instructions.

5. The head-pose tracking system according to claim 1, wherein the first light source is arranged to provide modulated IR light.

6. The head-pose tracking system according to claim 1, wherein the first light source is arranged to provide IR light with a tunable wavelength.

7. The head-pose tracking system according to claim 1, comprising a plurality of first light sources arranged to provide IR light into the waveguide.

8. The head-pose tracking system according to claim 1, wherein the control unit is further arranged to define a mean location of interest on the sensor side of the window, based on the head poses of a plurality of observers.

9. The head-pose tracking system according to claim 1, wherein the control unit is further arranged to define a plurality of locations of interest on the sensor side of the window, based on the head poses of a plurality of observers.

10. The head-pose tracking system according to claim 9, further comprising a plurality of second light sources, arranged to provide visible light, and wherein the control unit is arranged to control lighting of one or more of the plurality of locations of interest by the second light sources in dependence on one or more of the head poses of the plurality of observers and predefined instructions.

11. The head-pose tracking system according to claim 1 , wherein the window is the waveguide.

12. The head-pose tracking system according to claim 1, wherein the window comprises a transparent plate and waveguide foil laminated to the transparent plate.

13. A display unit (800) comprising a display window and the head-pose tracking system (1) according any one of the preceding claims, wherein the display window is the window (100) of the head-pose tracking system (1).

14. The display unit (800) according to claim 13, wherein the display unit (800) is selected from the group consisting of a shop window and a showcase.

15. A window unit (700) comprising:

a. a window (100) comprising a waveguide (110), wherein the window (100) comprises a sensor side (SS) and an opposite observer side (OS); wherein the waveguide

(110) comprises a grid (150) of light outcoupling structures (151) enabling IR light (201) to be coupled out of the wave guide (110) in directions from the sensor side (SS) and the observer side (OS); and

b. a first light source (200), arranged at an edge of a window (100), and arranged to provide the IR light (201) into the waveguide (110).