WO2008145957A2

WO2008145957A2 - Inter-active systems

Info

Publication number: WO2008145957A2
Application number: PCT/GB2008/001639
Authority: WO
Inventors: Ramesh Jagannathan; Alexander Loui; Christopher Peter Moore; Rajesh Mehta; Gilbert Allen Hawkins; Peter Thomas Fry; Graham Kiddle
Original assignee: Eastman Kodak Company
Priority date: 2007-05-26
Filing date: 2008-05-12
Publication date: 2008-12-04
Also published as: TW200917095A; WO2008145957A3; GB0710132D0

Abstract

A system comprises a device having at least one passive electrodynamic sensor capable of sensing existing bio and ambient electric fields and changes induced in these fields in the vicinity of the sensors. The system also includes means for transmitting the resulting signals to other systems in order to initiate some activity in that other system.

Description

Inter-active Systems

FIELD OF THE INVENTION

This invention relates generally to the area of human - machine interactions. More specifically, the invention provides a means to use gestures, electroencephalographic ("EEG"), electro-myographic ("EMG"), electro- oculargraphic ("EOG") and other bio-potentials such as the electrocardiogram (ECG) to interact with the functioning of electromechanical and electro-optical systems. The invention relates to the field of non-verbal thought communication to electrical, electromechanical and electroptical devices. The invention also relates to, in some special cases, non-verbal communication of thoughts to external systems, which then convert the signals to sound and speech.

The invention also relates to the interaction of living forms and/or dielectric objects with existing electric fields and using the extent of interaction to create a logical sequence of commands to activate and manage the functioning of electro-mechanical and electro-optical systems.

BACKGROUND OF THE INVENTION

The field of bio-sensing relates to detection of electric potentials generated within the human body. For example, electroencephalographic ("EEG") biopotentials or brainwaves demonstrate continuous electrical activity in the brain. The intensities of the brain waves or EEG range from zero to 300 microvolts, in a frequency range of 1-50 Hz. EEG signals are classified in frequency groups, namely α (8 Hz to 13 Hz), β (14 Hz to 50 Hz), θ (4 Hz to 7 Hz), and δ (below 3.5 Hz). Various EEG frequency groups have been correlated to mental states (sleep, relaxation, active thought, etc). EEG spectra from different areas on the scalp can be qualitatively related to a function of a specific part of the brain.

The skeletal muscle contractions are associated with changes in the muscle nerve fibre membrane potential called action potential. This results in a flow of electrons up to the skin surface. The average of electric potentials at the skin surface due to simultaneous muscle actions is referred to as electromyographic biopotentials (EMG).

The average EMG signals are generally below 100 Hz. They are typically detected at the site of muscle activity. EMG signals have so far been only used to activate an on/off switch to control an external device.

Electrooculography (EOG) is a technique for measuring the resting potential of the retina. The resulting signal is called the electrooculogram. The EOG does not represent the response to individual visual stimuli.

Usually, pairs of electrodes are placed either above and below the eye or to the left and right of the eye. If the eye is moved from the centre position towards one electrode, this electrode "sees" the positive side of the retina and the opposite electrode "sees" the negative side of the retina. Consequently, a potential difference occurs between the electrodes. Assuming that the resting potential is constant, the recorded potential is a measure for the eye position. EOG signals have been used to track and activate cursors on a computer screen but with limited success due to the slowness of response and signal drift. Moreover, the technology development was hindered by head positional changes during eyeball movement, leading to corrupted signals.

Also, the sensors used for these measurements are gel based AgCl electrodes which are sticky and not very nice to use in consumer and commercial product applications. The sensitivity of these sensors is also not very high and they have to be placed in locations on the face that are unattractive and have low market value.

Therefore, even though there has been significant work with EEG, EMG and EOG biopotentials for several decades, there have been few practical results.

Examples of apparatus and methods for controlling devices with brain and bio-potentials are described in US 5692517, US 6636763, US 5474082, US 7078911 and WO 03/048789. hi some of these documents a user wears a headband containing several sensors for detecting brain and body signals at the forehead. The signals are transmitted to an "interface unit" for amplification, digitization and decoding for machine control. Multiple function control is possible.

Additional capabilities were introduced to make this technology more accessible and useful for a greater number and diversity of individuals. Graphical user interfaces (GUI) were developed and the requisite brain and body control skills were made more intuitive and comprehensible. Novel methods were created to analyze signal characteristics and signal patterns for cursor management for display applications. For example, methods to achieve left and right mouse button clicking, cursor speed and activation and deactivation of the hands free mouse/keyboard controller were developed. In US 5692517 the focus was solely on using the EEG signals generated through eye lid opening and closing to move a cursor on a display screen.

US 5914610 discloses the design of a sensor which generates an electric field. A person or object to be sensed intercepts a part of the field emanating from the sensor, the amount of field intercepted depending on the size and orientation of the sensed person, whether or not the person provides a grounding path, and the geometry of the distributed electrodes. Coupled with the nonlinear spatial dependence of the field and multiple electrodes the sensor is configured to detect not only static positions and orientations, but also motion through a defined space.

US 7078911 and WO 03/048789 use high impedance circuitry for their sensors. Whilst US 7078911 reports the disclosed sensor as one which measures the ambient electric fields without disturbing them in reality it does, because it draws a small but finite charge current from the system in order to make the measurement.

It is to be appreciated that most of the sensors available now are what are known as active systems. They emanate an electromagnetic field around them and then detect changes in the field due to interaction with the immediate environment. They are powered systems and are therefore limited in their applications. - A -

WO 03/048789, on the other hand, uses a displacement current technique, which is actually a rate of change of voltage measurement and therefore does not load the system. In concept, it acts as a perfect voltmeter being able to measure very low, bioelectric potentials.

PROBLEM TO BE SOLVED BY THE INVENTION Until recently it has not been conceivable to detect ambient electric fields in a reliable, tuneable and low cost fashion using passive high impedance e-field sensors. The rapid emergence of digital technology has created needs for pervasive sensing and computation. As stated above, most of the sensors available now are what are known as active systems. They are powered systems and are therefore limited in their applications. There is a need for systems that detect ambient electric fields using passive miniaturised e-field sensors.

SUMMARY OF THE INVENTION

The present invention is directed towards a system that provides interaction between humans, or possibly animals, and machines, in which data is derived from bio-signals or movement to control a machine in a wireless fashion. A system of miniaturized passive e-field sensors is provided that detects perturbations to the ambient electromagnetic fields by dynamic body potentials, enabling control of a device in a wireless fashion.

According to the present invention there is provided a system comprising a device with at least one passive electrodynamic sensor capable of sensing existing bio and ambient electric fields and changes in these fields in the vicinity of the sensors induced by living forms and means for transmitting the resulting signals to other systems to initiate some activity in the other system.

The invention further provides a method of enabling a human-machine communicative relationship based on sensing the motion of an individual in an electric field by at least one electric field sensor associated with the machine, the method including the steps of; establishing a sound enabled communicative relationship between the individual and the machine, the relationship comprising audio feedback from the machine responsive to physical motion of the individual and acknowledgement of the feedback by the individual; and exercising one or more defined motions by the individual which are uniquely acknowledged by the machine by actions taken by the machine in addition to the audio response.

ADVANTAGEOUS EFFECT OF THE INVENTION

The system of the invention is passive, low cost, non-invasive and can be integrated with existing technology to enable pervasive sensing. It uses an ultra high impedance sensor and can detect very weak fields such as biopotentials in a robust manner with excellent signal to noise ratio. The sensor system is highly tuneable so that in an array of such sensors, each one can be tuned to pick up signals emanating from a different source without any cross talk. Since the sensor system does not draw any charge current during the detection process, a true picture of the object, which is recognized, is obtained.

The current invention uses passive sensors which can be non invasively distributed everywhere and thereby enable pervasive sensing and computing in a true sense. They detect the perturbations to the existing electromagnetic fields present in every day environments.

Advantageously, the extent and reliability of the human-machine interface enabled in accordance with the present invention is much greater than that achieved previously and requires fewer computational resources. Additionally, the type of communication modes and their functionality can be tailored to individual users or can be optionally selected from a common body of user modes available to a multiplicity of users. These communication modes and functionalities are enabled on a multiplicity of devices which are responsive to the common body of user modes available to a multiplicity of users or to the communication modes specifically tailored to individual users. Further, the human-machine communicative relationship established in accordance with the present invention accounts automatically for the case in which objects held by the engaged individual are themselves moved.

BWEF DESCRIPTION OF THE DRAWINGS The invention will now be described by way of example with reference to the accompanying drawings in which;

Figure 1 illustrates a method of practice of the invention; and Figure 2 illustrates a further method of practice of the invention. In Figure 1 sensors 2 are built into the rim of a baseball cap 1 worn by a user. A wireless transmitter 4 is worn in the ear of the user, the transmitter being in connection with the sensors 2. The transmitter is tuned such that it is capable of sending signals from the sensor to a set top box 6 placed on top of a television 8. The set top box 6 is hardwired to the television 8 and can therefore operate and control the television functions based on commands received from the transmitter 4.

Figure 2 illustrates only the sensors and transmitter. The receiver and actuation/control processes can be similar to those shown in Figure 1. In Figure 2 the sensors 2 are worn as a fashion accessory, in this case as jewellery. The transmitter is also worn in the ear but in this case may be covered by a headscarf. The sensors 2 measure EMG signals. The user can control and move images on a display screen by controlled movements of the eye or blinking.

DETAILED DESCRIPTION OF THE INVENTION

The present invention uses miniaturized, mass produced ultra-low noise, ultra-high input impedance passive e-field sensors which do not load the system being detected. The sensors do not draw any current charge, i.e. no electron transport is involved. The sensors measure a displacement current, which is actually a rate of change of voltage measurement. The sensors are passive, i.e. they do not create a reference electric field. They are designed and tuned such that they can measure extremely low electric potential such as bio potentials. The sensors operate on the principle of feedback enhanced and stabilized electrometer based amplifier technology. The sensors can be used remotely or in contact.

The sensors are designed to measure, in real time, the primary bio-signals, namely, EEG, EOG, ECG and EMG and use them as digital on/off and combinations of several of the on/off signals to manipulate the functional features of various systems. More specifically, a system is envisioned which consists of a miniaturized sensor or a sensor array which is part of an integrated electrical circuit.

The general circuit system, as illustrated broadly in Figure 1 , includes a sensing system 2, data processing/transmission system 4, a receiver/actuation system 6 and an output system 8 which is actuated or controlled.

All these so called boxes are miniaturised so that they can be integrated in a non-invasive fashion into normal, everyday things such as clothing, caps, headbands, mobile phones, wrist watches, shoes, sandals, eye glasses, hearing aids etc. The specific utility and purpose of the sensor circuit will influence the nature and location of the system.

The present invention also has applications in gesture sensing. It enables applications that involve interpreting motions remotely from the sensor array in an arbitrary electrical environment. For such applications, it is envisioned that a communicative relationship is pre-established between the engaged machine and the engaged individual. In a human-machine communicative relationship, the machine is able to track and uniquely interpret at least a subset of motions of the individual and respond accordingly. Typically, it is in the form of a common database that maps individual's body motions or facial expressions to electrical potential measurements and finally to intended actions. Such a database may be separately pre-established by a variety of commonly available means such as simultaneous optical or audio, and electric field sensing. When e-sensors are worn in fixed body positions, such as e-sensors worn on a headband, such a database is easy to establish - e.g. the recording of muscle activity from the eye occurs in a reasonably well defined physical space and can be approximately modeled, allowing for some differences in the physiology of the individual. Also, when a specific signal such as a heartbeat is detected, the structure of the acquired signals is basically known and thus the information content can be reliably extracted even in the presence of background noise and uncertainties in the spatial location and transmission efficiency of the source of e-sensor information. Examples of such machines envisioned to be included in such applications are computers, keyboards, printers, television sets, mobile phones, displays, digital recording devices and players, gameboxes, automobiles, home appliances, and the like.

The use of e-sensors to expand the applicability of human-machine interface to other fields is enabled in accordance with the present invention by incorporating bi-directional audio communications between the engaged individual and the machine engaged, the type of feedback being based on the purpose of the interface and on a period of interactive training involving audio and video feedback from the machine to the individual and motional and audio feed back from the individual to the machine.

The actions taken by the machine in addition to the audio response are selected from a predefined database which associates particular responses of the electric field sensor with particular types of machine acknowledgments. This database is updateable and may reside locally on the machine or on a remote server.

The database is built by the machine from the interactive training between the individual and the machine. The acknowledgement of feedback from the individual may comprise deliberate physical motions and/or audio communication. The acknowledgement could also comprise deliberate physical motions of the individual and an object held by the individual.

The machine, in addition to its audio response, may also communicate with additional connected machines. This may be, for example, the transfer of data files such as image data from a camera to a printer. These are examples only and should not be understood as limiting the scope of the invention. BeIo w are described various examples and applications of a system in accordance with the invention. It should be understood that these are examples only and in no way should be read as limiting the scope of the invention.

Example 1

An integrated sensor system is incorporated along the inside rim of a baseball cap. A number of sensors and sensor arrays are then placed at strategic locations around the inside circumference of the cap. These sensors are grouped to collect, in real time, EEG, EOG and EMG signals, in a distinct and differentiated manner with minimal or no cross talk. The circuit also consists of signal processing software, which will be able to interpret the pattern sequence of the signals into a set of logic commands.

More specifically, we can imagine a situation where a person is watching television and he wants to turn the TV off. He will then close his eyes for a short period of time. This period of time will be pre-assigned to be longer than, for example, approximately 5 seconds. He will then open his eyes. He will not repeat this process for at least a minute. The closing of the eyes results in the generation of a distinct alpha wave (EEG) signal. This signal will be analysed by the signal processor and if no other alpha wave is generated in less than a minute, a command will be transmitted by the wireless transmitter box in the baseball cap to turn the TV off. The TV receiver will turn the TV off.

Example 2

We can envision a scenario where the sensor system is built into the sides of a person's baseball cap and he is driving a car. The wireless transmitter in the cap is capable of sending signals to, for example, the car's music system, his mobile phone, his seat adjustment controller, the car's external lighting switches, the windshield wipers switch and the sunroof latch mechanism. These are examples only and should not be read as limiting the scope of the invention in any way. The top section of the car's dashboard has a white horizontal line drawn across the entire length of the dashboard. There are six brightly coloured, dummy icons of the electrical systems which the driver intends to operate while driving the car, painted on the white lines spaced evenly from each other, spread as far away as possible from each other. The ends of the white line are marked by black circular dots.

The driver wears his sensor-laden cap, enters the car and starts the engine. He creates an electronic reference of his eyesight direction and location by a standard calibration procedure. He would, by this process, create in the car's computer system, virtual, x-y spatial panels, in space, in front of the driver and each panel would be electronically assigned to the various electrical control systems that the driver intends to operate while driving the car.

While driving the car, the driver can gaze in the region of a specific virtual panel and trigger the electronic event by performing a pre-selected act such as blinking rapidly, or a set of blinks followed by an up or down or sideways eye ball movement. For example, he can activate the windshield wipers, the music system or open and close the sunroof.

Example 3 Recently new products, such as electronic picture frames have been introduced into the consumer electronics field, such as the Kodak Digital Photo Frame. The technology allows for people to electronically download and display selected pictures. With the invention a collection of pictures including art and photographs can be pre-selected in a computer or a central server system such as the Kodak Easy Share Gallery. The picture frames would be embedded with the circuitry to receive bio-signals as described in this invention. The pictures, which are displayed in the frames, can then be changed by a person wearing a cap or a head band with built-in sensors, The collection of pre-selected pictures can be scrolled through by alpha wave generation, EOG patterns and/or EMG signals.

Example 4 A person wearing a cap or a headband with the built-in sensors described in this invention, here after referred to as the electronic cap or the electronic headband, can turn on his iPOD, scroll and select a displayed music from his iPOD by using commands using bis bio-signals such as blinking at a particular frequency or moving his eyeballs up and down. He can use similar bio-signals to cause the selected music to be transmitted in a wireless fashion to another person's speaker headset to be heard instantaneously or stored in that person's iPOD and played at a later time. If it is sent for storage, a phone message can be simultaneously created to be sent to that person's mobile phone, for example by pre-coding the transmitter to ring that number. The same activity can be carried out such that the music selection can be sent to a group of friends along with phone messages as well.

Example 5 This example describes a mobile phone, which has the sensor and/or sensor arrays built along the edges of the phone. Every time the owner of the phone picks it up he identifies himself by pressing a pre-coded button. Every time the owner of the phone grabs the phone he makes contact with the sensors and they will record his electrocardiograph (ECG). The mobile phone then transmits the information to a central server and stores the information in pre-assigned lock box, which is protected. Authorized personnel can access the lock-box with a special security token. The information in the lock-box is then compared with a standard, reference ECG of the owner of the mobile phone. If the comparison is a proper match within specified tolerances, then a green coloured heart icon with an "all o.k." sign splashed on it can be activated in the mobile phone. If the comparison indicates a deviation from the accepted tolerances, then an amber coloured heart icon with "?"splashed across it, can be displayed on the mobile phone. Information alerts of a similar nature may also sent to the owner's health care practitioner. If the comparison of the ECG signatures reveals a potentially life threatening abnormality, then alerts are sent to the emergency services in the local area where the mobile phone user is located at that moment. Arrays of the sensor may be located on the steering wheel of an automobile. The electrocardiograph of the person grabbing the steering wheel is recorded similar to as described above. Icons displaying whether or not the electrocardiograph is within accepted tolerances may be displayed on a screen found in the steering wheel. As described above alerts may be sent to a local emergency service should abnormalities be detected.

Arrays of sensors may also be located on for example, a fridge door. Electrocardiograph readings can then be taken every time a user opens the fridge. It will be understood that these are examples only of devices which may be provided with sensors. The invention is not limited to such examples.

Example 6

The sensors could be used as a means of gathering further information for a user. For example, a mobile device such as in example 5 could be used to sense aspects of a persons environment such as the people present when a photograph is taken, or to make inferences as to a persons physical/emotional state from heart signals. This information can act as input to a learning system about the individual. This in turn could form part of a personal augmented memory. This augmented memory would always be accessible to its owner, for example in making context-related reminders when browsing a photo collection. The sensors could also initiate output from the augmented memory, for example when sensing the presence of a person you haven't seen for a while relevant information such as a photograph could be recalled.

Example 7

This example relates to gesture based displays. Similar to the description in example 3, this example relates to electronic picture frames. In the present invention, a collection of pictures including art and photographs are pre-selected in a computer or a central server system such as the Kodak Easy Share Gallery. The picture frames would be embedded with high impedance sensors or a sensor array. A suitable sensor is disclosed in, for example, US 7078911 or WO 03/048789. The sensor system is capable of mapping the electric field in three dimensions around the picture frame. A person waving his hand or a hand holding a dielectric object changes the ambient electric field in the vicinity of the picture frame and this disturbance is electronically interpreted and imaged by the sensor circuit to be either an up or down wave or left or right wave. If it is an up wave, then the picture which is next in the queue will be displayed. If it is down wave, then the previously viewed picture will be re-displayed. If it is a right wave, then the next group of selected pictures will be addressed and the first in the queue will be displayed. If it is a left wave, then the previously selected group of pictures will be selected and the first picture in the group will be displayed.

Example 8

The present invention can be used to obtain the ECG of, for example, a horse, cow or a ranch animal. The ranch may be enclosed with one gate for entry and one gate for exit. An e-field sensor is mounted at several locations on the gate to measure bio signals from the animals when they pass through the gate. This procedure can be used to track and count the number of animals passing through the gate.

It will be understood that the descriptions above are examples to illustrate the invention only and that many more applications fall within the scope of the claims.

Claims

CLAIMS:

1. A system comprising a device with at least one passive electrodynamic sensor capable of sensing existing bio and ambient electric fields and changes in these fields in the vicinity of the sensors induced by living forms and means for transmitting the resulting signals to other systems to initiate some activity in the other system.

2. A system as claimed in claim 1 wherein the activity initiated is feedback.

3. A system as claimed in claim 1 or 2 wherein the sensor senses the generation and dissipation of alpha waves of the brain induced by closing and opening of the eyes.

4. A system as claimed in claim 1 or 2 wherein the sensor senses electro- oculargraphic signals.

5. A system as claimed in claim 1 or 2 wherein the sensor senses electro myographic signals.

6. A system as claimed in claim 1 or 2 wherein the sensor senses and tracks electromyography signals generated by the controlled movement of eyes from one referenced spatial position on a display screen to another, in a horizontal and vertical fashion.

7. A device fitted with sensors that detect the ECG signals of a user and including means for comparing said ECG signals with those of a registered user of the device.

8. A device as claimed in claim 7 further including means for transmitting this information to a central information storage system.

9. A system including the device of claim 7 and means for comparing the ECG information transmitted from the device with a standard ECG to detect for any abnormalities.

10. A system as claimed in claim 8 further including means for transmitting a result of the comparison to the device and/or for transmitting the results of the comparison to a health care practitioner and/or hospital related to the user of the device.

11. A device as claimed in claim 7, the device being a mobile telecommunication device.

12. A device as claimed in claim 7, the device being an automobile steering wheel.

13. A device as claimed in claim 7, the device being a domestic appliance.

14. A system containing an array of passive electric potential sensors integrated into the back plane of a display screen, the sensors being able to detect and track perturbations to the ambient electromagnetic field by gestures and/or movement of a sharp dielectric object in front of the display screen.

15. A method of switching an electromagnetic system on or off, changing the volume of an audio system, changing the channels of a television system, changing the tracks on a disc or making telephone calls, using the system as claimed in claim 1.

16. A system for activating electrical systems within an automobile, a dashboard of the automobile being divided into as many sections as there are systems to activate, an electronic reference being created of a driver's eyesight direction by means of sensors located within a cap worn by the driver when the user starts the engine, each of the sections of the dashboard being assigned to one electrical system, a selected system being activated when the user gazes at the appropriate section and triggers the system by performing a preselected act.

17. A method of selecting and transmitting text and/or image from a display screen of an electro optical system to another electro optical system, using the system as claimed in claim 1.

18. A method of enabling a human-machine communicative relationship based on sensing the motion of an individual in an electric field by at least one electric field sensor associated with the machine, the method including the steps of; establishing a sound enabled communicative relationship between the individual and the machine, the relationship comprising audio feedback from the machine responsive to physical motion of the individual and acknowledgement of the feedback by the individual; and exercising one or more defined motions by the individual which are uniquely acknowledged by the machine by actions taken by the machine in addition to the audio response.

19. A method as claimed in claim 18 in which the actions taken by the machine in addition to the audio response are selected from a pre-defined database which associates particular responses of the electric field sensor with particular types of machine acknowledgements.

20. A method as claimed in claim 19 in which the pre-defined database is updateable and resides locally on the machine.

21. A method as claimed in claim 19 in which the pre-defined database is updateable and resides on a remote server.

22. A method as claimed in claim 19 in which the pre-defined database is built by the machine from interactive training between the individual and the machine based on audio and video feedback by the machine responsive to the motions of the individual.

23. A method as claimed in claim 18 in which the acknowledgment of feedback by the individual comprises deliberate physical motions of the individual.

24. A method as claimed in claim 23 in which the acknowledgment of feedback by the individual additionally comprises audio communication from the individual.

25. A method as claimed in claim 18 in which the acknowledgment of feedback by the individual comprises deliberate physical motions of the individual and an object held by the individual.

26. A method as claimed in claim 18 in which actions taken by the machine in addition to its audio response include video display.

27. A method as claimed in claim 18 in which actions taken by the machine in addition to its audio response include actions communicated to additional connected machines.

28. A method as claimed in claim 27, the actions including transfer of data files to the connected machines.

29. A method as claimed in claim 28, the actions including transferring images from a camera to a printer.

30. A method as claimed in claim 22 wherein the interactive training between the individual and the machine includes audio and visual displays, the visual display being in correspondence with the actions taken by the machine in addition to its audio response.