WO2002001876A1

WO2002001876A1 - Processing image data

Info

Publication number: WO2002001876A1
Application number: PCT/GB2001/002801
Authority: WO
Inventors: Ben John Dixon Whitaker; Timothy Charles Horsfall; Jenny Adams; James Hay
Original assignee: Inview Systems Limited
Priority date: 2000-06-26
Filing date: 2001-06-25
Publication date: 2002-01-03
Also published as: AU7433101A; GB0015524D0; GB2364192A

Abstract

An image processing system is displayed which a first video camera (101) produces a first video signal from a first image view. A second video camera (103) generates a second video signal derived from a second image view. The first video signal is combined with a second video signal to produce a combined video signal in which the first image is displaced from the second image in a vertical direction. A micro-display (105) generates a combined image from the combined video signal. Image manipulating optics (109) receive light from the display device and arrange for the combined image to be displayed with a first image alongside the second image in a horizontal direction.

Description

Processing Image Data

Background of the Invention

1. Field of the Invention

The present invention relates to a method of processing image data.

2. Description of the Related Art

A synthesised virtual image electronic display having a micro-display is disclosed in United States Patent number 5,973,845. The micro-display is arranged to form a source object, that preferably has a surface area of less than about one hundred square millimetres. A first stage magnification optic magnifies a source object to produce a magnified image.

Micro-displays are high technology elements that are available at competitive prices due to processes of mass production. The manufacturers have particular target markets in mind and as such the definition of the arrays and the aspect ratio of these arrays tend to be consistent with known applications. Consequently, arrays of this type are not available for unusual aspect ratios, including situations where a very wide angle is required. Under these circumstances, systems must be built with redundancy in order to obtain wide angle view, that will in turn result in these devices being unavailable for many commercial applications.

Brief Summary of the Invention According to a first aspect of the present invention, there is provided image data processing apparatus, comprising a first video signal generating means configured to derive a first video signal from a first image view; a second video signal generating means configured to generate a second video signal derived from a second image view; combining means for combining said first video signal with said second video signal to produce a combined video signal in which said first image is displaced from said second image in a first direction; a display device for generating a combined image from said combined video signal; and image manipulating optics configured to receive light from said display device so as to display said image alongside said second image in a second direction, different from said first direction.

In a preferred embodiment, the display device has a pixel size of from ten micrometers to fifty micrometers and an output from said display is enlarged by means of optics. Devices of this type are generally referred to as "micro-displays". In a preferred embodiment, the image manipulating optics is configured to display the first image alongside said second image in a horizontal direction. In a particular application, three images are displayed alongside each other derived from respective cameras mounted to a motor vehicle. In this way, it is possible to generate a combined image for the purposes of viewing traffic to the rear of the vehicle and, in this configuration, the device provides a replacement to a conventional rear view mirror which, when three images are combined, allows side views to be presented to a driver without the need for wing mirrors extending from the sides of the vehicle.

Brief Description of the Several Views of the Drawings

Figure 1 shows an image data processing system in which a first video camera generates a first video signal;

Figure 2 illustrates an expansion of the technique illustrated in Figure 1, in which three images are combined vertically and then separated to be displayed horizontally;

Figure 3 illustrates an example of the application of the present invention, to replace conventional rear view mirrors on a motor vehicle;

Figure 4 illustrates examples of images produced by the cameras shown in Figure 3;

Figure 5 shows an example of a view presented to a driver using the system illustrated in Figure 3;

Figure 6 provides a schematic representation of the system illustrated in Figure 3; Figure 7 details an image processing subsystem identified in Figure 6;

Figure 8 details a display device of the type shown in Figure 3;

Figure 9 details optics contained within the display device of Figure 8; and

Figure 10 details a portion of an alternative embodiment to the optics illustrated in Figure 9;

Best Mode for Carrying Out the Invention

An image data processing system is shown in Figure 1 in which a first video camera 101 generates a first video signal on line 102 with a similar second video camera 103 generating a second video signal on line

104.

A processing system 105 combines the first video signal on line 102 with the second video signal on line 104, this is configured to produce a composite video signal in which a first video image 106 is displaced from a second video image 107 in a first direction which, in this example, is the vertical direction. Each video image frame may typically consist of eight hundred pixels in the horizontal dimension with six hundred pixels in the vertical direction. In this particular example, the image frame has been divided into two images, such that image 106 has eight hundred horizontal pixels with three hundred vertical pixels, while image 107 has a similar number of pixels in its horizontal and vertical dimensions. A combined video output signal is generated on a line 108 and supplied to a display device 109 having image manipulating optics. The display device generates a combined image from the composite video signal. The manipulating optics 109 are configured to receive light from the display device so as to display said first image alongside the second image. However, when displayed, the first image is displayed alongside the second image but in a second direction, different from the first direction of the combined image. Thus, as shown in Figure 1, image 106 has been manipulated as image 206 with image 107 being manipulated as image 207. When the totality is viewed, consisting of images 207 and 206, they are provided with a horizontal definition of one thousand, six hundred pixels and a vertical definition of three hundred pixels. Thus, cameras 101 and 103 are positioned side by side in order to achieve an appropriate panoramic effect when viewed in combination. However, this wide panoramic effect has been achieved using substantially conventional video processing equipment, having a conventional display aspect ratio which, in this example, substantially follows the super video graphics array (SVGA) standard.

The technique illustrated in Figure 1 may be extended further in which further enhancements are made to horizontal definition by reducing the available vertical definition. Thus, as an alternative to dividing the image into two slices, as shown in Figure 1, it is possible to divide the image into three slices, as shown in Figure 2. A first video camera 201 and a second video camera 202 have, in addition, a third video camera 203. Each video camera produces a video slice having eight hundred pixels in the horizontal dimension with only two hundred pixels in the vertical dimension. These horizontal slices are then combined vertically to produce image frames of eight hundred pixels by six hundred pixels consisting of three vertically displaced slices 204, 205 and 206 in a combining processing system 207. These image frames are generated in a substantially similar fashion to that illustrated with respect to Figure 1 but then divided optically to produce three horizontally displaced output slices 214, 215 and 216. The ability to use manipulating optics in order to reconfigure the topology of a display facilitates many applications where an image shape is required that differs significantly from that of conventionally available image generating devices. The technique is particularly adaptable to the exploitation of micro-displays, of the type described in United States Patent number 5,973,845. A micro-display is a high definition device having pixel elements too small to be viewed directly by the human eye. Typically, a micro-display has a pixel size of from ten micrometers to fifty micrometers and is made viewable by means of enlarging optics. The present invention takes this method of display a stage further by employing optics that manipulate the image in addition to merely enlarging it. Thus, in this way, image portions displayed in a particular first orientation may be manipulated in order to be displayed in an alternative orientation.

An example of an application for the present invention is illustrated in Figure 3. In this particular application of the invention, the system has been used to replace a conventional rear view mirror of a motor vehicle. A projected image from the micro-display is presented to a driver at a position substantially similar to the position of a convention rear view mirror. In addition, video cameras produce images derived from the sides of the vehicle at positions commonly associated with wing mirrors. However, these miniaturised video cameras are significantly smaller than wing mirrors and as such do not extend outwards from the vehicle, thereby reducing drag and minimising the overall width of the vehicle. The micro-display system is then used to combine the side images with the central image derived from a central video camera, thereby presenting to the driver, at the position of a conventional rear view mirror, an overall view of a landscape to the rear of the vehicle. A vehicle 301 has a centrally positioned video camera 302, a right- hand video camera 303 and left-hand video camera 304. Video camera 302 views a field of view identified by section 305. Similarly, camera 303 views a section 306 and camera 304 views a section 307. With careful alignment of the cameras and associated optics, it is possible that, within a relatively narrow horizontal band, it is possible for fields 306, 305 and 307 to abut exclusively along lines 308 and 309. Furthermore, if perfect abutment of the type illustrated in Figure 3 is not possible, it is possible for further electronic processing to remove areas of overlap to a significant extent such that a driver is presented with a substantially continuous view of a rearward landscape.

Examples of images produced by the cameras identified in Figure 3 are shown in Figure 4. Thus, in this example, an image 401 has been produced by camera 304, an image 402 has been produced by camera 302 and an image 403 has been produced by camera 303. These images are then processed to eliminate regions of overlap and thereafter produce a combined image 404, illustrating a panorama to the rear of the vehicle. This panorama is significantly wider than that available through a conventional rear view mirror and effectively combines images that would be presented by a rear view mirror in combination with wing mirrors or similar. However, the overall system is significantly superior to a three mirror system of conventional design, in that the totality of the image is presented to the driver at a substantially common position, that being the position of a conventional rear view mirror. Thus, by viewing the rearward projection embodying the present invention, in preference to a conventional rear view mirror, a much wider panorama is presented to the driver, thereby significantly assisting a driver in terms of the availability of information, thereby enhancing the driving experience while at the same time improving safety.

An example of the view presented to a driver is illustrated in Figure 5. Thus, the combined image 404 is presented to the driver at location 501, displayed by means of a conventional micro-display with manipulating optics. The manipulating optics present a virtual image to the driver that is perceived as being a long way in front of the driver (virtually at infinity) such that when looking ahead, it is not necessary for the driver to refocus in order to view image 501, as is known when using conventional mirrors. However, the image perceived by the driver is significantly wider than that presented by a conventional rear view mirror and does not include any obstructions due to components of the vehicle's structure. This is a particularly attractive application for sports vehicles where rear views tend to be restricted to the coupe configuration that such vehicles tend to adopt.

The system illustrated in Figure 3 is shown schematically in Figure 6. Video cameras 303, 302 and 304 perform an image capturing function and supply video signals to a processing sub-system 601. Output from processing sub-system 601 is then supplied to a display sub-system 602, presenting the images captured by image capture cameras 302 to 304 in a panoramic display by application by the present invention.

Cameras 302 to 304 are conventional analogue devices producing image frames consisting of five hundred and twelve horizontal lines and having a definition equivalent to six hundred pixel positions along each of said lines. The micro-display produces an output of eight hundred pixels by six hundred pixels therefore the horizontal definition is substantially similar to that of the cameras, ie having six hundred pixel locations. The vertical dimension of eight hundred pixels is divided by three therefore the vertical definition of the cameras, having five hundred and twelve lines, is larger than that required. Consequently, a portion of each image is rejected and, in the particular embodiment, the lower lines of the images are rejected and the system is aligned such that the area of interest is captured in approximately the upper two thirds of the image field. Processing sub-system 601 is preferably designed as a single integrated chip incorporating all necessary control electronics. Output signals from video cameras 302, 303 and 304 are supplied to processing sub-system 601 via cables 603, 604 and 605 respectively, consisting of base-band colour composite video signals having a luminance component and a chrominance component conveyed on a sub-carrier. Processing within processing sub-system 601 is performed substantially in the digital domain although an output from sub-system 601, over cable 606 is a substantially analogue signal supplied to display sub-system 602.

Processing sub-system 601 is detailed in Figure 7. Inputs on line 604, 603 and 605 are received by analogue to digital converters 701 , 702 and 703 respectively. Outputs from analogue to digital converters 701 to 703 are supplied to respective colour-space converters 704, 705 and 706, configured to produce individual red, green and blue words for each sample pixel.

Individual frame buffers receive samples for the red, green and blue components of each of the three outputs. These are defined logically as portions of Random Access Memory such that portion 711 stores red signals from converter 704, portion 712 stores green samples from converter 704 while portion 713 stores blue samples from converter 704. Similarly, red samples from converters 705 and 706 are stored at portions 714 and 717 respectively, with portion 715 receiving green samples from converter 705, portion 716 receiving blue samples from converter 705, portion 718 receiving green samples from converter 706 and portion 719 receiving blue samples from converter 706. Converters 704 to 706 supply data to the storage regions in parallel on a pixel-by-pixel basis, such that when an output is generated from converter 704, a contribution is made to portion 711, in the form of a red sample, a contribution is made to portion

712, in the form of a green sample and a contribution is made to portion 713 in the form of a blue sample.

The display sub-system incorporating a micro-display is a reflective device produced by Colorado Micro-Displays and typically enclose a miniature screen having a diagonal of typically twenty millimetres. It consists of reflective elements arranged to reflect light incident upon them. Consequently, the display device itself is not a colour device and a colour output is achieved by driving red, green and blue sources in a field sequential manner. Thus, rather than displaying separate red, green and blue components at a display rate of fifty frames per second, a red frame is displayed, followed by a green frame, followed by a blue frame at a frequency of one hundred and fifty frames per second. This is achieved using a field sequential converter 721. Thus, on each cycle, repeated at fifty cycles per second, field sequential converter 721 generates a red frame by reading data from portions 711, 714 and 717. This is then followed by the generation of a green frame by deriving data from portions 712, 715 and 718. Finally, a blue output frame is generated by receiving data from portions 713, 716 and 719. Output signals from converter 721 are supplied to a digital to analogue converter 722 that in turn provides an output signal over cable 606.

Display sub-system 602 takes information from the cameras 302 to 304 via processing sub-system 601 and presents it visually to the driver.

The processing sub-system 602 is detailed in Figure 8. A micro-display projector 801 projects an image towards an intermediate mirror 802. The projector 801 includes a micro-display and image manipulating optics. The intermediate mirror 802 allows the sub-system to be housed in the unit of acceptable size.

Consequently, light is reflected from the intermediate mirror 802 onto a paraboloidal mirror 803. The sub-system is also provided with a height adjustment control 804, allowing drivers of differing heights to adjust the device or allowing a driver after adjusting a seating position to adjust the device in a way substantially similar to the adjustment of conventional rear view mirrors.

Operation of the manipulating optics is illustrated with respect to Figure 9. In order to enhance clarity the operation of the intermediate mirror has been removed and a substantially straight line is shown from a micro- display 901 to a paraboloidal mirror 902 via manipulating optics 903. The manipulating optics 903 consists of three individual lenses 911 , 912 and 913. Lenses 911 to 913 receive light transmitted from sections 921, 922 and 923 respectively of the micro-display 901. Lenses 911 to 913 effectively take the form of prisms in addition to enlarging lenses and as such translate light from their respective sections of the micro-display to form a panoramic final image at display mirror 902. The parabolic nature of mirror 902 presents the viewer with substantially parallel rays so as to avoid the need to refocus when viewing images originating from the mirror. The mirror is operating off-axis, so as to ensure that the projecting source is not interrupted by the head of the viewer. The paraboloidal mirror allows differing vertical eye positions to see the full extent of the projecting display and highlights the requirement for a mirror of greater depth than the image displayed.

In alternative configuration, each prism lens 911 to 913 may be replaced by a combination of a prisms 1001, 1002, 1003 and a respective enlarging lens 1011 1012, 1013 as shown in Figure 10. Central lens 912 is substantially symmetrical but in order to achieve the required optical effect, end 931 of lens 911 has a narrower section than end 932. In lower lens 913, this position is reversed, such that end 933 is larger than its opposite end. This allows lens 911 to be rotated about axis 935 so as to direct light to portion 936 of mirror 902. Similarly, lens 913 is rotated about axis 935 in the opposite sense, so as to direct light emanating from portion 923 of display 901 towards portion 937 of mirror 902. A similar rotation is performed about a vertical axis and again the lenses are shaped in order to accommodate this, such that a lower edge of lens 932 is thinner than its upper edge. Similarly, lens 913 directs its image upwards effected by a horizontal rotation and compensation provided by its upper edge being thinner than its lower edge. Thus, when taken in combination, image portions from the micro-display 901, displaced in a vertical direction, are manipulated so as to present three image portions displaced in a horizontal direction. In this way, by embodying the present invention, it is possible for a conventional micro-display unit to be employed in an environment where advantage is taken from a wide panoramic display.

Claims

1. Image data processing apparatus, comprising a first video signal generating means configured to derive a first video signal from a first image view; a second video signal generating means configured to generate a second video signal derived from a second image view; combining means for combining said first video signal with said second video signal to produce a combined video signal in which said first image is displaced from said second image in a first direction; a display device for generating a combined image from said combined video signal; and image manipulating optics configured to receive light from said display device so as to display said first image alongside said second image in a second direction, different from said first direction.

2. Apparatus according to claim 1, wherein said first video signal generating means and said second video signal generating means are digital video cameras.

3. Data processing apparatus according to claim 1 or claim 2, wherein an additional third video signal generating means is provided.

4. Apparatus according to any of claims 1 to 3, wherein four or more video signal generating means are provided.

5. Apparatus according to any of claims 1 to 4, wherein parallel red, green, blue (RGB) signals are processed to produce field sequential colour signals.

6. Apparatus according to any of claims 1 to 5, wherein said first direction is the vertical direction, such that vertical definition is shared between said images.

7. Apparatus according to any of claims 1 to 6, wherein said display device has a pixel size of from ten micrometers to fifty micrometers and an output from said display is enlarged by means of optics.

8. Apparatus according to any of claims 1 to 7, wherein said image manipulating optics is configured to display said first image alongside said second image in a horizontal direction.

9. Apparatus according to any of claims 1 to 8, wherein said first video signal generating means and said second video signal generating means are mounted to a motor vehicle.

10. Apparatus according to claim 9, wherein a third video signal generating means is mounted to said vehicle and images generated from said respective video signal generating means are combined to provide a panoramic view of the rear of the vehicle to provide the functionality of a rear view mirror and two wing mirrors.

11. A method of processing image data, comprising the steps of receiving a first video signal derived from a first image of view; receiving a second video signal derived from a second image view; combining said first video signal with said second video signal to produce a combined video signal in which said first image is displayed from said second image in a first direction; generating a combined image from said combined video signal; directing light from an output display through optics such that said first image is displayed alongside said second image in a second direction, different from said first direction.

12. A method according to claim 11, wherein said first video signal and said second video signal are generated by video cameras.

13. A method according to claim 11 or claim 12, configured to receive a third video signal.

14. A method according to any of claims 1 to 13, configured to receive four or more video signals.

15. A method according to any of claims 11 to 14, wherein parallel red, green, blue (RGB) signals are processed to produce field sequential colour signals.

16. A method according to any of claims 11 to 15, wherein said first direction is the vertical direction, such that vertical definition is shared between said images.

17. A method according to any of claims 11 to 16, wherein said output display has a pixel size of from ten micrometers to fifty micrometers and an output from said display is enlarges by means of optics.

18. A method according to any of claims 11 to 17, wherein said image manipulating optics is configured to display said first image along said second image in a horizontal direction.

19. A method according to claim 18, wherein said image manipulating optics includes prisms and enlarging optics.

20. A method according to claim 19, wherein said prism and said enlarging optics are combined to form a single optic for each manipulated image.

21. A method according to any of claims 11 to 20, wherein said video signals are received from cameras mounted to a motor vehicle.

22. A method according to claim 21, wherein said images are combined to provide the functionality of a rear view imaging system.