WO2000042470A1 - Resolution invariant panoramic imaging - Google Patents

Resolution invariant panoramic imaging Download PDF

Info

Publication number
WO2000042470A1
WO2000042470A1 PCT/AU2000/000022 AU0000022W WO0042470A1 WO 2000042470 A1 WO2000042470 A1 WO 2000042470A1 AU 0000022 W AU0000022 W AU 0000022W WO 0042470 A1 WO0042470 A1 WO 0042470A1
Authority
WO
WIPO (PCT)
Prior art keywords
view
reflective surface
panoramic
mirror
imaging system
Prior art date
Application number
PCT/AU2000/000022
Other languages
French (fr)
Inventor
John Barratt Moore
Tanya Louise Conroy
Original Assignee
The Australian National University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Australian National University filed Critical The Australian National University
Priority to AU24249/00A priority Critical patent/AU757227B2/en
Priority to EP00902499A priority patent/EP1151349A1/en
Priority to JP2000593986A priority patent/JP2002535700A/en
Publication of WO2000042470A1 publication Critical patent/WO2000042470A1/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B13/00Optical objectives specially designed for the purposes specified below
    • G02B13/06Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B37/00Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe

Definitions

  • This invention relates to generating wide angle images of spaces, generally referred to as panoramic imaging.
  • Panoramic imaging is becoming an important tool in the area of mobile robotics and machine vision.
  • One simple method involves having a series of cameras mounted on a ring to give views around the entire 360° of horizon. This involves, say, four cameras if they each have a field of view of 90° and some integration of images.
  • There are also a number of single camera methods for panoramic imaging including rotating a camera about its vertical axis and taking pictures continuously to obtain a full panoramic view.
  • Another approach uses wide angle lenses to achieve a large field of view, but these lenses are heavy, expensive and distort the image.
  • An attractive approach to panoramic imaging is to mount a single fixed camera under a curved reflective surface covering a hemisphere such as with a conical, spherical, hyperboloidal, or other profile.
  • the optical axis of the camera is aligned with the central axis of the mirror.
  • a known family of constant gain reflective surfaces have the advantage that they can produce large fields of view such as for a hemispherical or hyperboloidal mirror yet preserve a linear relationship between changes in angles of incidence and reflection of light rays viewed by the camera. This linear relationship simplifies image processing and ensures constant elevational resolution of the image.
  • the shape of the surface is determined by the gain of the linear relationship.
  • the surface is a cone; for higher gains, the surface is specified by a family of polynomial functions.
  • the panoramic plane will be considered as being horizontal and the field of view as vertical as would be the case for a robot moving in a horizontal plane. It will be apparent that in the general case orientation of the planes is arbitrary. All the mirror shapes mentioned above share a common draw back. That is that the CCD cameras used for imaging invariably have uniform Cartesian arrays of pixels to capture the polar image of the scene, and so the pixel density per solid angle increases with the radius of the polar image.
  • the unwarping process transforms the image from polar to Cartesian coordinates so that the angular coordinate in the original polar image maps to the x-coordinate in the unwarped image while the radial coordinate maps to the y-coordinate.
  • the pixel density in the unwarped image varies from low for small x values which correspond to the centre of the original image to high for large x values which correspond to the outer rim of the polar image.
  • Figure 1 shows the unwarping of an image captured with a hyperboloidal mirror. The variation in image quality is clearly evident in the unwarped version.
  • this invention provides a panoramic imaging system including an imaging device having an image plane and a first field of view, a first reflective surface having at least one circularly symmetric portion convex in a radial direction disposed in said first field of view to provide an expanded panoramic second field of view, the profile of the or each convex portion providing a varying gain between the fields of view in the radial direction to limit variation in the solid angle of view across the image plane of the imaging device.
  • the profile of the convex portion provides a substantially uniform solid angle of view across the image plane. That is, the shape ensures that the resolution in the image is invariant to changes in elevation.
  • the shape of the reflective surfaces results in solid angle pixel density invariance.
  • r is the radial distance from the reflective surface to the imaging device
  • is the angle from the optical axis of the imaging device ( ⁇ ) is the mirror gain given by
  • ⁇ and ⁇ are the maximum and minimum elevations viewed ⁇ and ⁇ are the maximum and minimum radial angles imaged.
  • r can be plotted against ⁇ at selected intervals to describe the profile by solving the above equation for selected values of ⁇ . For example determining values of r for incremental values of ⁇ of about 1/5° has been found to produce a sufficiently accurate profile for practical application.
  • panoramic range finding There are a number of methods for panoramic range finding.
  • One method uses a cone mirror above a camera.
  • the camera mirror assembly is either displaced during image collection, or two camera mirror assemblies are used to obtain the two views necessary for range finding.
  • this method provides range information in the horizontal plane at video rates, its drawbacks are that no range information is available in the vertical (elevation) direction, objects must be more than a minimum distance from the camera and there may be a blind spot due to the second camera system.
  • a discontinuous, axially symmetric mirror which is in essence a coaxial mirror pair, mounted above a camera to obtain two views of a panoramic scene for stereo disparity range finding is known.
  • known constant gain mirror profiles have been generalised to derive a family of such coaxial mirror pair profiles for panoramic stereo imaging and processing based on disparities in the vertical plane.
  • this invention provides for range finding using a panoramic imaging system containing two resolution invariant mirrors.
  • the mirror or reflector surface has at least two of said convex portions arranged to respectively provide at least partially overlapping panoramic second fields of view for range determination.
  • the second fields of view are preferably substantially co-incident.
  • the two convex portions form a continuous mirror or reflective surface.
  • this invention provides a design for a back to back stereo mirror system with the desirable property of equal pixel sharing between two cameras and thus the two stereo images.
  • the stereo cone in this case is preferably symmetric in the directions orthogonal to the camera axis which is a desirable property for some applications.
  • the imaging system preferably includes two first reflective surfaces each having an associated image plane with corresponding first fields of view, and at least one convex portion of each first reflective surface providing respective panoramic second fields of view, said first reflective surface being arranged back to back such that said reflective second fields of view at least partially overlap.
  • a second reflective surface can, in some applications be interposed between the image plane and the second reflective surface. This allows positioning of the imaging device for example behind the first reflective surface. In some variations an apperture can be provided in the first reflective surface to provide the first field of view from the imaging device.
  • this invention provides a reflective surface for use in a panoramic imaging system including an imaging device having an imaging plane and a first field of view, said reflective surface having at least one circularly symmetric portion convex in a radial direction with a profile providing varying gain in the radial direction between an expanded panoramic second field of view provided by the reflective surface and the first field of view to limit variation in the solid angle of view across the image plane of the imaging device.
  • this invention provides mirrors having minimal intrusive designs, which intrude to a minimal extent into the viewing "hemisphere". These are also termed forward facing designs. They involve an additional planar mirror and camera relocation within the primary reflective surface. The attraction of this arrangement is that the first reflective mirror surface profile is the same design as in a more conventional arrangement.
  • Figure 1 illustrates an unwarping process for a prior art panoramic imaging system
  • Figure 2 schematically shows the relationship between camera image and horizontal view direction in a panoramic imaging system
  • Figure 3 illustrates geometric relationships between a reflecting surface and a camera used to derive mirror profiles according to this invention
  • Figures 4 A and 4B are graphs showing a comparison of a constant gain mirror with a variable gain mirror used in the imaging system according to this invention
  • Figures 5 A and 5B shows ray traced scenes respectively reflected in constant and variable gain mirrors
  • Figures 6A and 6B graphically illustrates a comparison of panoramic imaging systems respectively utilising double constant and variable gain mirror configurations
  • Figures 7 A and 7B shows raced traced images of scenes respectively corresponding to panoramic imaging systems utilising double constant and variable gain mirror configurations
  • Figure 8 schematically illustrates relationships between camera and reflective surfaces used in range calculation utilising a resolution invariant double mirror according to this invention
  • Figure 9 schematically illustrates a back to back mirror configuration according to this invention.
  • Figure 10 schematically illustrates a double back to back mirror configuration according to this invention
  • Figure 11 schematically illustrates a forward looking panoramic imaging system according to this invention.
  • Figure 12 shows a system utilising a combination of the arrangements in Figures 10 and 11.
  • resolution invariance is achieved by adjusting the mirror profile to image relatively less of the scene in the centre of the image and relatively more at the perimeter. That is, a mirror profile is selected to maintain a constant relationship between the pixel density and the angle of elevation in the scene or more precisely, the solid angle.
  • the mirror gain is the relationship between the change in elevation of rays incident on the mirror and the change in the angle of rays reflected into the camera as follows
  • Figure 3 schematically shows an imaging system including an imaging device in the form of a camera having an image plane and a first field of view.
  • a reflective surface or mirror is ... in the first field of view to provide an expanded panoramic second field of view.
  • the surface is circularly symmetric and convex in a radial direction.
  • is a function of image angle ⁇ (related to the radial coordinate in the image, p) so (4) becomes
  • the radius in the image, p is related to the radial angle of a ray reflected from the mirror, ⁇ by the focal length of the camera, / (a constant)
  • the image pixel density be invariant to angle of elevation in the scene which leads to more of the scene being imaged towards the perimeter, so
  • Figures 4A and 4B show for comparison a constant gain mirror and a variable gain mirror with the same camera field of view and range of elevations imaged.
  • the rays shown are constantly spaced in ⁇ , with about 2° between each ray. It is clear from Fig. 4A that in the constant gain case these rays are constantly spaced in ⁇ , with about 8.5° between each ray, and from Fig. 4, that the spacing between the rays in the variable gain case increases with increasing ⁇ . So, in the variable gain case, a greater proportion of the scene is imaged towards the outer edge of the polar image. This is also shown in Figures 5 A and 5B, ray traced images reflected in a constant gain and variable gain mirror with the same range of elevations visible.
  • a mirror with two convex portions or a double mirror is required.
  • the radial profile of a double mirror is shown in Figure 8.
  • the mirror arrangement for panoramic stereo with variable gain mirrors will necessarily be different than for constant gain mirrors due to the variation of the mirror gain, .
  • the gain must vary in a constant fashion over the entire double mirror so that the constant pixel density theorem will hold over the entire image. If the minimum and maximum elevations viewed ( ⁇ and ⁇ ) are to be equal for both mirrors in the double mirror system, the range of reflected angles (V - ⁇ ) cannot be equal for the two mirrors.
  • the minimum and maximum angles of reflected rays captured by the camera over the entire mirror surface are known from camera geometry. Therefore the minimum ray reflected from the lower mirror ( ⁇ ) and the maximum ray reflected from the upper mirror ( ⁇ 2 ) are known. So, since (12) holds over the entire mirror, B ⁇ is constant, and from (14)
  • tan 2 ( ⁇ ,) tan 2 ( ⁇ ,) tan 2 ( ⁇ 2 ) + tan 2 ( ⁇ ,)
  • Figures 6A, 6B and 7 A, 7B show graphical and ray traced comparisons of constant and variable gain double mirror systems viewing the same scene.
  • the information available for range calculation are the image angles for a single object reflected in both mirrors, ⁇ , and ⁇ 2 as shown in Figure 8.
  • the two mirrors ⁇ j and ⁇ 2 form a reflective surface.
  • the differential equations (6) for the surfaces are known. In the calculations that follow only the lower mirror is examined as the results are identical for the upper mirror.
  • m R] the gradient of the reflected beam from the lower mirror to the camera and dy,/dx, is the gradient of the lower mirror profile at the reflection point.
  • the gradient of the mirror profile for the lower variable gain mirror is found as in the constant gain case, from dy. dy dx. dx ⁇ d ⁇ «_ ,
  • variable gain mirror profile i ⁇ + ⁇ + tan2 ( ⁇ )
  • Equation constants for the incident beam equations from (18) require the polar coordinates of the reflection points from each mirror, (r lt ⁇ ,) and (r 2 , ⁇ 2 ) . Since the variable gain mirror equations are not known exactly, r ; and r 2 must be found using a differential equation solver to find solutions to (26) at ⁇ ⁇ and ⁇ 2 .
  • a key disadvantage of single camera stereo panoramic systems is that since there are two images of the "same" scene, the pixels assigned to each image is half that for non stereo panoramic imaging and the two images do not share an equal number of pixels in constant gain schemes.
  • the panoramic stereo double mirror method typically causes the view of a scene in one radial direction to be compressed into around 1/4 the field of view of the camera.
  • a method to achieve panoramic stereo with less image compression is to use two cameras and two single curved mirror surfaces back to back, as shown in Fig. 9.
  • This method compresses the imaged scene into 1/2 the field of view of the camera, and indeed each image has an equal share of the total number of pixels available.
  • Fig. 9 An advantage of the scheme proposed in Fig. 9 is that the stereo cone can be symmetric about the horizon using two cameras with equal fields of view and the maximum and minimum angles of elevation reflected by the two mirrors being equal.
  • the angle covered by the stereo cone in this case is 2 ⁇ - ⁇ .
  • Fig.9 shows the general case where the maximum and minimum angles of elevations viewed by each camera need not be equal. The range of elevations must still be equal for the fields of view to be aligned.
  • the mirror families can be either constant gain or resolution invariant.
  • Fig. 10 shows a back to back design incorporating double mirrors.
  • the double mirror can also have a variable gain.
  • the advantage to this system is that the stereo cone from the back to back configuration combines with the stereo cones from the double mirror configuration to increase the total area imaged in stereo. In this configuration, the fields of view of each double mirror pair need not be aligned as in previous examples.
  • FIG. 11 An example of a forward looking mirror design is shown in Fig. 11.
  • a panoramic camera looking out from, say, a hemisphere, somewhat as an eye of a bird, or perhaps two such on either side of a "nose cone".
  • This configuration is termed forward looking because the camera faces towards the scene.
  • Either a constant or variable gain mirror (double or single) could be used for the curved mirror in the system.
  • the planar mirror is an annulus or circle interposed such that all rays reflected from the curved mirror are reflected into camera o positioned behind the curved mirror.
  • the dotted lines in Fig. 11 show where the reflected rays would converge if the planar mirror was removed and the dotted camera shows the camera o ' for an equivalent system without the planar mirror.
  • the planar mirror In order for the rays reflected by the planar mirror to converge at the new camera position, the planar mirror must be the perpendicular bisector of the line joining the old and new camera locations. Hence the distance between the camera locations is 2D where D is defined in Fig. 11 as the distance from either camera to the planar mirror.
  • the introduction of the planar mirror into the system does increase the possibility of alignment difficulties as the planar mirror must be perpendicular to the camera axis and also be positioned so as to reflect all rays from the curved mirror into the camera without occluding the view of the curved mirror.
  • the maximum value for D,D is when the maximum beam reflected from the mirror system (the D beam reflected at point b on the planar mirror) into camera o grazes the curved profile at c. In this case
  • D defines the minimum height for the mirror system, H .
  • the value for D needs to be slightly smaller to avoid occlusion, leading to a larger mirror system height.
  • the general equation for the height of the mirror system is
  • must be greater than zero for camera o to be located behind the curved mirror. Also, ⁇ _ ⁇ if the minimum elevation ray ⁇ is not to be occluded by the planar mirror.
  • Fig. 12 shows a design that incorporates the ideas of Sections 2 and 3. It consists of two forward looking systems back to back, giving a design reminiscent of a eye mounted on a stalk, such as a crab's eye.
  • the "stalk" for this system would be hidden from view by the lower planar mirror.
  • portions are provided in the curved mirror to provide for reflection of rays from the curved surface to the camera by the plane mirrors.

Abstract

A panoramic imaging system includes an imaging device having an image plane and a first field of view, a first reflective surface having at least one circularly symmetric portion convex in a radial direction disposed in the first field of view to provide an expanded panoramic second field of view. The profile of the or each convex portion provides a varying gain between the fields of view in the radial direction to limit variation in the solid angle of view across the image plane of the imaging device.

Description

RESOLUTION INVARIANT PANORAMIC IMAGING
FIELD OF THE INVENTION
This invention relates to generating wide angle images of spaces, generally referred to as panoramic imaging.
BACKGROUND ART
Panoramic imaging is becoming an important tool in the area of mobile robotics and machine vision. There are many documented methods for recording a panoramic view of a scene. One simple method involves having a series of cameras mounted on a ring to give views around the entire 360° of horizon. This involves, say, four cameras if they each have a field of view of 90° and some integration of images. There are also a number of single camera methods for panoramic imaging, including rotating a camera about its vertical axis and taking pictures continuously to obtain a full panoramic view. Another approach uses wide angle lenses to achieve a large field of view, but these lenses are heavy, expensive and distort the image.
An attractive approach to panoramic imaging is to mount a single fixed camera under a curved reflective surface covering a hemisphere such as with a conical, spherical, hyperboloidal, or other profile. The optical axis of the camera is aligned with the central axis of the mirror. A known family of constant gain reflective surfaces have the advantage that they can produce large fields of view such as for a hemispherical or hyperboloidal mirror yet preserve a linear relationship between changes in angles of incidence and reflection of light rays viewed by the camera. This linear relationship simplifies image processing and ensures constant elevational resolution of the image. The shape of the surface is determined by the gain of the linear relationship. For a unity gain, the surface is a cone; for higher gains, the surface is specified by a family of polynomial functions. For ease of explanation in this specification the panoramic plane will be considered as being horizontal and the field of view as vertical as would be the case for a robot moving in a horizontal plane. It will be apparent that in the general case orientation of the planes is arbitrary. All the mirror shapes mentioned above share a common draw back. That is that the CCD cameras used for imaging invariably have uniform Cartesian arrays of pixels to capture the polar image of the scene, and so the pixel density per solid angle increases with the radius of the polar image. The unwarping process transforms the image from polar to Cartesian coordinates so that the angular coordinate in the original polar image maps to the x-coordinate in the unwarped image while the radial coordinate maps to the y-coordinate. Thus the pixel density in the unwarped image varies from low for small x values which correspond to the centre of the original image to high for large x values which correspond to the outer rim of the polar image. This is illustrated in Figure 1 which shows the unwarping of an image captured with a hyperboloidal mirror. The variation in image quality is clearly evident in the unwarped version.
One way to circumvent this problem is to use a specially designed CCD camera with a polar array of pixels with a pixel density which decreases with radius. There are alignment problems with such an approach.
DISCLOSURE OF THE INVENTION
In a first aspect this invention provides a panoramic imaging system including an imaging device having an image plane and a first field of view, a first reflective surface having at least one circularly symmetric portion convex in a radial direction disposed in said first field of view to provide an expanded panoramic second field of view, the profile of the or each convex portion providing a varying gain between the fields of view in the radial direction to limit variation in the solid angle of view across the image plane of the imaging device.
Preferably, the profile of the convex portion provides a substantially uniform solid angle of view across the image plane. That is, the shape ensures that the resolution in the image is invariant to changes in elevation. Thus, where the imaging system involves a device with an array of uniformly spaced pixels in the image plane, the shape of the reflective surfaces results in solid angle pixel density invariance. The profile of the reflective surface in polar coordinates is preferably determined by solving the equation dr = r cot V* dQ J" (1 + a(Q)) dβ
where r is the radial distance from the reflective surface to the imaging device θ is the angle from the optical axis of the imaging device (θ) is the mirror gain given by
α (θ) = Bα [tan (θ) + tan3 (θ)]
B = 2(φ - 40 tan2(θ) - tan2(θ)
φ and φ are the maximum and minimum elevations viewed ϋ and θ are the maximum and minimum radial angles imaged.
In one approach r can be plotted against θ at selected intervals to describe the profile by solving the above equation for selected values of θ. For example determining values of r for incremental values of θ of about 1/5° has been found to produce a sufficiently accurate profile for practical application.
There are a number of methods for panoramic range finding. One method uses a cone mirror above a camera. The camera mirror assembly is either displaced during image collection, or two camera mirror assemblies are used to obtain the two views necessary for range finding. Although this method provides range information in the horizontal plane at video rates, its drawbacks are that no range information is available in the vertical (elevation) direction, objects must be more than a minimum distance from the camera and there may be a blind spot due to the second camera system.
A discontinuous, axially symmetric mirror, which is in essence a coaxial mirror pair, mounted above a camera to obtain two views of a panoramic scene for stereo disparity range finding is known. There are however no proposals concerning specific mirror shapes to achieve specific desirable properties. Additionally, known constant gain mirror profiles have been generalised to derive a family of such coaxial mirror pair profiles for panoramic stereo imaging and processing based on disparities in the vertical plane.
In another aspect this invention provides for range finding using a panoramic imaging system containing two resolution invariant mirrors. Preferably the mirror or reflector surface has at least two of said convex portions arranged to respectively provide at least partially overlapping panoramic second fields of view for range determination. The second fields of view are preferably substantially co-incident. In the preferred form of the invention the two convex portions form a continuous mirror or reflective surface.
In a further aspect this invention provides a design for a back to back stereo mirror system with the desirable property of equal pixel sharing between two cameras and thus the two stereo images. The stereo cone in this case is preferably symmetric in the directions orthogonal to the camera axis which is a desirable property for some applications. In this aspect of the invention the imaging system preferably includes two first reflective surfaces each having an associated image plane with corresponding first fields of view, and at least one convex portion of each first reflective surface providing respective panoramic second fields of view, said first reflective surface being arranged back to back such that said reflective second fields of view at least partially overlap.
A second reflective surface can, in some applications be interposed between the image plane and the second reflective surface. This allows positioning of the imaging device for example behind the first reflective surface. In some variations an apperture can be provided in the first reflective surface to provide the first field of view from the imaging device. In another aspect this invention provides a reflective surface for use in a panoramic imaging system including an imaging device having an imaging plane and a first field of view, said reflective surface having at least one circularly symmetric portion convex in a radial direction with a profile providing varying gain in the radial direction between an expanded panoramic second field of view provided by the reflective surface and the first field of view to limit variation in the solid angle of view across the image plane of the imaging device.
In yet a further aspect this invention provides mirrors having minimal intrusive designs, which intrude to a minimal extent into the viewing "hemisphere". These are also termed forward facing designs. They involve an additional planar mirror and camera relocation within the primary reflective surface. The attraction of this arrangement is that the first reflective mirror surface profile is the same design as in a more conventional arrangement.
The invention will be further described, by way of example only with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 illustrates an unwarping process for a prior art panoramic imaging system; Figure 2 schematically shows the relationship between camera image and horizontal view direction in a panoramic imaging system;
Figure 3 illustrates geometric relationships between a reflecting surface and a camera used to derive mirror profiles according to this invention;
Figures 4 A and 4B, are graphs showing a comparison of a constant gain mirror with a variable gain mirror used in the imaging system according to this invention;
Figures 5 A and 5B, shows ray traced scenes respectively reflected in constant and variable gain mirrors;
Figures 6A and 6B, graphically illustrates a comparison of panoramic imaging systems respectively utilising double constant and variable gain mirror configurations; Figures 7 A and 7B, shows raced traced images of scenes respectively corresponding to panoramic imaging systems utilising double constant and variable gain mirror configurations;
Figure 8 schematically illustrates relationships between camera and reflective surfaces used in range calculation utilising a resolution invariant double mirror according to this invention;
Figure 9 schematically illustrates a back to back mirror configuration according to this invention;
Figure 10 schematically illustrates a double back to back mirror configuration according to this invention; Figure 11 schematically illustrates a forward looking panoramic imaging system according to this invention; and
Figure 12 shows a system utilising a combination of the arrangements in Figures 10 and 11.
BEST MODE FOR CARRYING OUT THE INVENTION
The various aspects of this invention will, for clarity, be described under separate subheadings.
1 Resolution Invariant Mirror Families This section describes a family of mirror designs that achieve the objective of resolution invariance, or equivalently solid angle pixel density invariance.
1.1 Constant Image Pixel Density - The Variable Gain (α) Mirror
In accordance with one aspect of this invention resolution invariance is achieved by adjusting the mirror profile to image relatively less of the scene in the centre of the image and relatively more at the perimeter. That is, a mirror profile is selected to maintain a constant relationship between the pixel density and the angle of elevation in the scene or more precisely, the solid angle. The mirror gain , is the relationship between the change in elevation of rays incident on the mirror and the change in the angle of rays reflected into the camera as follows
α δφ δθ (1)
where δφ is the change in vertical elevation and δθ is the change in angle of reflected rays received by the camera. With resolution invariance becomes a function of image angle θ which is related to the radial coordinate in the image, p, as shown in Figure 2.
Figure 3 schematically shows an imaging system including an imaging device in the form of a camera having an image plane and a first field of view. A reflective surface or mirror is ... in the first field of view to provide an expanded panoramic second field of view. The surface is circularly symmetric and convex in a radial direction.
Consider a mirror profile (r, θ) in polar coordinates where r is the radial distance to the camera and θ is the angle from the optical axis of the camera to the point on the mirror surface as shown in Fig. 3. The angle of incidence of a light ray relative to the mirror is γ and the angle of an incoming light ray with respect to the vertical is φ. Then
Figure imgf000010_0001
subject to the geometric constraint (from the law of reflection)
2γ + θ + φ = π (3)
Differentiating (2) and (3) with respect to θ
dy _d_ ( rd
Figure imgf000010_0002
Figure imgf000010_0003
so, substituting α from (1) gives
Figure imgf000010_0004
Now, for a variable gain mirror, α is a function of image angle θ (related to the radial coordinate in the image, p) so (4) becomes
Figure imgf000010_0005
or, rearranging -J = r cot[-'/2/(A α(θ))</θ] du (6)
The equation for the mirror gain, (θ) to achieve pixel density invariance can be found using the following theory.
1.1.1 Pixel Density Invariance Profiles
There aτ p(p) pixels in an area of radius p in the image. More formally, there are
/Xp) = πκp2
pixels in an area of radius p , where K is the number of pixels per unit area, a constant. Differentiating by p gives
(P) _ 2πκp dp (7)
Now, the radius in the image, p is related to the radial angle of a ray reflected from the mirror, θ by the focal length of the camera, / (a constant)
p =/tan(θ) (8)
so differentiating p(p) by θ and substituting (7) and (8) gives (P) _ (p) d θθ ep dθ θtan(θ)
= 2πκp/ (9)
= 2πκ 2 tan(θ)(l +tan2(θ))
Now, it is required that the image pixel density be invariant to angle of elevation in the scene which leads to more of the scene being imaged towards the perimeter, so
p{p) = βφ + C(φ) (10)
where β and C(φ) are constants. Differentiating both sides of (10) by φ and substituting (1) and (9) gives
β 5φ _ dp(ρ) 6φ 3φ β_ (P) . dd_ aθ ΘΦ (11)
= 2πκ/2 tan(θ)(l +tan2(θ)) o(θ)
Rearranging (11) gives
α(θ)
Figure imgf000012_0001
where Bα is a constant. Integrating this expression for α(θ) by θ gives an expression for φ (see (1)), the elevation of an object imaged at angle θ. That is D φ = _^ tan2(θ) + φ(θ = 0) (13)
where φ (θ = 0) is a constant of integration.
The constants Bα and φ (θ = 0) can be determined from the maximum and minimum values of θ and φ which are known for a desired mirror configuration, using (13).
2(φ - φ)
R tan2(θ) - tan2(θ) (M) φ(θ = 0) = φ - — tan2 (θ)
It appears not possible to find an analytical solution to (6) if is a function of θ, so there is no explicit equation for the mirror shape. Instead, a differential equation solver is needed to find solutions to (6) over the range of θ (the mirror surface).
Figures 4A and 4B show for comparison a constant gain mirror and a variable gain mirror with the same camera field of view and range of elevations imaged. The rays shown are constantly spaced in θ, with about 2° between each ray. It is clear from Fig. 4A that in the constant gain case these rays are constantly spaced in φ, with about 8.5° between each ray, and from Fig. 4, that the spacing between the rays in the variable gain case increases with increasing φ. So, in the variable gain case, a greater proportion of the scene is imaged towards the outer edge of the polar image. This is also shown in Figures 5 A and 5B, ray traced images reflected in a constant gain and variable gain mirror with the same range of elevations visible. 1.2 Panoramic Stereo Using a Variable Gain Mirror
A mirror with two convex portions or a double mirror is required. The radial profile of a double mirror is shown in Figure 8. The mirror arrangement for panoramic stereo with variable gain mirrors will necessarily be different than for constant gain mirrors due to the variation of the mirror gain, . The gain must vary in a constant fashion over the entire double mirror so that the constant pixel density theorem will hold over the entire image. If the minimum and maximum elevations viewed (φ and φ ) are to be equal for both mirrors in the double mirror system, the range of reflected angles (V - θ ) cannot be equal for the two mirrors. The minimum and maximum angles of reflected rays captured by the camera over the entire mirror surface are known from camera geometry. Therefore the minimum ray reflected from the lower mirror (θ ) and the maximum ray reflected from the upper mirror (θ2 ) are known. So, since (12) holds over the entire mirror, Bα is constant, and from (14)
Figure imgf000014_0001
tan2(θ,) tan2(θ,) = tan22) + tan2(θ,)
It is desirable to minimise the gap in the radial direction between the images from the two mirrors so as to maximise usage of the camera field of view. For minimum gap θx = θ^ , so
2 tan (θ,) = tan22) + tan2(θ.)
Figure imgf000014_0002
Figures 6A, 6B and 7 A, 7B show graphical and ray traced comparisons of constant and variable gain double mirror systems viewing the same scene.
2.4 Calculation of Range for a Variable Gain Panoramic Stereo System
The information available for range calculation are the image angles for a single object reflected in both mirrors, θ, and θ2 as shown in Figure 8. The two mirrors θj and θ2 form a reflective surface. The differential equations (6) for the surfaces are known. In the calculations that follow only the lower mirror is examined as the results are identical for the upper mirror.
In order to find the position of object P, the equations of the incident beams from P to each mirror reflection point (rλ, θ,) and (r2, θj) must be found. These equations can then be solved simultaneously to give the position of object P, (xP, yP).
Figure imgf000015_0002
Figure imgf000015_0001
where mn is the gradient of the incident beam to the lower mirror and Cn is the equation constant. The equation constant is given by
Cn =yrmnxχ (18)
where
Xj = r, sin θj y, = Tj cos θj (19)
are the Cartesian coordinates of the reflection point (r,, Q . The gradient of the incident beam is found using the law of reflection
mu (20)
Figure imgf000016_0001
where mR] is the gradient of the reflected beam from the lower mirror to the camera and dy,/dx, is the gradient of the lower mirror profile at the reflection point. The gradient of the reflected beam is mR] =tanθ, (21)
The gradient of the mirror profile for the lower variable gain mirror is found as in the constant gain case, from dy. dy dx. dxχ dθ «_ ,
Figure imgf000017_0001
dr. sinθ, +r, cosθ. dθ, ' ' '
where dr/dθ for either mirror of the variable gain mirror configuration is found by integrating (5) and substituting (12).
Figure imgf000017_0002
tan" (tan (θ) + tan3(θ)) dθ (23)
Figure imgf000017_0003
1 B„ ,
--Θ - Aitan2 (θ) + E>
2 4
where D is a constant of integration. Rearranging (23) gives
dr_ 1 /?
- r cot - ±Q - __ tan2 (θ) + (24) dθ 2 4
ow from (23) and (2),
= γ + i-θ + -≥ tan2 (θ) (25) 2 4
o, for the lower variable gain mirror profile = iι + ^ + tan2 (^)
similarly for D2, for the upper variable gain profile.
So, by substituting (24) into (22) gives the gradient of the variable gain mirror profiles at any point. Note that as in the constant gain case, the gradient depends only on θ.
The equation constants for the incident beam equations from (18) require the polar coordinates of the reflection points from each mirror, (rlt θ,) and (r2, θ2) . Since the variable gain mirror equations are not known exactly, r; and r2 must be found using a differential equation solver to find solutions to (26) at θλ and θ2.
2 Back-to-back Stereo Mirror Families
A key disadvantage of single camera stereo panoramic systems is that since there are two images of the "same" scene, the pixels assigned to each image is half that for non stereo panoramic imaging and the two images do not share an equal number of pixels in constant gain schemes. Actually, the panoramic stereo double mirror method typically causes the view of a scene in one radial direction to be compressed into around 1/4 the field of view of the camera.
A method to achieve panoramic stereo with less image compression is to use two cameras and two single curved mirror surfaces back to back, as shown in Fig. 9. This method compresses the imaged scene into 1/2 the field of view of the camera, and indeed each image has an equal share of the total number of pixels available. There are, however, possible alignment problems with this system as with any stereo system using two cameras to capture two views of a scene.
An advantage of the scheme proposed in Fig. 9 is that the stereo cone can be symmetric about the horizon using two cameras with equal fields of view and the maximum and minimum angles of elevation reflected by the two mirrors being equal. The angle covered by the stereo cone in this case is 2φ - π. Fig.9 shows the general case where the maximum and minimum angles of elevations viewed by each camera need not be equal. The range of elevations must still be equal for the fields of view to be aligned.
The number of free parameters to be specified are reduced here as the minimum angle of elevation (φ ) and from one mirror must be parallel to the maximum angle of elevation (φ ) from the other mirror. This is to ensure that the fields of view are parallel. So, with reference to Fig. 9
= π - φ (26)
In the scheme of Fig. 9, the mirror families can be either constant gain or resolution invariant.
2.1 The Use of Double Mirrors in a Back to Back Design
Fig. 10 shows a back to back design incorporating double mirrors. Although the figure shows constant gain mirrors, the double mirror can also have a variable gain. The advantage to this system is that the stereo cone from the back to back configuration combines with the stereo cones from the double mirror configuration to increase the total area imaged in stereo. In this configuration, the fields of view of each double mirror pair need not be aligned as in previous examples. For symmetry about the horizon φ^ = φ^ , φ^ = φ^ , φ3 = φ, and φ4 = φ2 . The constraints Φ3 = π - φ2 Φ4 = π - φ,
align the three stereo cones.
It is also possible to increase the total stereo cone further by allowing the mirror pairs to have different gains.
3 Forward Looking Mirror Design
An example of a forward looking mirror design is shown in Fig. 11. For many applications, it is desirable to have a panoramic camera looking out from, say, a hemisphere, somewhat as an eye of a bird, or perhaps two such on either side of a "nose cone". There are aerodynamic considerations or other protrusion considerations which motivate such a "forward looking" system. This configuration is termed forward looking because the camera faces towards the scene. Either a constant or variable gain mirror (double or single) could be used for the curved mirror in the system. The planar mirror is an annulus or circle interposed such that all rays reflected from the curved mirror are reflected into camera o positioned behind the curved mirror. The dotted lines in Fig. 11 show where the reflected rays would converge if the planar mirror was removed and the dotted camera shows the camera o ' for an equivalent system without the planar mirror.
In order for the rays reflected by the planar mirror to converge at the new camera position, the planar mirror must be the perpendicular bisector of the line joining the old and new camera locations. Hence the distance between the camera locations is 2D where D is defined in Fig. 11 as the distance from either camera to the planar mirror. The introduction of the planar mirror into the system does increase the possibility of alignment difficulties as the planar mirror must be perpendicular to the camera axis and also be positioned so as to reflect all rays from the curved mirror into the camera without occluding the view of the curved mirror. The maximum value for D,D , is when the maximum beam reflected from the mirror system (the D beam reflected at point b on the planar mirror) into camera o grazes the curved profile at c. In this case
_ _ r cos (θ)[tan (θ) + tan (θ)]
(27) 2 tan (θ)
D , defines the minimum height for the mirror system, H . In practice, the value for D needs to be slightly smaller to avoid occlusion, leading to a larger mirror system height. The general equation for the height of the mirror system is
H = r cos (θ) - D (28)
It should also be noted that θ must be greater than zero for camera o to be located behind the curved mirror. Also, φ _ θ if the minimum elevation ray φ is not to be occluded by the planar mirror.
Fig. 12 shows a design that incorporates the ideas of Sections 2 and 3. It consists of two forward looking systems back to back, giving a design reminiscent of a eye mounted on a stalk, such as a crab's eye. The "stalk" for this system would be hidden from view by the lower planar mirror. In this arrangement portions are provided in the curved mirror to provide for reflection of rays from the curved surface to the camera by the plane mirrors. The foregoing describes only some aspects of the present invention and modifications can be made without departing from the scope of the invention.

Claims

CLAIMS:
1. A panoramic imaging system including an imaging device having an image plane and a first field of view, a first reflective surface having at least one circularly symmetric portion convex in a radial direction disposed in said first field of view to provide an expanded panoramic second field of view, the profile of the or each convex portion providing a varying gain between the fields of view in the radial direction to limit variation in the solid angle of view across the image plane of the imaging device.
2. A panoramic imaging system as claimed in claim 1 wherein the profile of the or each convex portion provides a substantially uniform solid angle of view across the image plane.
3. A panoramic imaging system as claimed in claim 1 or claim 2 wherein the profile of the or each convex portion at least approximates a profile defined in polar co-ordinates by the equation:
Figure imgf000023_0001
where r is the radial distance from the reflective surface to the imaging device θ is the angle from the optical axis of the imaging device α (θ) is th& mirror gain given by
α (θ) = Bα [tan (θ) + tan3 (θ)]
2(φ - 4
5. - tan2(θ) - tan2(θ)
φ and φ are the maximum and minimum elevations viewed ϋ and θ are the maximum and minimum radial angles imaged.
4. A panoramic imaging system as claimed in claim 3 wherein the profile of the or each convex portion includes by a series spaced apart points defined by determining distance r for selected values of angle θ.
5. A panoramic imaging system as claimed in claim 4 wherein the selected values of θ are separated by about 1/5°.
6. A panoramic imaging system as claimed in any one of claims 1 to 5 including a first reflector surface having at least two of said convex portions arranged to respectively provide at least partially overlapping panoramic second fields of view for range determination.
7. A panoramic imaging system as claimed in claim 6 wherein said panoramic second fields of view are substantially co-incident.
8. A panoramic imaging system as claimed in claim 7 wherein said at least two convex portions form a continuous reflective surface.
9. A panoramic imaging system as claimed in any one of claims 1 to 8 including two of said first reflective surfaces each having an associated image plane with corresponding first fields of view, and at least one convex portion of each first reflective surface providing respective panoramic second fields of view, said first reflective surface being arranged back to back such that said reflective second fields of view at least partially overlap.
10. A panoramic imaging system as claimed in any one of claims 1 to 9 further including a second reflective surface interposed between the image plane and said second reflective surface.
11. A panoramic imaging system as claimed in claim 10 wherein the imaging device is positioned behind the second reflective surface.
12. A panoramic imaging system as claimed in claim 11 wherein an aperture is provided in said first reflective surface to provide said first field of view from the imaging device.
5 13. A panoramic imaging system as claimed in any of claims 10 to 12 wherein said second reflective surface is substantially planar.
14. A reflective surface for use in a panoramic imaging system including an imaging device having an imaging plane and a first field of view, said reflective surface having at least 10 one circularly symmetric portion convex in a radial direction with a profile providing varying gain in the radial direction between an expanded panoramic second field of view provided by the reflective surface and the first field of view to limit variation in the solid angle of view across the image plane of the imaging device.
15 15. A reflective surface as claimed in claim 14 wherein the profile of the or each convex portion provides a substantially uniform solid angle of view across the image plane.
16. A reflective surface as claimed in claim 14 or claim 18 wherein the profile of the or each convex portion at least approximates a profile defined in polar co-ordinates by the 0 equation: dr r cot V* dθ Jo a(θ)) dθ
where r is the radial distance from the reflective surface to the imaging device θ is the angle from the optical axis of the imaging device α (θ) is the mirror gain given by
5 α (θ) = Bα [tan (θ) + tan3 (θ)]
2(φ - φ)
S tan2(θ) - tan2(θ) φ and φ are the maximum and minimum elevations viewed ϋ and θ are the maximum and minimum radial angles imaged.
17. A reflective surface as claimed in claim 16 wherein the profile of the or each convex 5 portion includes by a series spaced apart points defined by determining distance r for selected values of angle θ.
18. A reflective surface as claimed in claim 17 wherein the selected values of θ are separated by about 1/5°.
10
19. A reflective surface as claimed in any one of claims 14 to 18 including a first reflector surface, having at least two of said convex portions arranged to respectively provide at least partially overlapping panoramic second fields of view for range determination.
15 20. A reflective surface as claimed in claim 19 wherein said panoramic second fields of view are substantially co-incident.
21. A reflective surface as claimed in claim 20 wherein said at least two convex portions form a continuous reflective surface.
PCT/AU2000/000022 1999-01-15 2000-01-14 Resolution invariant panoramic imaging WO2000042470A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
AU24249/00A AU757227B2 (en) 1999-01-15 2000-01-14 Resolution invariant panoramic imaging
EP00902499A EP1151349A1 (en) 1999-01-15 2000-01-14 Resolution invariant panoramic imaging
JP2000593986A JP2002535700A (en) 1999-01-15 2000-01-14 Resolution-invariant panoramic imaging

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
AUPP8191 1999-01-15
AUPP8191A AUPP819199A0 (en) 1999-01-15 1999-01-15 Resolution invariant panoramic imaging

Publications (1)

Publication Number Publication Date
WO2000042470A1 true WO2000042470A1 (en) 2000-07-20

Family

ID=3812396

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/AU2000/000022 WO2000042470A1 (en) 1999-01-15 2000-01-14 Resolution invariant panoramic imaging

Country Status (4)

Country Link
EP (1) EP1151349A1 (en)
JP (1) JP2002535700A (en)
AU (1) AUPP819199A0 (en)
WO (1) WO2000042470A1 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002008817A2 (en) * 2000-07-21 2002-01-31 Lee Scott Friend Stereoscopic omnidirectional imaging devices
FR2826221A1 (en) * 2001-05-11 2002-12-20 Immervision Internat Pte Ltd METHOD FOR OBTAINING AND DISPLAYING A VARIABLE RESOLUTION DIGITAL PANORAMIC IMAGE
DE102005029612A1 (en) * 2005-06-23 2007-01-04 Hochschule Bremen Mirror system for producing cube map panorama, has one of lens and mirror surface, two pairs of lens, and another pairs of lens arranged in orthogonal spatial directions to each other
WO2012124275A1 (en) * 2011-03-11 2012-09-20 Sony Corporation Image processing apparatus, image processing method, and program
US10204398B2 (en) 2016-02-16 2019-02-12 6115187 Canada, Inc. Image distortion transformation method and apparatus

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4554954B2 (en) * 2004-02-19 2010-09-29 康史 八木 Omnidirectional imaging system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4449786A (en) * 1978-06-19 1984-05-22 Mccord Robert C Rearview mirror
US4549208A (en) * 1982-12-22 1985-10-22 Hitachi, Ltd. Picture processing apparatus
US4566763A (en) * 1983-02-08 1986-01-28 Budapesti Muszaki Egyetem Panoramic imaging block for three-dimensional space
AU7486194A (en) * 1993-08-25 1995-03-21 Australian National University, The Panoramic surveillance system
US5502309A (en) * 1994-09-06 1996-03-26 Rockwell International Corporation Staring sensor
US5627675A (en) * 1995-05-13 1997-05-06 Boeing North American Inc. Optics assembly for observing a panoramic scene

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4449786A (en) * 1978-06-19 1984-05-22 Mccord Robert C Rearview mirror
US4549208A (en) * 1982-12-22 1985-10-22 Hitachi, Ltd. Picture processing apparatus
US4566763A (en) * 1983-02-08 1986-01-28 Budapesti Muszaki Egyetem Panoramic imaging block for three-dimensional space
AU7486194A (en) * 1993-08-25 1995-03-21 Australian National University, The Panoramic surveillance system
US5502309A (en) * 1994-09-06 1996-03-26 Rockwell International Corporation Staring sensor
US5627675A (en) * 1995-05-13 1997-05-06 Boeing North American Inc. Optics assembly for observing a panoramic scene

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002008817A2 (en) * 2000-07-21 2002-01-31 Lee Scott Friend Stereoscopic omnidirectional imaging devices
WO2002008817A3 (en) * 2000-07-21 2002-04-18 Lee Scott Friend Stereoscopic omnidirectional imaging devices
FR2826221A1 (en) * 2001-05-11 2002-12-20 Immervision Internat Pte Ltd METHOD FOR OBTAINING AND DISPLAYING A VARIABLE RESOLUTION DIGITAL PANORAMIC IMAGE
WO2002093908A3 (en) * 2001-05-11 2003-02-27 Immervision Internat Pte Ltd Method for capturing and displaying a variable-resolution digital panoramic image
US6844990B2 (en) 2001-05-11 2005-01-18 6115187 Canada Inc. Method for capturing and displaying a variable resolution digital panoramic image
DE102005029612A1 (en) * 2005-06-23 2007-01-04 Hochschule Bremen Mirror system for producing cube map panorama, has one of lens and mirror surface, two pairs of lens, and another pairs of lens arranged in orthogonal spatial directions to each other
WO2012124275A1 (en) * 2011-03-11 2012-09-20 Sony Corporation Image processing apparatus, image processing method, and program
JP2012190299A (en) * 2011-03-11 2012-10-04 Sony Corp Image processing system and method, and program
CN103443582A (en) * 2011-03-11 2013-12-11 索尼公司 Image processing apparatus, image processing method, and program
EP2671045A1 (en) * 2011-03-11 2013-12-11 Sony Corporation Image processing apparatus, image processing method, and program
EP2671045A4 (en) * 2011-03-11 2014-10-08 Sony Corp Image processing apparatus, image processing method, and program
US10204398B2 (en) 2016-02-16 2019-02-12 6115187 Canada, Inc. Image distortion transformation method and apparatus
US10748243B2 (en) 2016-02-16 2020-08-18 Immervision, Inc. Image distortion transformation method and apparatus

Also Published As

Publication number Publication date
EP1151349A1 (en) 2001-11-07
JP2002535700A (en) 2002-10-22
AUPP819199A0 (en) 1999-02-11

Similar Documents

Publication Publication Date Title
US6856472B2 (en) Panoramic mirror and system for producing enhanced panoramic images
Conroy et al. Resolution invariant surfaces for panoramic vision systems
US6304285B1 (en) Method and apparatus for omnidirectional imaging
KR100599423B1 (en) An omnidirectional imaging apparatus
Nayar et al. Catadioptric image formation
TW528924B (en) Panorama image acquisition system, magnifying device for the panorama distorted image capture system, and the alignment device containing four fixed reflective surfaces
US6744569B2 (en) Method and apparatus for omnidirectional three dimensional imaging
US7837330B2 (en) Panoramic three-dimensional adapter for an optical instrument and a combination of such an adapter and such an optical instrument
JP3523783B2 (en) Omnidirectional viewing angle sensor
JP2000515704A (en) Wide angle image forming system and method
CN106125280B (en) Zigzag type optical system for field stitching
WO2000042470A1 (en) Resolution invariant panoramic imaging
EP2040463A2 (en) System for increasing horizontal field of view of a camera
JP5128037B2 (en) Stereo image acquisition device
JPH1195344A (en) Omniazimuth stereoscopic picture taking device
CN100492170C (en) Omnidirectional stereo vision imaging method and apparatus
WO2023103500A1 (en) Blind spot-free multi-view panoramic three-dimensional imaging device
US4747647A (en) Optical scanning device
AU757227B2 (en) Resolution invariant panoramic imaging
US7253969B2 (en) Spherical and nearly spherical view imaging assembly
Hicks Differential methods in catadioptric sensor design with applications to panoramic imaging
CN112305851A (en) Panoramic annular imaging optical system
CN210803962U (en) Panoramic annular imaging optical system
EP0941529A1 (en) Imaging system
HICKS et al. Catadioptric sensors for panoramic viewing

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AU CA JP US

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 2000902499

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 24249/00

Country of ref document: AU

ENP Entry into the national phase

Ref country code: JP

Ref document number: 2000 593986

Kind code of ref document: A

Format of ref document f/p: F

WWE Wipo information: entry into national phase

Ref document number: 09868909

Country of ref document: US

WWP Wipo information: published in national office

Ref document number: 2000902499

Country of ref document: EP

WWG Wipo information: grant in national office

Ref document number: 24249/00

Country of ref document: AU

WWW Wipo information: withdrawn in national office

Ref document number: 2000902499

Country of ref document: EP