WO2012139636A1 - Online vehicle camera calibration based on road surface texture tracking and geometric properties - Google Patents

Abstract

The invention relates to a method for online calibration of a vehicle video system evaluated from image frames of a camera containing features on the road. A portion of a road surface is captured by the camera in an image frame. A selection of at least two different features within the image frame is taken, the selected features being chosen as respective reference points. A sequence of at least one more image frame is performed with the camera while locating within the new frame the features chosen as reference points to be tracked. Analysing in a virtual image space the trajectory covered by the reference points during driving interval between image sequence by identifying a geometrical shape obtained by joining the located reference points between each other in each used images and considering the respective trajectories. A deviation of the resulting geometrical object from a parallelogram with corners defined by the reference points from at least two subsequent images is calculated while any measured deviation being used to define an offset correction of the camera.

Description

ONLINE VEHICLE CAMERA CALIBRATION BASED ON

ROAD SURFACE TEXTURE TRACKING AND GEOMETRIC PROPERTIES

BACKGROUND OF THE INVENTION

FIELD OF THE INVENTION

[0001] This invention relates generally to an online calibration of a vehicle video system, and particularly to a method for online calibration of the vehicle video system evaluated from image frames of a camera. It is also related to a computer program product for processing data relating to the online calibration of the vehicle video system, the computer program product comprising a computer usable medium having a computer usable program code embodied therewith, the computer program code being configured to perform any of the method steps. The present invention is further related to an online calibration system for a vehicle video system for processing the computer implemented online calibration method.

DESCRIPTION OF BACKGROUND

[0002] It is known to mount image capture devices, such as, for example, digital or analogue video cameras, on a motor vehicle in order to produce a video image of an aspect of the environment exterior of the vehicle. For example, in order to assist in parking and manoeuvring a vehicle in confined spaces, it is known to mount such image capturing devices on respective opposite sides of the vehicle, for example, on side rear view mirror housings which extend sidewardly from the driver and front passenger doors of the vehicle. The image capture devices are mounted in the side rear view mirror housings with the field of view of the image capture devices directed downwardly towards the ground for capturing plan view images of the ground on respective opposite sides of the vehicle adjacent the vehicle. Typically, a visual display unit is located in the vehicle, either in or on the dashboard, or in a location corresponding to that of a conventional interiorly mounted rear view mirror. When a driver is undertaking a parking manoeuvre or a manoeuvre in a confined space, a plan view image of the vehicle with the respective plan view images of the ground on respective opposite sides of the vehicle can be displayed on the visual display unit. The plan view display of the vehicle and the ground on respective opposite sides of the vehicle assists the driver in parking, and in particular, carrying out a parking manoeuvre for parking the vehicle in a parking space parallel to a kerb of a footpath or the like.

[0003] However, in order that the plan view images of the ground accurately reflect the positions of objects relative to the vehicle, which are captured in the images, it is essential that the plan view images of the ground juxtapositioned with the plan view image of the vehicle should accurately represent a top plan view of the ground adjacent the respective opposite sides of the vehicle exactly as would be seen when viewed from above. In other words, the edges of the respective plan view images of the ground which extend along the sides of the plan view image of the vehicle must correspond directly with the edge of the ground along the sides of the vehicle when viewed in plan view from a position above the vehicle. Otherwise, the positions of objects in the respective plan view images of the ground will not be accurately positioned relative to the vehicle. For example, if the edge of one of the plan view images of the ground adjacent the corresponding side of the plan view image of the vehicle corresponds with a portion of a plan view of the ground which is spaced apart from the side of the vehicle, then the positions of objects in the plan view image of the ground will appear closer to the vehicle in the image than they actually are. Conversely, if one of the image capture devices is mounted on a side mirror housing so that an image of a portion of the ground beneath a side of the vehicle is captured, the positions of objects captured in the plan view image will appear farther away from the vehicle than they actually are, with disastrous results, particularly if a driver is parking the vehicle parallel to a wall or bollards. Similar requirements apply also for front or rear placed image capture devices.

[0004] Accordingly, it is essential that the plan view images of the ground when displayed on the visual display screen juxtapositioned along with the plan view image of the vehicle must be representative of plan views of the ground on respective opposite sides of the vehicle exactly as would be seen from a top plan view of the vehicle and adjacent ground. In order to achieve such accuracy, the image capture devices would have to be precision mounted on the vehicle. In practice this is not possible. Accordingly, in order to achieve the appropriate degree of exactness and accuracy of the plan view images of the ground relative to the plan view image of the vehicle, it is necessary to calibrate the outputs of the image capture devices. Calibration values determined during calibration of the image capture devices are then used to correct subsequently captured image frames for offset of the image capture devices from ideal positions thereof, so that plan view images of the ground subsequently outputted for display with the plan view image of the vehicle are exact representations of the ground on respective opposite sides of the vehicle. Such calibration can be accurately carried out in a factory during production of the motor vehicle. Typically, the image capture devices are relatively accurately fitted in the side mirror housings of the motor vehicle, and by using suitable grid patterns on the ground, calibration can be effected. However, the environments in which motor vehicles must operate are generally relatively harsh environments, in that side mirror housings are vulnerable to impacts with other vehicles or stationary objects. While such impacts may not render the orientation of the side mirror housing unsuitable for producing an adequate rear view from a rear view mirror mounted therein, such impacts can and in general do result in the image capturing device mounted therein being knocked out of alignment, in other words, being offset from its ideal position.

Additionally, where a vehicle is involved in a crash, or alternatively, where a side mirror housing requires replacement, re-calibration of the image capture device refitted in the new side mirror housing will be required. Such re-calibration, which typically would be carried out using a grid pattern on the ground, is unsatisfactory, since in general, it is impossible to accurately align the vehicle with the grid pattern in order to adequately calibrate the image capture device, unless the calibration is being carried out under factory conditions. The same applies for rear or front placed image capture devices even in the case, those devices are placed in the interior of the vehicle since harsh conditions apply also there. [0005] In WO2009/027090 is described a method and system for online calibration of a vehicle video system using vanishing points evaluated from frames of a camera image containing identified markings or edges on a road. In US2009/0290032 is described a system and method for calibrating a camera on a vehicle by identifying at least two feature points in at least two camera images from a vehicle that has moved between taking the images. The method then determines a camera translation direction between two camera positions. Following this, the method determines a ground plane in camera coordinates based on the corresponding feature points from the images and the camera translation direction. The method then determines a height of the camera above the ground and a rotation of the camera in vehicle coordinates.

SUMMARY OF THE INVENTION

[0006] In view of the above, there is a need for a method and a calibration system for calibrating an output of an image capture device or camera mounted on a vehicle to compensate for offset of the camera from an ideal position, such method and calibration system being independent from road marking while still very reliable.

[0007] According to a first aspect of the embodiment of the present invention a method for online calibration of a vehicle video system is evaluated from image frames of a camera containing features on the road. The method comprises the following steps of capturing by the camera of a portion of a road surface in an image frame. Then, a selection of at least two different features within the image frame is taken, the selected features being chosen as respective reference points. A sequence of at least one more image frame is performed with the camera while locating within the new frame the features chosen as reference points to be tracked. Analysing in a virtual image space the trajectory covered by the reference points during driving interval between image sequence by identifying a geometrical shape obtained by joining the located reference points between each other in each used images and considering the respective trajectories. A deviation of the resulting geometrical object from a parallelogram with corners defined by the reference points from at least two subsequent images is calculated while any measured deviation being used to define an offset correction of the camera. In some alternative, a row of features can be selected within the image frame for the reference points. In such a way, multiple features can be tracked allowing a parallel analysis of different geometrical shape.

[0008] Usually, no pre-selection of texture features of the ground from the road are performed. However, some selection of the texture features can be possibly applied to achieve some higher performance. Also, a specific window size can be used for the selection of features within the image frame, the specific window size being possibly correlated to driving parameters of the vehicle like the speed or the steering angle of the vehicle. A correlation search window is defined for tracking the selected reference points, the size of the correlation search window being adapted possibly according to available processing power. Accordingly, the size of the window could be chosen relatively big in case strong processing power could be used for the tracking step. A filtering can be applied using an image filter with band-pass like properties.

[0009] In an alternative embodiment according to the invention, a correlation search window is defined for each reference point. In a further embodiment, a subtraction of possible distortion of the camera on the identified geometrical object is performed such to obtain a quadrilateral to be compared to the defined parallelogram. This is particularly relevant for a fish-eye camera. The performed subtraction corresponds then to a transformation to a non-fish-eye rectilinear image space.

[0010] In some embodiment, driving parameters of the moving vehicle are used for optimizing the tracking of the reference points. Also the speed of the moving vehicle can be used for calculating the distance of the reference points when tracked between image frame sequences allowing determining the height of the camera and in case correcting it.

[0011] In a particular embodiment according to the invention, an offset rotation as online calibration is defined with respect to a measured deviation from the parallelism between lines obtained from subsequent image frames by joining the tracked reference points within an image frame for an offset rotation about the x-axis longitudinal to the vehicle. A similar approach can be applied for an offset rotation about the y-axis transverse to the vehicle, this time by considering the trajectory of the tracked reference points at subsequent image frames. Also, a deviation can be measured such that the lines joining the tracked reference points in subsequent image frame being rotated in the frame corresponds to an offset rotation about the vehicle z-axis for a camera located at a wing mirror, at rear or at the front of the vehicle.

[0012] In another embodiment according to the invention, steering angle information of the vehicle can be taken into account. This information can be of use in the case the vehicle is turning when performing sequence of image frames to transform accordingly the identified geometrical object such to obtain a quadrilateral to be compared to the defined parallelogram. Alternatively, some error function can be used to be minimized to take into account a deviation from a straight trajectory by the vehicle turning when performing the sequence of image frames. This is of advantage when the steering angle is not easily available for the analysis of the trajectory covered by the reference points. And then the resulting geometrical object is compared to a quadrilateral like lying on concentric arcs.

[0013] According to a second aspect of the embodiment, a computer program product for processing data relating to online calibration of a vehicle video system is also described and claimed herein. The computer program product comprises a computer usable medium having computer usable program code embodied therewith, the computer usable program code being configured to perform the above summarized methods.

[0014] According to a third aspect of the embodiment, an online calibration system for a vehicle video system is also described and claimed herein. The online calibration system comprises a computer program product for processing data relating to an online calibration method and an image processing apparatus with a camera for taking image frames to be used by the online calibration method such to perform the above

summarized methods. [0015] Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better

understanding of the invention with advantages and features, refer to the description and to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

[0017] FIG. 1 illustrates a side view and a top view of a vehicle with a 3d co-ordinate system;

[0018] FIG. 2 illustrates a flow diagram according to the invention;

[0019] FIG. 3 illustrates one example of feature tracking for a wing mirror camera

according to the invention;

[0020] FIG. 4 illustrates another example of feature tracking according to the invention;

[0021] FIG. 5 illustrates a further example of feature tracking for a wing mirror fish-eye camera according to the invention;

[0022] FIG. 6 illustrates a way to measure error correction according to the invention;

[0023] FIG. 7 illustrates one example of feature tracking for a rear- view camera.

[0024] The detailed description explains the preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings. DETAILED DESCRIPTION OF THE INVENTION

[0025] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0026] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0027] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0028] Program code embodied on a computer readable medium may be transmitted within the vehicle using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0029] Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the vehicle's computer, partly on the vehicle's computer, as a stand-alone software package, partly on the vehicle's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the vehicle's computer through any type of wireless network, including a wireless local area network (WLAN), possibly but not necessarily through the Internet using an Internet Service Provider.

[0030] Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0031] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

[0032] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0033] Present invention is a means of calibrating a camera on a vehicle to determine the extrinsic camera rotation parameters relative to the vehicle co-ordinate system when the camera is positioned in an undetermined relevant position on the vehicle body.

Calibration allows the vehicle manufacturer to provide a geometrically representative and more importantly a useful view to a vehicle user. Additionally, by extension, the height of the camera off the ground plane (XY-plane) can be determined if the speed of the vehicle is known.

[0034] Turning now to the drawings in greater detail, it will be seen that in Figure 1 is shown a side view and a top view of a vehicle 1 with a defined three dimensional coordinate system XYZ as used in the following. X-axis is chosen along the longitudinal direction, Y-axis along the transverse direction and Z-axis along the vertical direction of the vehicle 1. [0035] The camera, whether placed at rear, front or wing-mirror, captures a significant portion of road surfaces in the scene. Such portion of road surfaces is stored in image frames from which at least two different features of the road surfaces are selected. The selected features are then chosen as respective reference points.

[0036] Fig. 2 shows a flowchart according to present invention. The online calibration method is started 10 at some stage when vehicle is driving. Such starting event 10 can be programmed in different ways e.g. after a predefined time during driving or after travelling some predefined distance or a combination of several parameters.

[0037] In image frame N captured 1 1 (i.e., the first frame of the online calibration) two features are selected for tracking. Arbitrary positions can therefore be selected though this is extendable to using different methods to select the first calibration feature. At least two different texture features on the road surfaces are chosen as respective reference points 12 to be tracked.

[0038] A selected feature or texture feature is defined in the present invention as a window of any defined size (m x n pixels) in the image frame that is associated with a given point on the road surface. Some implementation uses a window of 16 x 16 pixels in the luminance channel of the frame. However, this is extendable to a texture feature of any size. It can also include information from the chroma channels, or from the R (red), G (green) and B (blue) channels if RGB color space is utilized. Alternately, information of RGBA (A for alpha) color space could also be considered if used for the definition of the selected feature as reference point on the road surface.

[0039] No pre-selection of 'strong' features is usual performed. Any kind of textures are always tracked from the same pre-defined starting positions. Textures that are not suitable for tracking will be most likely matched at the wrong position in the next frame. For a pair of tracks that consist of more than two points, having a texture matched at the wrong position will cause a noticeable deformation across the track's length such as a sharp bend which is detected algorithmically during the track-validation stage of the algorithm and causes the entire track to be rejected. Alternately, some sort of feature preselection can be applied possibly improving overall performance of the used method.

[0040] In image frame N + 1 captured 13 (i.e. the second frame after the start of online calibration), the algorithm according to the present invention selects 14 a search window in which it searches for the feature being tracked. The window location is selected based on the location of the texture feature in the previous frame. Within this window, a cross- correlation can be performed to determine 15 the location of the tracked texture feature in the current frame. But any type of correlation could also be used to find the texture in the search window such as cross-correlation, SAD (sum of absolute difference), SSD (sum of squared differences) or other, the choice being possibly taken according to the measured efficiency in terms of processing time and detection performance. A filtering with an image filter with band-pass like properties is advantageously applied on the texture and the search window.

[0041] The chosen size of the window can impact on the maximum speed of the vehicle at which this algorithm will operate successfully. That is, should the vehicle be traveling at a high speed, the feature will not be found within the chosen window size and location as easily. However, the size of this window is programmable, and is dependent on processing power available. A larger window size will allow for operation at greater speeds. A predictive tracking can be applied using the speed of the vehicle gathered via CAN (Controller area network) bus. This allows to use a relatively small search window placed around the expected location of the feature in the image at each frame.

[0042] Once the reference points are located at the image frame N + 1 , a geometrical shape is then identified 16. Such shape is build in a virtual image space by joining the located reference points between each other in image frame N and N + 1 and joining the same reference point from image N to N + 1 thus considering the respective trajectories of the reference points. The obtained geometrical shape shows a shape which possibly deviates from a parallelogram with corners defined by the reference points from the image frame N and N + 1. Such deviation is directly related to a slight alteration of the rotation from the camera of the video system to be calibrated. Therefore, once a deviation is defined 17, a calibration of the camera which captured the two analyzed image frameN andN + 1 can be performed 18. The online calibration method may be stop here 19 before being restarted at a later time or can be further performed on subsequently captured image frame 20 during some time interval.

[0043] Fig. 3 shows the combination of successive image frames captured by some camera of the vehicle video system in a virtual image space. The showed example corresponds to a camera which exhibits no fisheye, and no rotation i.e. the camera is looking along the vehicle Z-axis at the ground plane with no rotation about the vehicle X, Y or Z-axes. The two left most visible corners 21 and 22 correspond to the two features within the image frameN selected as reference points. Both reference points are selected within a specific window size 23 defined in the present case by 16 x 16 pixels while other sizes could be applied also. Advantageously, the window size can be correlated to driving parameters e.g. to the speed and/or the steering curvature of the vehicle when performing the online calibration. The two selected reference points are then tracked using some correlation search window 24 for each reference points 21 and 22. The search window 24 includes respectively the initially selected reference points 21, 22 at image frameN and the location where the respective reference points 25, 26 shall be located at the successive image frameN + 1 . The search window will be of a size possibly also related to driving parameters and to the time interval between two successive image frames. The search window can also be defined according to the available processing power. Fig. 3 shows the selected reference points 21, 22 tracked over a sequence of 5 image frames from image frameN tillN + 4 using a side or wing mirror camera therefore capturing not only the ground of the road but also part of the car 30.

[0044] The two reference points 21, 22 are joined between each other within an image frame, hereN, to obtain a straight line 31 in a virtual image space. Such straight line is compared to the straight lines 32-35 obtained by joining the tracked reference points within successive image frame tillN+ 4. If the straight lines 31-35 are parallel (as it is shown on fig. 3) then no deviation about the X-axis from the ideal position can be measured.

[0045] A similar comparison is performed by joining this time the reference points 21, 22 from image frameN with the same respective reference points 25, 26 from successive image frame, hereN + 1 tillN + 4. Two portions 36 and 37 of two straight lines are obtained in the virtual image space, the two portions 36, 37 being then compared between each other to determine a possible deviation from a parallel property. Such a comparison is performed using the successive image frameN tillN + 4 allowing measuring a deviation as soon as captured (not the case on Fig. 3) about the Y-axis.

[0046] The analysis of a possible deviation from a parallelism between the straight lines 31, 32 obtained by joining the reference points 21, 22 within image frameN and reference points 25, 26 within image frameN + 1 as well as between the straight lines 36, 37 obtained by joining the same reference points 21, 22 from image frameN and from image frameN + 1, namely respectively 25, 26 corresponds to the analysis of a geometrical shape built using the four obtained straight lines 31, 32, 36, 37. The deviation of such geometrical shape with four corners 21, 22, 25 and 26 from a parallelogram allows to define an offset correction to apply to the corresponding camera of the vehicle video.

[0047] Fig. 4 shows the capture of successive image frame using the same camera as in Fig. 3 but this time showing some rotation of the camera. Indeed, the obtained geometrical shape deviates from a perfect parallelogram in the way that straight lines 41, 42 joining the two reference points 21 , 22 in image frameN and 25, 26 in image frameN + 1 are no more perfectly parallel. This is an indication of a rotation from the camera about the X-axis. Also the straight line 43, 44 joining the same reference points from image frameN to image frameN + 1 respectively 21, 25 and 22, 26 are no more parallel either. This is an indication of a rotation from the camera about Y-axis. The use of those indications allows to define an offset to apply to calibrate accordingly the camera to correct those deviations. [0048] Usual vehicle video system uses fish-eye camera. Accordingly when tracking texture features from the ground using such fish-eye camera will return position in a distorted fish-eye space. Fig. 5 shows such distorted fish-eye space with two references points 51 , 52 selected using some window 53 within image frame N. A correlation search window 54 is defined for each reference points 51, 52 (only the one for reference point 51 is shown on fig. 5) to track the reference points from image frame N to image frame N + 1 till image frame N + 4. Prior to using the geometric properties to estimate the necessary parameters for the online calibration, the selected reference points are transformed to their position in non-fisheye rectilinear image space (resulting in the geometric shape as shown on fig. 4) using previously determined fisheye parameters, possibly determined in factory based calibration.

[0049] The number of frames across which features are tracked is a programmable parameter and can be limited by the speed of the vehicle and/or other practical considerations.

[0050] As shown in Fig. 3, if the camera of the vehicle video system is ideally located and the vehicle is travelling in a straight line at a constant speed (with acceptable tolerances), the tracked points form parallelograms in the captured frames. And as shown in Fig. 4 this does not hold in the presence of rotations about one of all of the X, Y and Z- axes. Such a rotation can be described using basic rotation matrices following known methods as explained e.g. at http://en.wikipedia.org/wiki Rotation matrix like Euler angle rotations.

[0051] Appropriate angles that describe the rotation of the cameras about the vehicle axes need to be determined such that the tracked features form the desired parallelogram shape. This can be done by using any solver or error minimization method, by

transforming the features using the rotation matrices and the estimate of the appropriate angles. Analytical methods based on homography could also be used. Rotation about the various axes will have the following effects:

• For a wing-mirror located camera, rotation about the vehicle X-axis will result in the lines that join the tracked points in the same frame to be non-parallel i.e. lines obtained by joining track points of spatially adjacent tracks - not points on the same track.

• For a wing-mirror located camera, rotation about the vehicle Y-axis will result in the lines that join the tracked features in subsequent frames to be non- parallel, and the distances between them to be unequal.

• For a wing-mirror located camera, rotation about the vehicle Z-axis will result in lines that join the tracked features to be rotated in the image.

[0052] All these effects can be used as a measure of error of the estimated rotation parameters, and can be used as an error function for any solver or minimization algorithm. Figure 6 outlines the effect of the rotations of the used camera. The rotations described here assume a wing-mirror based camera. Figures 6 shows the selected reference points 60, 61 between the different frames along respectively track 62 and track 63. The error function is then composed using following results:

• By the differences in dx (dxl, dx2) between opposite track segments. A

smaller difference gives a better parallelogram. This is affected only by the angular deviation in the X-axis.

• By the differences in dy (dy 1 , dy2) of individual track segments which is affected by the angular deviations in the Y-axis and Z-axis.

[0053] The different methods will be used on the appropriate axes for front-view and rear-view cameras. Fig. 7 shows the use of present invention for a rear- view camera. The ground 67 of the road with characteristic markings 68 and borders 69 as well as the rear deck 70 of the vehicle is clearly visible. Two reference points 71 , 72 are selected using some specific window size 73. The selected reference points 71 , 72 are tracked in the subsequent image frame by using some correlation search window 74 to be localized 75, 76. And a geometrical shape can be obtained in a virtual image space in a same way as for the side or wing mirror camera by joining the four corners 71, 72, 75 and 76 while performing a projection to take into account the fact that the camera is capturing the road surface with some inclination. Such projection is shown on the top of fig. 7 with the respective projected points 7Γ, 72', 75'and 76' . A deviation (not visible on fig. 7) from a parallelogram allows defining a correction function to be used as offset correction of the camera as for the wing or side mirror camera.

[0054] There is currently also the constraint that the tracks are parallel to the X-axis of the vehicle although this can be eliminated for by using the steering information or by having a range of predefined expected steering curvature. Additionally, by appropriate defining the positioning of the camera with reference to the vehicle coordinate system, the method can be used by vehicle located at any position on the vehicle, as long as an appropriate portion of the road texture is visible to the camera. If initial estimates of the camera extrinsic parameters are known (e.g. from vehicle mechanical data), they can be used as starting points for the calibration. Advantageously, the speed and steering information possibly available on the vehicular network (e.g. via CAN, LIN, Wireless or other) can be used to remove (relax) the necessity for the vehicle to be moving in a straight line and at constant speed. Furthermore, if the speed of the vehicle is considered, then the actual physical distance between the tracked features on the road surface is known. Therefore, using triangulation, the height of the camera can be determined. In application, a minimization method would be used to minimize the error over all the tracked method.

[0055] In case, the online calibration is most suitable only when vehicle is moving straight ahead then it is important to continuously monitor steering angle CAN signal and invalidate the tracking when the steering angle is not zero (within acceptable tolerance). It is possible to adapt present invention to recognize if the vehicle or car is travelling straight without dependencies on the steering CAN signal by only analyzing the geometry of the tracks. As a step further, present invention can be adapted to work also when the vehicle comprising the video system to be calibrated is not moving straight ahead.

Steering angle information can be used when analyzing the trajectory covered by the reference points to obtain a geometrical shape close to a quadrilateral i.e. removing any bending of the trajectory coming from the known turning action allowing a comparison with the defined parallelogram. [0056] However, using the steering angle value is not every time possible and possibly not desirable because it can create some dependency of the calibration on the accuracy of this value, latency etc. Therefore, an error function can be devised to take into account the fact that the quadrilaterals being the projection in the virtual space of the trajectories lie on concentric arcs (if no distortion like fish-eye distortion is present) when the vehicle is turning. Such error function will take into account the unknown steering angle as well as possible unknown offsets. And this error function will have to be minimized to achieve some calibration of the online vehicle video system.

[0057] Not only two reference points could be selected but three or more possibly stored in parallel in a row. Multiple features could be tracked from pre-defined starting positions on the image between the two frames to obtain motion vectors in a similar manner to the motion estimation techniques used for MPEG (moving picture experts group) compression. A grid of initial target positions is set and the motion of each target/texture is tracked in two consecutive frames. Parallelograms share an edge between rows of the grid but the columns are independent. The columns only allow to gather more information (motion vectors) in parallel like a pipeline.

[0058] Same online calibration method according to the present invention could then be applied to this grid like tracking. Possibly, a method for rejecting outliers (for the reference points) could be applied which should not necessarily be based on track continuity information across many adjacent frames in the case only pairs of adjacent frames are analyzed while each pair could be disjoined in time from the other pairs. Simple median filtering of the motion vector angles or other more sophisticated method can be used to effectively reject outliers.

[0059] The capabilities of the present invention can be implemented in software, firmware, hardware or some combination thereof.

[0060] As one example, one or more aspects of the present invention can be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has embodied therein, for instance, computer readable program code means for providing and facilitating the capabilities of the present invention. The article of manufacture can be included as a part of a computer system or sold separately. Additionally, at least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform the capabilities of the present invention can be provided.

[0061] While the preferred embodiment to the invention has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.

Claims

1. A method for online calibration of a vehicle video system evaluated from image frames of a camera containing features on the road, the method comprising the following steps of

• Capturing (11) by the camera of a portion of a road surface in an image frame; The method being characterised by the further steps of

• Selecting (12) at least two different features (21, 22, 51, 52, 71, 72) within the image frame, the selected features being chosen as respective reference points;

• Performing (13) with the camera a sequence of at least one more image frame while locating (15) within the new frame the features (25, 26, 75, 76) chosen as reference points to be tracked;

• Analysing in a virtual image space the trajectory covered by the reference points during driving interval between image sequence by identifying (16) a geometrical shape obtained by joining the located reference points between each other in each used images and considering the respective trajectories;

• Calculating a deviation (17) of the resulting geometrical object from a parallelogram with corners defined by the reference points from at least two subsequent images while any measured deviation being used to define an offset correction of the camera.

2. The online calibration method according to claim 1 whereby selecting texture features of the ground from the road as reference points.

3. The online calibration method according to claim 1 whereby using a specific window size (23, 53, 73) for the selection of features within the image frame, the specific window size being correlated to driving parameters of the vehicle.

4. The online calibration method according to claim 1 whereby a correlation search window (24, 54) is defined for tracking the selected reference points, the size of the correlation search window being adapted according to available processing power.

5. The online calibration method according to claim 1 whereby applying an image filter with band -pass like properties when analysing the trajectory covered by the reference points.

6. The online calibration method according to claim 1 whereby subtracting possible distortion of the camera on the identified geometrical object such to obtain a quadrilateral to be compared to the defined parallelogram.

7. The online calibration method according to claim 6 whereby the distortion being at least partly given by the distortion parameters from a used fish-eye camera such that the performed subtraction corresponds to a transformation to a non-fish-eye rectilinear image space.

8. The online calibration method according to claim 1 whereby using driving parameters of the moving vehicle for optimizing the tracking of the reference points.

9. The online calibration method according to claim 1 whereby using the speed of the moving vehicle for calculating the distance of the reference points when tracked between image frame sequences allowing to determine the height of the camera.

10. The online calibration method according to claim 1 whereby an offset rotation as online calibration is defined according to a measured deviation from the parallelism between lines obtained respectively

• from subsequent image frames by joining (31-35, 41, 42) the tracked reference points within an image frame for an offset rotation about the x-axis longitudinal to the vehicle,

• by the trajectory (36, 37, 43, 44, 62, 63) of the tracked reference points at subsequent image frames for an offset rotation about the y-axis transverse to the vehicle.

11. The online calibration method according to claim 1 whereby a deviation such that the lines joining the tracked reference points in subsequent image frame being rotated in the frame corresponds to an offset rotation about the vehicle z-axis for a camera located at a wing mirror, at rear or at the front of the vehicle.

12. The online calibration method according to claim 1 whereby using steering angle information of the vehicle to transform the identified geometrical object such to obtain a quadrilateral to be compared to the defined parallelogram.

13. The online calibration method according to claim 1 whereby using some error function to be minimized to take into account a deviation from a straight trajectory by the vehicle turning when performing the sequence of image frames while the resulting geometrical object being compared to a quadrilateral like lying on concentric arcs.

14. The online calibration method according to claim 1 whereby selecting a row of features within the image frame for the reference points.

15. A computer program product for processing data relating to online calibration of a vehicle video system, the computer program product comprising a computer usable medium having computer usable program code embodied therewith, the computer usable program code being configured to perform the steps of any of the preceding claims 1 to 14.

16. An online calibration system for a vehicle video system, the online calibration system comprising a computer program product for processing data relating to an online calibration method and an image processing apparatus with a camera for taking image frames to be used by the online calibration method such to perform the steps of any of the preceding claims 1 to 14.