WO1991018366A1

WO1991018366A1 - A method of detecting skew in form images

Info

Publication number: WO1991018366A1
Application number: PCT/US1991/003102
Authority: WO
Inventors: Yongchun Lee
Original assignee: Eastman Kodak Company
Priority date: 1990-05-21
Filing date: 1991-05-09
Publication date: 1991-11-28
Also published as: US5054098A; EP0482188A1; JPH05500285A

Abstract

Document contour vectorization and the use of a modified Hough transform are used in combination to detect the skew angle of a digitized form image so that image skew may then be corrected in the preprocessing of form images prior to document analysis and classification.

Description

A METHOD OF DETECTING SKEW IN FORM IMAGES

Technical Field of the Invention

The present invention relates to image processing techniques in general, and more particularly, to the automatic detection of skew in form images.

Backσround of the Invention

Digital coding of graphic information is commonly called for in a wide variety of contexts from facsimile data transmission to computerized photograph analysis and pattern recognition, to computer-aided-design applications. The first step in such digitizing is to scan the document in a controlled fashion, measuring the graphic value of the image at each point. Currently available scanning devices are capable of substantially simultaneously delivering a binary output signal for each of n lines of resolution cells, each cell being approximately 0.01 mm square. Thus a one meter long scan line of an engineering drawing for example,

5 would contain 10 such resolution cells; a single square centimeter would contain 10 resolution cells.

Where, as indicated above, the digitized information is in the form of raster output data from 0.01 mm resolution cells, a typical 80 character alphabetic line might then be coded as approximately 200 information signals for each 20 cm long scan line, a reduction of 99 percent compared

4 to the 2 x 10 bits of raw raster output data. When it is considered that a sheet of A4 paper contains 6 x 10 such resolution cells, it can be seen that such a coding is still very cumbersome, requiring over a million information signals to code a single page of bi-tonal writing, scan line by scan line. This inefficiency is addressed in the prior art by a number of techniques which look for broader patterns by correlating the run length compressed data across a second dimension, typically by comparing contiguous adjacent scan line data and coding the difference.

Electronic document deskewing is an essential preprocessing capability necessary to enable further document processing of a digitized paper-based form. The present invention provides a fast and accurate method for detecting skew angle of the form image.

To correct a skewed image requires two processing steps: first, establishing the amount of skew and then deskewing of the image by using skew correction. The skew of a document is usually defined by the orientation of the side boundaries of the document and/or the straight lines contained in the image. Secondly, for a form document, it is predominated by straight lines which are the basic element for constructing a form or tables and the orientation of straight lines which help indicate the status of the form skew. Skew of the form image can be implied when a majority of straight lines deviate from either a horizontal or vertical direction.

To detect the orientation of straight boundaries in the past, a Hough transform was used. A direct application of this transform to a bitmap document for line detection has a major disadvantage. It requires extensive computation to make a histogram in parametric space (ρ,θ) due to the large number of pixels. This can be prohibitive for practical applications. Disclosure of the Invention The present invention combines document contour vectorization with the use of a modified Hough transform for the fast detection of a skew angle of a digitized form image. Skew correction is performed by a matrix multiplication. Brief Description of the Drawings

Figures 1A-1C show a pixel neighborhood and the pixel search order;

Figure 2A illustrates graphically the linear approximation of a polygon which results in a number of short almost coincident vectors that are replaced by a single vector in their place;

Figure 2B illustrates how the deviation from a substitute vector is diminished; Figure 3 is a flow diagram for the detection of skew angle;

Figure 4A is a graphical example showing three points on a form line in Cartisean space; Figure 4B is a graphical example of sinusoidal curves in parametric space corresponding to three points on the line in Figure 4A;

Figures 5A-C illustrate graphically the use of histograms in peak detection; and

Figures 6A-D illustrate the transformation and skew correction for the document form. Modes of Carrying Out the Invention

The present invention uses a polygon-based method which overcomes many of the limitations associated with bitmap techniques mentioned earlier. Contour vectorization converts a digitized document into a collection of polygons. This conversion requires contour pixel tracing followed by piecewise linear approximation of the contour pixels. The contour pixel tracing operation starts by scanning pixels in a horizontal direction. Whenever a transition from 0 to 1 is detected, it traces the contour in a rotational order until it returns to the starting point for the completion of a contour. The rotational order that the search is performed is illustrated in Figure 1. The pixel p, , is a transition which is detected by scanning the document page from left to right. Using a three by three grid shown in Fig. 1C with the center pixel being p. the first transition and looking in the specified search order 1-8 of Fig. 1C until the next transition is located. In this instance, ρ_? was located in cell number 3 of the three by three grid. That pixel was located using the rule of adding a value of four to the previous direction using module 8 arithmetic, and adding one to the result. p_ in Fig, IB is now the central pixel in the three by three grid in the same search order thus locating the transition in cell 3. The next pixel in the search starting from pixel p_ in direction 8. The process is repeated until a closed contour is completed. After completion of a contour tracing, scanning resumes to find the next transition and then traces the next contour. This process is repeated until the last contour has been completed. Piecewise Linear Approximation

The piecewise linear approximation process converts a sequence of contour pixels into a sequence of vectors by merging colinear pixels. The sequence of vectors forming a closed boundary becomes a polygon as shown in Figure 2A The piecewise linear approximation process is modified somewhat by imposing an inner product constraint which allows for the detection of sharp corners during iteration. Sequentially scan the contour pixel and merge the consecutive pixels which lie on the same straight line into vectors. This forms a polygon which is composed of a sequence of short vectors as shown in Figure 2A The vertices of a polygon are denoted by (V , V₂,

V3-...Vl....V ). Calculation of normalized inner product of any two consecutive vectors (e.g. V. and v_i+1) where

I tl l'i i I and

-1 ≤ ι _i ≤ 1

By considering any three consecutive points vl.-l,, vl., vl.+l, and vl.-l,, vl.+l. form the sides of a triangle . ,, v., v. . The line segment v. _, v. _ is the base of Δ - v.__η, vl., vl.+l, . The height of Δ vl.-l, , vι.,' v. . serves as the deviation for approximating the series of V. .,, V. and v., v. ., to v. ., vl.+l.. If the deviation is smaller than a predetermined threshold (ε) and I. is greater than a predetermined negative value the approximator described above is applied. Otherwise, the part v. is kept and the next two consecutive segments are exposed for linear approximation. ik In Fig. 2b vectors v. ₁ v. and \. ¹~-^{L 1} v. v. . are shown if the values for d which is the deviation from a replacement vector v. . v. _ is below a given value, the replacement will be made. However, in the event d is above a predetermined value, the original vector will be preserved. The value of d is given by: ύ - «ι<^«ι+ι - ^yι-ι> - ^yι⁽*ι+ι^* «ι-ι* ^{+ (y}ι-ι^xι+ι - *ι-ι ^

A i*i - ^xi-ι^}* ^{+ ljr}i« ^" ι-ι^,if

5 Accordingly, the sharp corner preservation is accomplished by evaluating normalized inner product values of each of two consecutive vectors during iteration and skipping over the segment merging operator when the value is smaller than a negative

10 threshold. The negative threshold value is selected because the inner product of edge segments at a sharp corner must be a negative value. Thus, the smaller the normalized value the sharper the corner it indicates. For this particular application, the

15 threshold is set a (-0.5) in radians. By incorporating this constraint in a piecewise linear approximation, it has been found that the process preserves sharp turning acute corners while smoothing out noisy short segments. It should be

20 noted that this capability is particularly critical when linear approximation is applied to line-like objects. It should be noted that during operation of the piecewise linear approximation algorithm, the smoothing threshold is started with one and

25 incremented up to the predetermined value while iteration goes along in order to minimize the distorted results obtained in linear approximation. When completed, the contour vectorization process converts a bit map image into a collection

30 of simple polygons. The polygon representation allows for the extraction of the straight line orientation conducted in a vector domain requiring less data.

After the application of contour

35 vectorization, in general, a form image will produce a number of contour polygons of widely varying sizes. A collection of closed polygons is obtained which represent object contour components (either inner or outer) . The larger polygons represent the larger graphic outlines (contours) of the image. The graphic outlines can be boundaries of frames or tables in the image. By applying a size filter to the collected polygons, the larger polygons which represent graphic boundaries are extracted for use in line angle detection. In a form image, these large graphical contour components are composed of straight boundary lines. Therefore, the skew angle detection of a form turns into the detection of the orientation of straight lines. The polygon vectors associated with these graphic boundaries are inputted to a modified Hough transform for detection of straight lines. The Hough transform technique of mappiny image points into a parametric domain where image structure can be easily recognized and is commonly used for straight line detection. When this line to point transformation is applied to image points, it can be used to detect image points that lie along a given straight line. The modified version developed for this application uses the center coordinates (x,y) of each vector in the extraction of large graphical contour components. Furthermore, the angle and length of each vector are computed from the coordinates of the vector end points as prior information. Transformation of the center coordinates (x,y) of the vectors incorporated with the vector angles into the parametric (ρ,Θ) space, the speed to make a histogram is significantly improved, over application of the standard Hough transform in a bitmap document.

The Hough transform for a point (x,y) in Cartisean space is given by: p, = x cos Θ, + y sin Θ, where p is the perpendicular distance from the origin, as shown in Figure 4A and θ is the inclination in degrees of that line. Thus, any line in image space is described by a point in the parametric space (p,Θ). Similarly, a point in Cartisean space (x,y) corresponds to a curve in parametric space. The parametric domain curves that corresponding to colinear points in the image space intersect at a common (ρ,Θ) point. The set of points, of which image points can be connected by a straight line, will produce a count in the Hough transform domain of magnitude N at the position (ρ»Θ). where (ρ,Θ) describe the connecting lines.

In practice, the Hough transform of a point (x. , y. ) is performed by computing p from the above equation for all n values of Θ, into which Θ is quantized in m intervals of width Δp. In this way, a quantized sinusoidal curve is obtained and along the quantized curve each cell is incremented an equal amount. This procedure is repeated for all points. Colinear points in the image show up as peaks in the parametric (p, Θ) space.

The modified Hough transform used to extract straight lines takes the center points (x. , y. ) of vectors in the extracted graphic contour polygons as the point to be transformed points. This will detect sets of vectors that lie along a straight line, and it is the orientation of these detected straight lines that are used for the defining of a form document's skew angle. Modified Houαh Transform

1. Read a previously extracted large polygon. 2. Calculate the maximum p which is defined by equation ^ W 2) + (H/2) where W=width of the encompassing rectangle and H=height of the encompassing rectangle. Next, set the origin of the point in Cartisean space as the center point of the encompassing rectangle of the polygon.

3. Quantize the value p into m intervals of width Δp, and sample the value of Θ every

ΔΘ in the range of 0° to 360°.

4. Calculate the center point coordinate (x., y-), vector length and angle (Θ.) of the vector.

5. Using the center coordinates (x., y.) and the angle (Θ) of the vector (p), can be computed according to Hough transform equation. Note that this is a point-to-point mapping rather than a point-to-curve mapping described previously.

6. Add the value of vector length at the coordinates (p. , Θ. ) in the histogram.

7. Repeated step (4) - (6) until last vector of the polygon has been processed.

8. Peak detection from the transformed domain which is described in detecting skew angle below the peak Θ and is defined as the skew

P angle of the polygon. 9. Repeat step (1) - (8) until last polygon of the large group has been processed.

10. The document form skew angle is the average of the skew angles collected from all of the large polygons. There are three advantages in using the algorithm set forth above: (I) There is a tremendous reduction in data points resulting from the vectorization process. This results in a substantial reduction in the computation loop in computing the Hough transform. (II) The use of vector angular information, the transformation of data points from Cartisean space (x,y) to parametric space (p,Θ) is a one-to-one mapping instead of one-to-multiple mappings as in the standard equation. This greatly reduces the number of processing steps in computing the accumulator array. (Ill) Use of a weighted accumulator can greatly enhance the peaks which correspond to more reliable long vectors and de-emphasize the noisy short vectors. This significantly improves the detectability of the peaks in parametric space (p,θ). Detection of Skew Angle

If a form document is skewed in a particular orientation, the value of Θ is determined when all possible values of p are scanned and yield clusters of high peaks in the accumulation array (histogram Fig. 5A) . The values of Θ associated with the clusters of peaks indicate the potential orientation of dominant contour lines which is defined as the deskew angle of the form.

The detection of valid peaks is described by an example as shown in Figures 5A-C. The example assumes that three large graphical polygons are extracted. The modified Hough transform yields three accumulator arrays corresponding to the three polygons 1, 2, and 3, respectively. By scanning all possible values of p, the majority of peaks are found at p,, p_ and p₃ corresponding to -li¬ the polygons 1, 2, and 3, respectively as shown in Figures 5A-C. The values of Θ associated with the cluster of peaks indicate the orientations of major contour lines. To validate the true peak which reflects the skew angle of a form, the peak is required to meet two criteria: first, the peak value must exceed a global threshold; secondly, when the Θ value associated with the peak is added to 180°, a similar cluster of local peaks should be found. The first requirement is to ignore short segments and keep longer line segments for peak detection. The longer the segments, the more reliable the data will be. The second requirement is to avoid the false line detection which may result from slanted lines. A pair of anti-parallel vectors confirms that they are contour lines of a skewed rectangular-like box. Imposing both of these restrictions to the peaks shown in the example, only the peaks in polygon 1 meet both the criteria. The polygons 2 and 3 fail to meet both the criteria due to the peak values being smaller than the threshold or the corresponding pair is not to be found. In order to obtain better estimation of the actual skew angle, the first few highest and qualified peaks are collected and the mean of the Θ angles with the collected peaks is taken as the skew angle of the form. Note that the working range for Θ in peaks detection is confined in 60° ≤ Θ ≤ 120° and 240° ≤ Θ ≤ 300°. The working range of Θ is dependent on the expected maximum skew angle of a form to be detected. The range of Θ defined above assumes a maximum of 30° for the skew angle of the form to be detected. Document Skew Correction After the skew angle has been determined, the document skew correction on a vectorized document uses geometrical transformation that both translates and rotates vertices of each polygon by the matrix operation shown in the following equation:

wherein t. = X _.L(1-cosθ5) + Y, sin Θ. and t2_ = Y1, (1-cosΘS) - X1. sin Θ_βS

The computational sequence is illustrated in Figures 6A-D in which the vertice of each of the polygons is rotated about the center point of the document p. The transformation process comprises a three step sequence.

The first step performs the translation of the vertice of each of the polygons so that the center point p is at the origin as shown in Figure

6B. The second step rotates the vertices of polygons with a degree of Θ . The result of such a rotation is shown in Figure 6C. Figure 6D is the result of the third step that translates such that the point at the origin returns to the center of the document. The (Xd, Yd,) and (Xo, Yo) are coordinates of vertice of a polygon before and after transformation, respectively. The determined skew angle is Θ and the (X, Y. ) is the coordinate of the center point P. Flow Chart

A paper document is scanned and digitized in step 10 to convert the document into a digital image. In step 12, a thresholding operation is applied to each pixel of the digitized image.

This produces a binary (bitmap) image. In step 14, an object contour following operation is used to extract edge pixels of objects (i.e. outlines of objects) in the bitmap image. A linear approximation operation is applied next in step 16 for merging colinear contour pixels into straight segments. This results in a collection of polygons. Each polygon represents either an inner or outer contour of an object. In step 18, polygon bounding is performed which calculates the size (width and height) of a bounding polygon by subtracting the extreme coordinates of that polygon in both the horizontal and vertical directions. In step 20, a predetermined size threshold value is applied to the collection of polygons. This results in two sets of polygons. The ones collected in step 22 are a collection of small polygons. The collection established in step 24 are associated with large graphic contours. Next, a modified Hough transform is applied to the polygon vectors in step 26. Use of this transform results in mapping the center coordinates of polygon vectors into a two dimensional accumulator array in a parametric domain which is easier for straight line detection. In the next step 28, the accumulator array is scanned to locate the highest peak in the parametric array. Step 30, the final step, the angular value of the highest peaks are read and that angular value is defined as the skew angle of the form document. Advantages and Industrial Applicability

The present invention is useful in computer based systems that provide automated analysis and interpretation of paper-based documents. The present invention uses the geometrical spatial relationship of the contours and a modified Hough transform for the fast detection of any skew angle of a digitized form image and the skew correction is then performed. The present invention has advantages over previously applied bitmap techniques in accuracy, robustness, efficiency of data structure, storage and document analysis. Accordingly, the present invention is more appropriate in determining the skew angle of forms in the preprocessing steps of document analysis and classification.

Claims

WHAT IS CLAIMED IS:

1. A method for reliably determining the skew angle of a digitized form document image characterized by the steps of: thresholding the digitized image to produce a binary image; use contour vectorization to convert the binary image into a collection of closed polygons formed by a series of vectors; calculate the width and height of an encompassing rectangle for each polygon; establish a threshold using a either the height or width of the rectangle to separate all the polygons into two categories with a first category containing small polygons having either heights or widths smaller than a predetermined value and the second category of larger polygons wherein either the height or width is greater than a predetermined second value; transforming the vector data for each polygon in the second category of polygons into the parametric domain employing a modified Hough transform resulting in a histogram of (p, Θ) for each polygon; detecting peaks in the histogram in the parametric domain using a predetermined third threshold; determining the angular value Θ of the peak values p above said third threshold; determining if any angular values Θ, have additional corresponding peaks at Θ. + 180° and if so determined, average all values of Θ. so determined and the average value Θ, will be defined as the skew angle of the form document.

2. The method of determining skew angle as set forth in Claim 1 wherein said contour vectorization is comprised of the steps of contour pixel tracing and piecewise linear approximation.

3. The method of determining skew angle as set forth in Claim 2 wherein the center ccordinates of the rectangle that encompasses the polygon is used as the origin of the Cartisean coordinates.

4. The method of determining skew angle as set forth in Claim 3 wherein said modified Hough transform includes the steps of: a) determining the center coordinates of each vector in a given polygon from the second category of large polygons; b) calculating the length and angle (Θ) of each vector; c) using the center coordinates and the angle of each vector in each polygon to determine the value of p in parametric space, using p - x cos Θ + y sin Θ and plot the histogram of (p, Θ) using vector length for each vector in said polygon; d) include on said plot of the histogram all vector lengths for all vectors in a polygon in a cumulative fashion; e) detecting the peaks that exist at

Θ, in the parametric domain and determine if corresponding peaks exist at Θ, + 180 and f) repeat the steps a-e until all the large size polygons in the second cetegory have been processed.

5. The method of determining skew angle as set forth in Claim 4 wherein p is the length of the line from the origin normal to the vector in the Cartisean coordinate system.

6. A polygon based method for determining the skew angle of a digitized form document image comprising the steps of: a) thresholding the digitized image to produce a binary image; b) use contour vectorization to convert the binary image into a collection of closed polygons formed by a series of vectors; c) calculate the width and height for an encompassing rectangle that encompasses the polygon; d) establish a threshold to separate all the polygons into two categories with the first category including smaller polygons that have either the height or width of an encompassing rectangle less than a predetermined value and the second category containing larger polygons haivng either the height or width of an encompassing rectangle larger than a predetermined value; e) determining the center coordinate of each vector in each polygon from the second category of larger polygons; f) determine the length and angle of each vector; g) using said center coordinates and said angle of each vector in each polygon to determine the value of p in parametric space using p = X cos Θ + Y sin Θ and plot the histogram of (p, Θ) using the vector length for each vector in said polygon; h) include on the plot of the histogram all vector lengths for all vectors in each polygon in a cumulative fashion; i) detect the peaks that exist in the parametric domain and determine a corresponding value for Θ; j) repeat the steps e-i until all the larger size polygons having heights or widths of an encompassing rectangle greater than said predetermined value have been processed; k) determine the angular values of the peaks above a predetermined threshold; and

1) determine if any of the specifically determined angular values Θ, have corresponding at Θ.. + 180° and if so determined define Θ, as the skew angle of the form document.