CA2182514A1 - Detection of mutation by resolvase cleavage - Google Patents

Abstract

A system, apparatus and method for testing the functional components of an electronic digital imaging system is described. The system includes apparatus for image acquisition, storage, display, communication and printing. The system relies on a closed loop analysis to test system components by measuring a set of statistical image quality metrics. The expected set of statistics are in the form of special purpose features stored as a data set representative of an expected reference object. The closed loop analysis measures, for example, the quality of the printing component of the system by ouputting a copy of the expected reference image, using the acquisition component to input the copy of the excepted reference image, and then comparing the statistics against threshold values representative of an ideally operating component. The comparison of statistics against the threshold values provides a go/no-go measure of component performance and can indicate sources of system degradation. The image quality metrics is known as a metrics set which contains a comprehensive set of tests regarding primary modes of image quality degradation: pixel value integrity, pixel location integrity (geometric distortion) and spatial resolution. The metrics set allows for the automatic execution of a large number of image quality measurements and a reduction of the resulting test data to a manageable result so that a simple go/no go result may be given to the system operator. In addition, the metrics set is useable in indicating and locating system faults based upon the analysis data for field service personnel.

Description

W095131869 2182~14 r~ ~ c ~
AU'I~A~ED II\~AOE QUAI~'I Y a~[ROL
FIELD OF THE INVENT~ON
The present invention relates to o,uality assurance and control of acquired and stored imagery and in particular the present invention relates to amethod and apparatus for ~ lly insuring image quality in a medical imaging system.
BACKGROUND OF THE INV~T~ON
Electronic digital imaging systems are known in the art in which images are digitized and stored as digital pixel elements in a computer based system. The images can then be retrieved and displayed on display 15 " ~ ";,~ for later use. An example of this type system is the medical field where x-ray images, computed Lulllo~a~ / (CT~ scan images, magnetic resonance imaging (MRI) data, ultrasound data and the like may be digitized and the images can be stored and retrieved from a mass storage device. By cor~necting a graphics monitor or a plurality of monitors to the system, a 20 medical ~ ,. " .~ . can retrieve the images on such a monitor on demand.
Various types of quality control have been applied to electrorlic digital imaging systems. For example, in U.S. Patent No. 4,939,581 to Shalit, an attempt is made to measure the quality of gray-scale of a video monitor screen by placing a gray- scale test pattern on the CRT screen and measuring 25 features of the test pattern usmg a ~llv~ulll~,t~,l. The gray-scale test pat~n is then photographed using an electronic calnera, a hard copy film is produced from the electronic camera image and a .Irl 1` 'I 111 11-1rl reading is taken of the hard copy. ~he results of the ,l. .I~lllllllrlrl readings are then used to adjust the electronic camera irnage for ideal luminance to ~ t~ on a pixel-by-30 pixel basis to produce a gray-scale which matches the developed film.
The system of the Shalit patent only addresses control of a single image quality aspect: the gray-scale accuracy of hard copies as ~ Lio.~s of the CRT image. In addition, the system does not treat the case of matching a CRT display to film and does not address the problem of WO95/31869 - -= 2 21 82514 r~ - ~ ~
the original CRT image qualit~. Since the Shalit system is only designed to match a hard copy to a CRT device, an objective display of image quality is ruled out. In order to reproduce accurate images, an objective standard needs to be applied to CRT matching and to locate sourccs of ~Ir~
5 throughout all points within an electronic imaging system, not just through the CRT display device. ~ithout any form of system- wide ,1~ ;, ." analysis, cu~ ullcllb of the Shalit system may affect the ove~ll image quality of the video monitor causing the x-ray to match the video morlitor for a ~holly inaccurate display.
In U.S. Patent No. 5,115,æ9 to Shalit, a method and system in video image reproduction is described using a gray-scale test pattern on CRT
screens to compare two or more video screens. The objective of this system is to achieve a CRT-to-CRT match without regard to an objective set of criteria for CRT alignment. One of the drawbæks of the Shalit system is that 15 it will match a good CRT with a bad CRT &splay such that both CRT
displays will produce imagery only up to the capabilities of the poorest of thc tu~ screens. There is 110 ability to match the CRTs to any objective criteria to not only align the C~Ts to produce the best image quality possible from that particular CRT but also to locate CRTs operating below a minimum 20 æceptable threshold. In addition, the Shalit system merely deals uith a single image qu~lity aspect: tlle gray-scale accuracy between CRTs. There is a need in the art therefore to contro~ the image quality m rrlany categories ~;""~ "~ ly such as pixel value, geometric and spæial resolution ~l~L~
In the paper entitled "Quality MonitoriLng of Soft-Copy Displays for Medical R~diu~ y" by Reiker et al., published in the Journal of Digital Imaging, August 1992, luminance ~ lb from a plurality of CRT
screens within a hospital or irnaging center are used to compile a database of lum~nance ;..rl",..~,;.,.. A low cost ~IIULUI--~LCI instrumcnt with an RS-232 30 interface allûws the device to be connected to CRT screens t~roughout a hospital to measure the luminance values on e~ery display station. A software method and procedure for displaying test images of single valued luminance wo 95/ ~1869 - ` 2 1 ~ 2 5 1 4 r~-~u~
info~rn~tion allows a software program to generate luminance responr,e curves for eve.y display device within the ul~li~liull. This provides a system for quality control of the CRT displays. The author has der,cribed this system as 'oeing necessary to calibrate the CRTs to conform with a standard luminance S curve by adjustmg brightness and contrast controls of the CRT stations. The `~IIU~L~UII~I~ of this system, however, are that the lack of image quality control ~3roughout the entire electronic i ,maging system may result in erroneous ~'j being made to CRTs at the various locations throughout the network.
Iû There is a need in the art to control the qua'lity of the image within an electronic digital imaging ~l~vilulll~ which is uncatisfied by existing systems. Present electronic digital imaging systems lack the ability totest, maintain and ensure the irltegrity of the quality of the image throughout all stages of the system including the stages of ~-icition Ir,~
15 display and hard copy generation. There is also a need in the art to measure and report system 1~ r.""~ to a system operator in a user-friendly manner such as a simple go/no-go mdicator of acceptable system prl l'o~ r Further, there is a need in the art for remote diagnostic testing, predictive arld ~ ,v~ L;lLiv~ servicing and computer assisted fault isolation of image 20 11r~ lll and system failures. There ic also a need in the art to provide system 1,. . r. " ",~ testing by using the wlll~oll~ b of the system to test themselves without the need for a field service person to carry expensive test and calibration equipment to the site. The present invention solve~c these and other problems which will be recognized by those skilled in the art upon 25 reading and ll.l.l. .~lrl.ll;ll~ the following r~r~ifi~tir)n SUMMARY OF THE INVl~ON
The present invention is a system, apparatus and method for ~-~fomRti~lly testing the functional ~ of an electrorlic digital 3û imaging system. The system includes apparatus for image A~.icit~nn, storage, display, ~1~1111.11111;l '~1 ;l ll l arld printing. The present invention relies on a closed-loop computer analysis to test system ~UIll~(l~llL~ by ~--tflm~ti~lly WOgS/31869 ~ 1 825 1 4 ~ o ~ ~
measuring a set of imagf quality metrics derived from an analysis of a kno~n set of features contained in special rcference images. The metrics are compared to values obtained from a prior~ ;"r... ",~l;.... about tbe expccted ~.~ . r~" " IAI If r of the systf m component under test. Ihe closed loop analysis 5 measures, for example, the quality of the printing component of the system by using the acf~uisition component to measu}e the output of a printing component where the input is a refe}cnce image and then t~sting the statistical metrics obtained from a~l acquired sample image to locate sources of system The image quality metrics are known as a metrics set which contains a Wl~l,Ul~llCl~ c set of tests regarding primary modes of image quality llr~ pixel value integrity, pixel location integrit~ (geometric distortion) and spatial resolution. The metrics set allows for the automatic execution of a large nulnber of image quality lll~UI~ and a reduction of the resulting test data to a ",~ ,~f,,~ result so that a simple go/no go result may be given to ~he system operator. In addition, the metrics set is useable in indicating amd locating system faults based upon the analysis data for field service person~el.
BRIEF DESC~RIPTION OF THE DRAWINGS
In the drawings, where like numf~rals describe like throu~hout the several views, Figure I is a block diagram of the ~ull~l~u~ of a ~f nPr~li7PA
electronic imaging system;
Figure~ is a block diagram of a particular i"-~ .". .~ n Of an electrorlic imaging system used in a medical t;llVilUIIIII,II~, Figure 3 is a process flow diagram for image quality metrics for the present invention applied to an electronic imaging system;
Figure ~A and 4B are block dia~ams of the automated image 30 quality control system of the present invention applied to an electronic imagjng system;
.

wo 95131869 ~ . ~, 2 1 ~3 2 5 1 4 ~ J r o FiglAre 5 is an ex?Ample of a reference image used to test the geometric accuracy of the electronic imaging system;
Figure 6 is an example of the stepwedge patterrA for the reference irnage to test gray scale;
Figure 7 is a block diagram of a portion of the autornated image quality control software CVI-~uIl~ b of the preferred ~ lrllI Of the present inverAtion showing the LiJTs;
Figure 8 is a graphicaA Ie~ ll aA~iOIl of a typicaA density response curve for the Lumisys model LS150;
Figure 9 is a graphical I~lc~ lidlioll of the calibrateA' values for LUT A;
Figure 10 is a graphical ~ ,lldlioll of the caAibrate~A vaAues for LUT B;
Figure 11 is a gr. phicaA l~l~,lI~ii.~ll of the calibrated values 15 for Ll~T C;
Figure 12 is a block diagr;3m of the automated image quality control software IA~ of the preferred c,IlLA l of the present invention;
Figure 13 is a state diagram of tne procedures of the automated 20 image quality control process of the preferred ~".~.o~l~.". ..1 of the present invention;
Figure 14A is a flow ch~Art showing the ~A II I I~ I I of metrics and message reporting for each procedure of Figure 13 for the automated quaAity control process of the preferred . .. "l ~. i;, . .~ of the present invention;
Figure 14B is a flow chart showing the test, log, and status update (TLS~I) re,porting procedure of Figure 14A;
Figure 14C is a flow chart showing the result, messaging and next state processing (RMNS) pr~v~cedure of Figure 14A;
Figure 15 is a flow cha~t for the film digitizer calibration 30 procedure of Figure 13;
Figure 16A and 16B are flow charts for the film digitizer density test procedure of Figure 13;

WO 9~/31869 2 1 8 2 5 1 4 P~ J.,,''O: / ~
Figure 17A and 17B are flow charts for the film digitizer geometric test procedure of F;gure 13;
Figure IBA and 18B are flow charts for the laser imager calibration procedure of Figure 13;
Figure l,C`A and 19B are flow charts for the laser imager densit,v test procedure of Figure 13;
Figure 2DA and 20B are flow charts for the laser imager - geometric test procedure of Figure 13;
Figure 2] is a flow chart for the image storage and 0 If ~ testprocedure of Figure 13;
Figure 27 is a state diagram of the procedures of the automated image quality control p~ocess for the image review station of the preferred r~ J-1; l l If ~ I~ of the present ir~vention;
Figure 23 is a flow chart for the CRT calibration procedure of 15 Figure 22;
Figure 2~ is a flow chart for the CRT test procedure of Figure 22; and Figure 2S is a flow chart for the CRT monitor matching procedure of Figure 22.
DETAILED DE~;CRIPTION OF THE PREFERRED E~MBODIM~NT
In the following detailed description of preferred r,~ lOll; ~ I Irl ll reference is made to acw ~ y--,g drawings which form a part hereof, and in which is shown, by way of illustration, specific preferred ~ ,o-l.."~ in 25 which the invention may be practiced These, ' ' are df scribed in sufficient detail to ena~le those skilled in the art to practice the invention and it is to be understood ~hat other cll~l~oLll~ lL~ may be utilized and that structural, logical and electrical changes may be rnade Yithout departing from spirit and the scope of the present invention. The follo~Ying detailed 30 description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is deflned only by the appended clair~s.

WO95/31869 ~ 2 1 825 1 4 P~ c SYstpm OverviP,w Figure I describes a bæic electronic digital imaging system.
This system consists of a number of bæic ~ which fall into general categories. The image acquisition device 10 shown in Figure I includes such S devices as film digiti_ers, scanners and other selected modalities. The image acquisition devices 10 may be a number of medical scanning modalities such æ ultrasound, ~1, CT, digital radiography, and digiti_ed ~UIIVI~ iUllal x-ray film. The image display device 20 may be some type of CRT device such æ
a high resolution computer color or Illullo~ ul~l., monitor. The image storage 10 device 30 is typically a mæs storage device such as disk storage. The image hard copy device 40 of Figure 1 would typically be a læer printer for paper copies of the images, a pen plotter, a printing plate output, or læer imaged onto film.
The illllll,..., 1~ of the ~PrAli7P~I electronic i}naging system 15 of Figure I may be used in a variety of rndustries in which irnage Artrli~itinn, storage and retrieval is required. For exAmple, rn the publishing industry, the acquisition device 10 may be a document scanner. The hard copy device 40 in the publishing industry could be læer printers or perhaps even the output going to printing plates for offset printers or the like. The display device 2û
20 used in tbe publishing industry may be used in the layup and production process.
The preferred embodiment of the present invention includes software methods for controlling the image quality witbin the electronic imaging erlvironment shown on Figure 1. The present invention utilizes a set 25 of objective image quality metrics computed ~ ly from a number of sampled images. This quality control system provides consistent and thorough ",~",~" of the quality of the image throughout the electronic imaging system, includmg the stages of A~1i~ition, 1,,."~",.~l..", display, and b~rd copy generation. Further, the preferred ~",1~ ' of the present invention 30 describes methods of remote diagnostic checking, predictive preventative ser~icing and computer assisted fault isolation.

wogs/3ls6s 21 825t 4 r~ s~c ~~~
For purposes of example, but not by limitation, the preferred Pmho(lirnrnt of the present invention is described in conjlmrtion with a medical imaging and picture archiving system where each of the four blocks of Figure I is typically part of a large network in which a plurality of each ofS the four ~ h~ lt~ is duplicated and illt~W~ ~L~I by an i.-~ w..ll~-,Lion network 50. In other words, there are a pluralit~ of irnage acquisition devices 10 and the images may be stored at a plurality of image storage device sites 30 A nurnber of image display devices may be utilized rn such places as intensive care units (ICU), cardiac care units (CCU), emergency rooms (ER), 10 in ~ hll..t~ or doctors' offices. The irnages are typically stored at a central location in an image storage device 30 and can be printed through the use of image hard copy device~, 40 at a number of sites in a hospital or clinic to make duplicate x-rays, laser prints, film, etc. Those skilled in the art will readily recogniæ that the preserlt inverltion is not limited to medical imaging 15 systems but is in fact applicable to all types of electronic imaging systems. Figure 2 is a block diagram describing the prefelred embodiment of the pres~nt invention used in a medical environment. The system of Figure 2 is an electronic digital imaging systern known to thc inventors in this case as a PACS (Picture Archiving and Cll,llll,~
20 System Illdlluf~L~cd b~ Minnesota Mining and ~~ Ul ill~ Company of St. Paul, Minnesota. Tllis example shows atl electronic imaging system in which an ICU, CCU and ER areas are equipped with image review stations for displaying medical image~y. The image review stations are cormected by an illL~Iwllll~iul~ netuork to irnage storage devices, image hard copy devices 25 and image ac~uisition devices.
The i~t~ u~--~lion network 50 may be one of any number of varieties of local or wide are~ networks using ~f rhnr,lr,~j~ e such as ethernet.
FDDI, ATM, token rin~ etc. As shown in Figure 2, an image selver system with local ~ of the network is a part of the irnage storage device 30 30 of Figure 1. This would comprise a local control CPU and a large disk storage subsystem 201 fûr storing the imagery and data associated with the images which may be in the form of a netu~ork file server. A local monitor W095131869 ' - -' 2 1 ~25 1 4 P~ o 202 may be equipped ~vith a local printer 203 and a local disk subsystem 204.
The control CPU 201 of the image storage device 30 may also be connected by remote connection 205 through modem 206 to other lo~ations to obtain and trarlsfer additional imagery and data associated thereto.
S By way of illll~hon; but not by way of limitation, the image hard copy device 40 may be a laser irnager 207 coMected locally to the control CPU 201 of the image storage device 30. The laser imager 207 may also be coMected through illt. lW~ v Liu - neh~ork 50 as a remote node on the nehNork. A variety of outputs may be obtainable from the image hard copy 10 device 40, but in the preferred ~ l,.J l""r ~l of the present invention, a film hard copy 208 is obtained to reproduce medical image~y.
Image acquisition device 10 may also be cor~nected as a node on the neh,vork through a control CPU 209. Digitizer 210 accepts, for example, original x-ray films 211 for input into the system. The original x-15 ray film 211 is digitized by digitizer 210 and stored locally in control CPU209. The acquired irnage is then transferred via illt~lwlln~liu., network 50 to the image storage device 30 where it is stored on the disk subsystem 204 under conhrol of the control CPU 201.
All image data stored on disk subsystem 204 is available via 20 ~llL~IwlL~ iull network 50 at the varjous image display devices 20. As ..~,,Il,li~it~A in Figure 2, these locations may be an ICU, a CCU, or an ER
Each of these locations has an individual irnage review station 212a-c cormected to il.l~lwlllle.,Liull network 50 via local conhrol CPUs 213a-c. The image review stations may corlsist of single screen or double screen display 25 devices. The double screen display device may be a double-width CRT for displaying hA/o images side-by-side or it may consist of t~o physical CRTs placed side-by-side for the ~ J~ of images.
Also available on the ill~l-UllllC~,~iUII ne~vork S0 is a patient census ' " r. " " 1~ station 214 consisting of a local computer attached to 30 ~ t;lWllll~jUII nehA/Ork 50.

2 1 8 2 5 1 4 r~ c .

Typical Oper~t;--q of Tm~ Arn~ iti~n St~r~f anfl Di~l~, By way of example, the electrorlic in~ging system of Figure iq a hospita~ CIIVU~JIUIICIIL may be used iq the ICU to care for a patieqt Typically, a portable x-ray is taken of the patient iq the I_U aqd il must be 5 processed to determine such things as correct tube or electrode placemeqt, the presence of l " ~f l~ Y, etc. The electronic imaging system aids the workflow by making the imagery readily available iq an electronic form in the ICU and preveqting film loss that might otherwise occur if the films were returned to the ICU afte~l processiqg in a radiology depattmeqt.
In operation, the x-ray image is first taken and the filrn cassetts will be sent to film processiqg iq a radiology departmeqt. The radiology depattment may be quite a distaqce removed from the ICU. The exposed f Im cassette is processed andl UlUll~d;~.~l.y transferred onto the imaging system through digitizer 210 where it is s~ored iq a central image storage device 30 1~ for vie~ving by the image review station 212a iq the ICU. The key to efficient operation of this system is how quickly that this tumaround can occur.
Another feature of the present system is the efficient high bandwidth capacity of Ul~lW~Ulv_t network 50. Since medical images of high 20 resolution can typically occupy between 4 and 20 megabytes, the transfer rateon network ~0 must ha~e the capacity to deliver multiple images to multiple sites m a near-~;",~ fashion. A goal for imaging medical imaging systems of the type shown in Figure 2 is to deliver an irnage in about two second's time period aftet it has been requested.
A~ltr,m~tf~1 Tm~r f~lAlity Cr,ntrol Ove~iew The auto~nated image quality control system for the electronic imaging system of the l~referred c~ll~di~ of the present invention al t--m~tirlll ly dete~Tnines the ~l r~ of the functional ~ l lrl ll ~ of the 30 electronic imaging system by compiling a set of statistical metrics (Ill~ Uc lll~llL~). The statistics are computed using the pixel values and coordinates of image pixels acquired from reference objects having a number WO 95/31869 ~ : ' `. . 2 1 8 2 5 1 4 P~
Il of ~rP~ i~ features. ~he p~lru -l~l r test consists of a .~, ." ,1.,.. ;~. ., 1 of each metric to one or more thresholds. ~hese thresholds vre ~iPtprminpd by prior e~r~min~til)n or derivation of the metrics computed from a kno~n good component of the same type. If the metrics show that the l~rl r( ll l l l~ of a 5 specific functionvl component is below an acceptable minimum threshold, or outside an acceptable range of values, a simple failure result is reported to a system operator. The specific details of how this is ~ l lr~l is best shown by actval examples.
The choice of metrics is based upon an i(lPntifi~ti-ln of all 10 likely modes of image quality ~ " (described more fully below) using defned features which will display statistically significant chvnges in either its pixel values or ~vUldlll~ when such rlP~tif)n~ occur. The process of locating and extracting feature pixels required for a given metric is performed using established image processing techniques. The image pr~v~cessing is 15 performed using a ., ." ,1.;. ,,.1;. " . of off-the-shelf image processing library functions and custom software for ~ j l" ~ .... of the special statistics.
St-~r(`P~ l-f Tm~ ~P~ til~n The preferred r,ll 11l~llllll. ~11 of the present invention is concerned 20 about a number of possible ~1~ " ,;, 1;~ ab~errations which may be introduced into the viewed, stored or printed digital images as they are processed by the electronic imaging system of Figure 2. These aberratiorls may be due to electronic, electro-l . lr~ 1 or optical faults in any of the functional system l.lllr~ These faults generally manifest themselves as a vle~iull in 2~ one or more of three image quality categories: pixel value integrity, geometric acculacy and spatial resolution.
By way of example, during the image acquisition process at the image acquisition device 10, the most common source of problems is .1lll1,,lll;ll,..l1~ within the optical path. Since most film digitizers use either 30 scanned laser or a line-scan CCD as a me~ns for ,1. .I~ lr~ ",;",.lir)n within the optics will learvls to a density error at one more fixed positions producing "streaks" in the cross scan direction. The same "streaking" problem WO95131869 ~: = 21 8251 4 P~,l/u..._.'C: I
arises with the hardcop~ device printing device 40 although the source of thc streaking can also include ~ l.;, IA~ within the film processor.
Other for~ns of pixel ~ alue distortion rnclude electronics calibration drift, aging of the image sensor (such as the rhntnmlllfirli~ tube 5 in the film digitiær 210), random errors in any serial ~'1~11111111.1;. ~I;nn and ~ wulLIg paths and "stuck-at" bit faults in bit-parallel data paths. The latter can occur in any of the images memories or host buses such a3 SCSI or other custom bit parallel data paths. Random bit errors may occur in high speed bit-serial lir~s such as cthernet or fiber. Generally, random bit errors leading10 to image quality ~P~(I.~tinn are not detectable with the preferred ~1ll' ' of the present invention since a reference-based technique is used. However, if the statistical probabiEity of these errors is high enough to ensure that therandom bit errors occur during any image " ,~ ;- ", step, then the random errors can be treated as a pseudo-.lr1r, 1, ,;. ,;~l ;. source and are detected using 15 the present metrics approach.
Sources of geometric distortion m the imagery are usually electro-m~h~ni~l These include scarming and film transport 1,, . 1, - ,;~, "
which have non-linear ~elocity 1 ll~. ,,. Ir~ due to bindin& wear, belt slippage, roller .~""1~",;, .,ll;"" and .1~ i~"",~;("" etc. Also, CRT displays are 20 especially prone to scarl non-linearly as a result of electronic aging and component failure. The preferred emh~1imPnt of the present invention is not equipped to detect the geometric distortions in the display devices, as described more fully below.
Loss of spatial resolution in the image is perhaps the most 25 subtle and potentially h~rmful form of dl~gra~l~tinn In a hospital environment, medically important features may be obscured giving risc to the possibility of ",;~ ,n~;~ Loss of spatial resolution can result from all of the faults described abcve. The difficulty in detecting loss of spatial resolution is because the degradation is not striking m its ~ ranr.- rn a 30 given image. The present inveneion provides specialized tests ~vhich are verysensitive to resolution loss to prevent the loss from going undetected during norrnal daily use of the system.

WO 95131869 ~ ' 2 1 8 2 5 ~ 4 r~

Table 1 ~""""A~;,r~ the metrics identified as necessary to assure complete detection of known sources of ~1P~AflAtir~n in an elechronic image processing system. Also listed are the features (described more fully below) used to detect the sources of I~A~ I and the statistics used to 5 quantify the quality of the particular feah~re in the image.
TABLE 1: Summary of Image Quality Mehics, FPAhlreC Anrl StAti!~tirA~ AntifiP~

M~hic l~tle Applicable Feat~e Statistical Componer~s l~ ~ ~E~
Ahsolute pixel FD, CR, Ll Gray scale step Mean, 15 value acculacy wedge standard deviation of absolute error 20 ~ ' FD, CR, Ll, CRT Ciray scale step Linearbest Display wedge fit error response lineanty 25 Cor~ast FD, CR, Ll, CRT Gray scale step Ratio of Resolution wedge standard deviation to mean 30 lnscanl~F FD, CR, LI Vertical bar Histogram patterns min/max.
C~ss-scanl\~F FD, CR, Ll Hor~zontal bar Histogram patterns min/max.

AngularMIF FD, CR, Ll Diagonal bar Histogram patterns min/max.
~scanvelocity FD, CR, 1~ Hor,70ntal bar Run-length 40 Imifonnity mean, standard deviation WO 95/31869 2 t g 2 5 t 4 . ~ C , Table 1: (Corltinuç~
Metric llt~e Al~plicablle FeatDre Statistical ~ ~ r~ ~
Closs-scan FT)~ CR~ Ll Vertical bar Run-length posilion~d jitter pattern mean, standard deviation Laser beann FD, CR, Ll Straight Linear best wobble horizontal edge fit erro~
Sta t-of-scan FD, CR, LI Straight vertical Linear best 15 uniformit~ edge fit error ': ' FD, CR Ll Straight vertical Linearbest ~Lformi~ edge fit error 20 Absolute pL~el FD, CR, Ll Vertical and Run-length size horizontal bar absolute patterns error Pixel aspect FD, CR, Ll Vertical and Run-length 25 ratio horizontal bar relative error patterns l~uge alea FD, CR, LI Gray scalestep Maximum ~iformity wedge standard and peak de~iation Periphely FD, CR, Ll Constant density Peak Imiformity border region deviation from mean deviation Glare Effects FD, CR, LI High contrast Normalized solid box glare area from density histogram St~ak l:~D, CR, 11 Grey scale step Streak count, detection wedge width, y-extent, polarit~

woss/3ls6s . ' -' = 2 1 825 1 4 .~u~ ~c Metric l~Ue Applicable FeahLIe Sta6stical ~ spected Disclrte FD, CR, LI Grey scale step Total anomalies wedge discrete anomaly count Legend: FD = Film Digitizer a~ = Computed Ra lio~ll~
Ll = Laser Imager (Hardcopy) a~= CRT Display Pror~ Flow for Tm~,vt~ f~y ~ lric Cnn~lt~finn Figure 3 is a process flow diagram for Uhe image quality 20 metrics c(lmrllt~tif\n of the preferred hllb~ of the present invention.
Reference features are frrst defmed at 301 which are selected to locate and compute the statistical measures shown in Table 1. The reference features correspond to the features inspected in Table 1. A reference image 303 is ll",~l",. lrll usrng a variety of techniques 302 described more fully below, 25 such as printed circuit board artwork generation tools and Gerber Scientific plotters. The referenoe image or images 303 are then used to generate phantoms or targets as a reference object 305 using selected ,..~.".r.. "" ;"g techniques such as film " ~ " l r .. 1, ,, ;., p techniques 3W if the reference object is a filnL The reference object 305 is then applied to the system to be tested 30 at 306. The ac~uisition of the reference object rnay be tbrough CT, MRI or ultrasound for 3D modalities or by scanning into the electronic imaging system using scanning or digitizing techniques for filnL The rnput modality thus produces a sample image 307 from the reference object 305. Image processing 308 is then performed on the sample images 307 to locate and 35 extract features within known regions of interest. The results is a collection of sampled feature pixels 309 r~ illg selected features which will be used to measure system l~ r(.,."~ Metrics ~ 1iOIl 310 is performed on these selected features to produce a set of features statistics 311 indicating .. . .. . .... . .. . .. . . . ... _ . .. .. .

WO95/31869 21 ~2514 r~u o system ,uclrolll~l,c. Tlle results of these r~mrlf~fir,n~ are then stored in a results file 31æ
The reference object 305 is also used to generate data regarding expected ~ ru~ A~I~;c of the system if all ~ I'i of the system are 5 operating at their normal or peak pr- r~ This data set of known good system ~ llrlll~ 313 is usually generated at the factory where the systems is first assembled. From the l~ r~ of the system for the data set of - known good system ~ lllrlll~ 313, threshold and III~LIICIII~ is performed on the data and a threshold and parameters file is generated based up 10 minimum l,clru~ e values selected for acceptable system operation. Any system ~lrol~ e which falls below these values is considered a system failure. A metrics Wll~ UII 315 is performed at the site each time the automated image quality confrol system is executed to compare the thresholds and parameters file to the feature statistics file to indicate overall system 15 p~:lrulll~l~ and to indicate any failures. The results of the metrics Wl--,uAli~u.. is also stored in the results file 31æ
A simpl~ golno-go result is presented to a system operator for each component bæed upon a ~ 111 of the individual statistical results to pre-flPfPrminpA goocl or adecluate lJr~ r~ )I I I IA- 1~ r limits. The results file 312 20 may also be used by a field techr)ician to locate and identify ~ of the system which need ser~icing.
Al-h m:ltr~l Tm~P (~l~lif~y C--nfrol Pror~ Flow Figure ~A and Figure 4B describe the automated image quality 25 control system as it operates in the electronic imaging system. Startmg in the upper left hand comer of Figure 4A, fwo fypes of reference images are defmed. A digital ref~rence image def~ition 401 is defrned for the testing of the m~illlAfir,n transfer function (MTF) and geometric ~ within the system. Digital reference irnage definition 401 is designed to test 30 geometric accuracy and spatial resolution features in the electronic imaging system and WllC~JUlld~ to testing itemS 4-17 of Table 1. A digital reference WO95/31869 ' ;. 2 1 825 1 4 ,~"1,~ co~ , image definition 402 is defined to test and measure gray-scale L~ Ir~ irC
and ~;u~ lda to testing items 1-3 of Table 1.
The image definition used for the lllt;dau~ lUl~ of m~ ti~n transfer function (MTF) or geometry begins as a data set which descTibes the 5 specific inch-wise geometries of a digital reference image in the form of a definition file 401. Definition file 401 for the geometric reference image specifies the inch-wise locations of bounding rectangles for the structures - shown in Fi~ure 5. These structures may be rectangles 501, horizontal lines 502, vertical Imes 503, diagonal lines 504, horizontal resolution coupons 505, 10 vertical resolution coupons 506, diagonal resolution coupons 507, horizontal single frequency bar 509, vertical single frequency bar 510, etc. The defnition file 401 will typically contain the upper-left and lower-right corner locations for a rectangle. For a line, the defnition file will typically containthe end-point locations and its width in pixels. In the case of a diagonal line,15 the definition file 401 will typically contam the upper-left and lou~er-rightcomer end-point locations. Coupons 505, 506, 507 are located by deflning the upper-left and lower-right comer locations for each coupon. All these location values are m mch-wise cùuld;lldL~ based on absolute comer locations or measured relative to a registration target 508 located ~UIII~;W~ vithin the 20 reference image (usually the center). The definition file 401 may be generated using industry standard graphics tools to produce the definitions in the form of PostScript (Adobe Systems) gtaphics definition files, Gerber ploner output and other drawings programs, to give some examples but not by way of limitation.
The definition file 401 for the geometric reference is loaded into the image se~ver 413 of Figures 4A and 4B. The definition file 401 is stored æ a birlary image of two-valued pixels lc~ t;llLu~g the cre~ifil~tir~nc for an expected image. The location of the structures in the defmition file 401 are converted to X-Y digital coordinates for exact pixel locations bæed 30 on the scan resolution and size selected for this expected image. This expected reference image created from file 401 and stored m image server 413 can later be used to test the precision ~vith which the two valued pixels WO 95/31869 2 1 8 2 5 1 4 F~ll~J.. S/C: I

can be (~ ;"~,";~ r~ ly and other features shown in Table 1.
Expected images ~ alt~l by the definition files 401 and 402 can then b~
retrieved and revie~ved on image review station 413 to test that component of the system (as described more fully belo~v). The expected images ~ d S by the definition files 40 l and 402 can also be retrieved and printed by laser imager 417 to produce printer sample files 420 and 421, respectively, to test that component of the sy~tem (as described more fully belo~v).
Vertical resolution rr, r(" " IAIII ~ of the electronic imaging system can be measured and compared to statistics thresholds using a series of 10 vertical bars 505 of blacl; and white to determine the exact spacing. width, separation and pitch. Also, to measure pixel size within the system, a patte~n 501 may be used ~vhich is perfectly square in the digital reference image as defined in the definition file 401. Thus, if the target 500 has a perfect I inchby 1 rnch square 501, the electronic imaging system will be tested for its 15 abilit~ to reproduce that exact image. For this type of resolution , the digital reference image defined in definition file 401 does not contain ;"li.""~ ", about the absolute density of the image. Metric analysis of system ~Jt;lrUI ll~l~e using these expected definitions is describedmore fully below.
The imag~ used for the Ill~ulrl~ of gray-scale system response also begins as a data set which describes the specific gray-scale intensity and locations of the step wedge reference image in the fomm of a definition file 402. The definition file 402 for the gray-scale reference image specifies the inch-wise locations of the steps and the absolute 12-bit pixel 25 values for the optical density at e~ch step. A diagram of the step wedge referenoe image is shown in Figure 6. This definition file 402 is also loaded into the image server 413 for later use as the ~rifi~tinns of an expected image. Definition file 402 may also be produced in a manner similar to the manner used to generate definition file 401 as described above.
In the gray-scale domain tested by the digital reference image defined in definition fil~ 402, geometry is not as important as the ac~al optical densities. Since it is r~lihl~ly difficult to make a physical test t 3 .i WO95/31869 ~ = = 21 825 t 4 r~l~u~ c target which " '~ tests both gray-scale densit~ and geometric resolution (m~ fion transfer function or geometry), both reference definition files 401 and 402 are needed and two ~ ullLIg reference targets 500 and 600 are needed, ~ Itt ~iv~ly. In testing system p~lrul~
5 regardmg gray-scale using the digital reference defnition file 402, a wide range of optical densities and low levels of noise must be measured in the physical target and hence, the original physical reference 600 must be of ~t~ ly high quality to test the optical density required for most medical x-ray film. ~reri~li7Prl medical x-ray films have optical densities reaching 10 3.7, which is a very, very dense and dark film. Thus, to be able to reproducesuch a large dynamic range of optical densities, the acquisition device such as the film digiti_er must be capable of It:~UlUdUUillg the density scale on a piece of x-ray film To be able to represent the original medical x-ray images in a gray-scale pixel format, a 12-bit pixel is used to adequately represent the 15 dynamic range of the original optical density.
Phy~ir~ f~ renrP Film~
To erlsure the quality of image input retention and display, and to test the l~rl r ." "~"- ~ of system ~1 " . ~l tl ~ j the quality control system of 2û the prefe~red ~ ' of the present invention compares the data regarding the untested system's response in handling "real" or physical reference targets or films to that of a known system usmg the same references. To ~ this, a physical reference film 4û4 is generated from the same definition file 401 that is loaded into image server 413 as an 25 expected image definition. Physical reference film 4ûS is generated from the same defrnition file 402 that is loaded into image server 413 as an expected image definition. The reference fiLms are used when the input modality is a film digiti_er. Different types of targets 404, 4û5 may be used as described below.
Aspeciali_edtestfilm ~ r~l1.,l;"~ process403 isusedto generate physical reference films 404 and 405 which correspond to the example physical reference films 5ûO and 60Q respe~tively, shown in Figs. 5 wo gS/31869 ~ 2 1 8 2 5 1 4 P ~ o and 6, 1~liv~ly. Physical reference film 404 is used to measure system prl r." " IA~ 1~ P for geometric .1~ t~ "~; jr~A and spatial resolution resporlse of the system and physical refererlce film 405 is used to test pixel value integrit~
response of the system. To Arcnmrlich this, physical reference film 404 is S generated as a high qualit~ rhf tr,f~rhi~A film, such as an x-ray target imageof knoun quality to be illpUt as an original x-ray 211 (as shoun in Figure 2) totf~stthel,~.. r.. ".A.. ~( of imageacquisitiondevicelOorother.. ".. l.. r.. l~Of the electronic imaging s~stem. The high quality reference images 404 and 405 include features uhich are serlsitive to as m_ny of the potential sources of .1f ~, ,,,1AI ;- -. . as possible. As described above, the best possible quality control reference images are used as physical targets that mirnic the features to be tested in the system. The physical referf nce targets are constructed to be compatible with the type of acquisition device used in the particular image acquisition system.
For example, in digital l~liO~d~lJIly, the target 404 may be imaged onto lead or gol~ foil and the like, processed using etching processes similar to those used in the printed circuit board industry to produce a target to test for the geometric accuracy in the image acquisition device 10. The target is then used with a digital 1dd;o~Ld~ y cassette to input the sample 20 images uhen the image acquisition device 10 is a digital 1ddiof~lly cassette sca~ner. Als4 for a gray-scale target, conventional metal step wedfA3es can be used as the target 405 which then can be x-rayed usirlg the digital ~ddio~l,y input device as a sample or target of known gri~dient density.
In anothf r example, when the image acquisition devioe 10 is a 25 film di~,itizer 210, geometric reference rmages can be produced by plotting the geometric patterns usin~, high-precision ~""'1"'' ` ~ typically used for printedcrrcuit board artwork gf neration. These geometric plots are then ~,l",l"~ ,fA onto ;u11~ iu11~1 or x-ray film to produce the sample targets 404. For gray-scale targets, ~UII\'~ iUlldl metal step w~edges can also be used 30 as a gray-scale target ~hich then can be x-rayed onto film to produoe an x-ray sample target 405 of known gradient density. Gray-s ale patterns can also be WO 95131869 ' ' '- 2 1 8 2 5 1 4 r~ c ~ /

made using laser imagers or by direct exposure onto conventionaii or x-ray filrn using step and repeat exposure teciL,niques.
For other scanning mod~ ities used as the image acquisition device 10 such as MR~, CT scan or u'itrasound, special puipose tiLiree-5 .1;1 l ,~. ,~;. .l ,,.l test phantoms may be used. The exi~mple, a phi^intom having aplurality of test t ibes spaced at fixed locations relative to one anotiLier and conti-iining gradients of liquid densities ma~ be used to me~isure the ulllr~ c of tiLie MRI scan input device.
For optimali testing and qua'iity conti-ol of tiLie electronic 10 imaging system where a fiiim digitizer is used as the acquisition device 10, extremely accurate reference fiims 404 and 405 must be produced from defnition files 401 and 402, ~Jc~liv~ly~ This accuracy ~c~iu.,cll~ is to ensure uniform ~ui~lll..a of the system U~ UII~ t~ for metrics .^,,I.^IIlAti~n Thus, one of the keys to the pi^esent invention is tiLie production 15 and use of extremely higLi quaiiity digitai reference images.
Tin a'il of the examples described with the present invention, the number of fiims7 targets or phantom~, wiLiile shown to be two, may be of any number. It is aliways desired to limit the number of phantoms so as to minimiæ the time required for acquisition and hence ti-ie overalil qualiity 20 i e~r~^m,-. ,t Prn~ ^t~ n l-f (~Tî r. m~ri~ fi ren~^P Film FiglAre 5 is an exi^imple of a reference iri^iage used to test ihe geometi-ic accuracy of tiLie electronic imaging system. TiLie geometric 25 reference fiim 500 of Figure S UIIC~)UIIJ~ to the reference fiim or target 404 used to test high contr~ist geometi-y and MTF (mi r~11l1i tir. n ti^ansfer function).
TiLe digitali reference film or target 404 is produced in the preferied ~111 ' using Cullvcllliull~ y systems such as the type used for printed circuit artwork. In tiLiis fashion, perfectly parallel lines dra~n 30 as long as I meter which deviate by as little as 0.1 mm a.^e easily genei-ated for measuring perrect siæ precision.

WO 95/31869 2 1 8 2 5 1 4 Pf-r/US95104857 The reference film of Figure 5 contains a number of high f ontrast items sufficient to measure each of the geometric features of Table 1.There are two sets of si~lgle frequency bars, 509 and 510, extending in both the vertif~al and horiwntal axes of the irnage. These are analyæd by the S metric analysis describe~ below to test the uniformity of the bar width and spacing between adjacellt bars to determine sf an velocity uniformity in these drrections. The analysis utiliæs statistics derived from run length f~ llf~ tif)ns on bar patterns. These statistics f oupled with the previously stored knowledge of the bar patterned pitf h leads to drrect ~J~ A~ of the pixel siæ and 10 aspect ratio. The l~ UI~ll border bars 511 as well as the fine ~ertical and horiwntal lines, 505 and 506, allow for f~ l ;' 1' l of the start and end of scan uniformity and las~r beam wobble statistif~s. Also included are precisely straight edges extending the full width 502 and length 503 of the image, rectangle 501, diagonal line 504, horizontal rf solution coupons 505, vertical 15 resolution coupons 506, diagonal resolution coupons 507, horizontal single fref~uency bar 509, vertical single frequency bar 510, etc.
The lc~u;~ for reference film 500 require that the geometric features be placed on a film with stringent positional accuracy and only need to be bi-tonal. The fea~rres for the image are generated on a 20 graphics art plotter such as a gerber scientific plotte.r for creation of theorig~nal image. The image is transferrefl to film using the same ef~uipment developed for the printl~d circuit board a~t work industry. A flat bed step and expose machine, such as those from '~erber Scientific, is ideally suited to produf~tion 403 of the geometric reference film 404.5 p~rlllrtir n of ~ray-Scale Rr-fçrenrP Film In order to generate a gray-scale reference film 405, a continuous tone imaging process is used. The preferred approach is to first generate a master ste~wedfqe in which unexposed x-ray film is exposed in a 30 stepwise fashion with multiple exposures for a certain number of millicrrr)nrlc to create a starrstep pa~.tern of optical image density exposures on the x-ray filnL Thus, one small strip of the unexposed x-ray film is exposed for a fixed s ~ ~
woss/3ls6s ~ 2 1 825 1 4 P .,.)~. ~c period of time. Exposure is stopped and a next line increment of the x-ray film is exposed and both the preYiously exposed increment and the presently exposed increment are then exposed to incre se the amount of eAposure. Ihis step and repeat process thus exposes the very fhrst line a multiple number of S times and the Yery last line only once. The result is a stepped gray-scale with no scan line artifæts. Ihe crispness of the gray-scale master reference film 405 is determined by the shlpness and opacity of the edge that is used to produce the master.
The l~l~iul~lh~ of the density versus the step position of the 10 gray-scale image must be controlled Yery carefully since the master image is going to be used to produce a plurality of other x-ray films. This is A~ by laying the master on top of another unexposed x-ray fihm and expose it to a uniform ill "-)n usmg a contact exposure process.
This results in a duplicate of the master with uniform density steps of exposed 15 are s. Since the optical derl3ity is a lr)~, ;~ 11"` functionS the exposure Ll IA~ and ~- -~-,l-,--~l,~ of the original is nonlinear. Thus, the density steps on the first generation master are not linear. To produce a stepwise function of linear optical density of the film, the speed ~ of the fihm, and the exposure times must be properly calculated to produce a linear 20 result. The resultirlg reference film 405 for gray-scale stepwedge is a monotonic density step function.
The reference fhm 405 generated to test the gray-scale is formed using a step wedge pattern such as that shown m Figure 6. A stepwedge master fihm 600 is used to make contact copies as described above. The copies are then 25 used for testing of the quality assur_nce of the ~ It~ of the electronic digital imaging system. In the preferred ~ ~ ' of the present invention, the copies are x-ray image film measuring 14" wide x 17" high (35 cm x 43 cm) reference letters A and B, ~Liv~ly. Double emulsion film is patterned usmg a 32 step wedge each IlL`liaJllL~l]ly aligned across the full fihm width.
30 As shown in Figure 6, the first step (step 0) has an optical density of less than or equal to 0.2. The las~ step (step 32) density should be greater than or equal to 3.6. Each nominal density step lL~ v~ll should be SlJBSTiT~TE ~HEET ~UL~ 26~

WO 9~/31869 ' ~ 2 1 8 2 5 1 4 r~ ;o , ~y~JIu~dillldt~ly 0.11. Step uniformity across the film width should be plus or minus 0.1. The nominal step height is 0.53" or 135 cm. measured across th~
height of the film.
~tnt Fil~c When the reference films 404 and 405 are produced and ready for use for input into the elec~ronic imaging system, a description file is rnade to calibrate the reference films 404 and 405. Each description file 406 and 407 contains ~ t~ of the reference films 404 and 405 as part of the 10 calibration for that particular electronic imaging system. This is necessary since the transfer of the master digital reference images through the transfer processes 403 to produce the reference films 404 and 405, Ica~Cliv~ly, is not always uniform. ïhe ICIII,UCI~UIC of the film developing process, the particular sensitivity of the film between different batches, and the 15 Ic~ of a vety high optical density for this film nccessarily will produce variations among the reference films. Because of t~ese variations, the individual reference films 404 and 405 must be calibrated to the particular systcm and WllC~JUlldlllg description files 406 and 407 are produoed. The description file 407 describes the density of each step, the width of the step, 20 and the uniformity of each step for the step wedge reference film 405.
Sevc~ III~IJICIII~ of the refercnce films are produced and an an ay of values for these different points on the reference film are stored in the reference files.
For the description file 406 of the refcrence film 404 to 25 measure high wntrasl~ geometry and MTF, no attempt is made to calibrate the geometry or MTF target. What is included instead is a description of the bar pattcrn pitchcs, widths and sizes of any of the features on the refercnce film CUII~IJI ' ~ to the defnition 401. This is in tc~rns of XY locations of features rather than density III~IIICIII~ of gray-scale inf-rmslti/-n such as 30 found in description file 407 for reference film 404.
The step of crcating the reference films 404 and 405 and the cwll~a~ulldlllg description files 406 and 407 respectively is done once and the WO 95/31869 ` "'-, ' ' 2 1 8 2 5 1 4 P ./u~ ~o~ , reference films and description files are stored for later calibration and I~UICI~C~ of the pclrull~ c of the electrorlic image system.
('.AlihrAlilln The electronic imaging system is setup, calibrated and tested m the factory. Initial test data is saved for later trend tracking (~.~v..l~Li~e servicing). The reference films 404 and 405 and their l,UllC~JUlldlllg description files 4û6 and 407 al~vays remain with that particular electronic imaging system for quality testing and calibration purposes. The reference 10 films are mput into the system using film digitizer 408, which in the preferred elllL '- ' of the present invention, is a Lumisys model LS-150 film digitizer. ~he rcference images 404 and 405 can then be scanned in and digitized dunng any test and calibration tirne to measure the lJr~ rOI " .~"~ ofthe digitizer 408 over the life of the system through the use of the image 15 qualit~ metric analysis software system 411 to produce analysis results files412, which indicate the ~..rull~ c of tbe &gitizer, as described more fully below.
The image quality metric analysis 411 is used to measure the l`A 11111 l~ ll l. . ,t~ of the electronic imaging system described above in cormection 20 with Figures 1 and 2, namely: the image acquisition devices 10, the image display devices 20, the image storage devices 30, and the image hard copy devices 40. All four ~l ." q " " lr l ll~ of the electronic imaging system can be measured and ~ of the image quality and ~.rulll~.~ of these lr~ can be analyzcd.
~rAtiAI Rl~c~ -tifn T~ct FrAtllreC
Spatial resolution is a measure of the ability of an image , ' component to preserve image sharpness. It is also known as the point spread function, or in the frequencv domain, as the mi~ Ation tlancfer 30 function (~I~;). The former is the ~ell-known two~ ;o"~l impulse response of the system while the latter is the magnitude of the Fourier transform of the point spread function These are equivalent means of W0 95/31869 ' _: ` 2 1 8 2 ~; 1 4 r~ c . , l~lQ~llLi lg image shar~ness but MTF is the most frequently-used measure.
The MTF is tl~e contul--uous measure of the contrast response of the system to a range of spatial r~ u~llciQ. In practice, the MTF is sarnpled by ~omrllt~tinn of the contrast resronse to a set of discrete spatial ~UGll-,iQ.
S By use of a set of test patterns containing a number of single frequency bar patterns 509, 510 as shown in Figure 5, the contrast response can be obtained by observation of the h~stogram l ~ r ~ ` sampled around each bar frequency. Ihe set of numbers obtained is then a sparse sample of the continuous MTF and a qualit~ assessment can be made by eY~min~ti~n of the 10 contrast roll-off .1~ i~ of the MTI; .
In the preferred ~ ' ' of the present invention, the MTF
must be calculated from a set of pixel values ~llCS~JUlldil~g to image 11~111`.~1i~i(~11 not density. This conversion takes place during the feature extraction process desclibed above in e~ l, with Figures 4A and 4B.
15 The preferred test features for this measure are sho~n as three sets of multi frequenc~ bar pattern Al 1~ t~ 505, 506 arld 507 as shown in Figure 5.
The multi frequency bar patterns are oriented vertically, horrzontally, ar!d 45 degrees off axis. Ihis allows the system to sample the MrF at the three most important rotation angles where a scanned image device is most apt to incur 20 ~F loss.
Pixel V~ P TntP~rity T~t F~tllrec To test ~or pixel value integlity, an image containing a broad range of densitiQ is required such as that described above in ~ ~ " .j l. l~ ith25 Figure 6 which CUI1QIJUIId~ to reference film 405 described above in e~njlm~tinn with Fig lrQ 4A and 4B. A sufficient nurnber of sample points on reference film 405 is required to reveal any non-linearities over small ran~es of input values. To measure the unifolmity of response, the input regions of e~ual densi~y should be as uniform as possible. The test features 30 for this form of quality monitoring is ~ l using the stepwedge pattern as described above in . . l, ~j. l. ,~ l ;l l" with Figure 6. ~he step~vedge of Figure 6 has a dynamic range that equals or exceeds that of the component ~ WO95/31869 - -' 2 1 825 1 4 ~ u~C
under tf~St. A wedge with 32 steps and a maximum derlsity of at least 3.6 OD
is used in the preferred ~ o.~ lrlll to the present invention.
To reduce the absolute accuracy ~ and l~edl41 ilily of the stepwedge ~ l .. ;",~ process, each reference film 405 must be S ;~ y ~ rl;~rJI This ;llr(llll~ ;llll will then be conveyed to the metrics fA II I ~ ;I II I process in the separate descriptor file 407 described above in l . ., j, .", l i. .l, with Figures 4A and 4B. The descriptor file 405 is - unique to each reference film and ~ r~ each reference film as it is stored m memory for use by the preferred ~"~1 n~fl; ., .,l of the present 10 invention.
Tm~e ~ ty Metrice ArlAlysis FY~n~nle.~
By way of illllct~tif~n but not by limitation, the present mvention is capable of measuring the "laser beam wobble" in a laser-based 15 film digitizer. In such a digitizer, a laser spot scarls l~lllll;.l.lllll~I.y across the width of a film while the film is moved in a path orthogonal to the lasff scan direction. Ideally, the trajectory of the lasff spot across the film should be astraight Ime. Howevff, the opto-mP~h~nif~l system used to scan the lasff beam can make tlAe trajectory something othff than straight. This aberration 20 can occur due to such physical lllA~l;rr~ as warp in the mirror surfaces, ,";~ "". .~1 of the mirror segmffnts (if a rotating polygonal mirror is used), IfflS distortion and bearing run out or vibration in the moving 1 l~l . ~l ., ., ,... , The resuAt will be that the straight edges in the reffffflce image will be convffted to curved lines in the sensed or printed image. To quantify this 25 distortion, the prefffred ~" ,1~(1;" .. . ,1 of the presffnt invffntion produces a single numbff output from the metrics anaAysis which is a measure of the total deviation from ~h~ighfnf ~e in the sample image due to this effect.
In the prefffred rlllllo,l;."r"l of the present invffntion, this metric first isolates all of the edge pixels m a sample image resulting from a 30 scan of a perfectly straight line or edge oriffnted parallel to the scan direction of the refffence image as shown as the bordff of Figures 4A and 4B. The result is to establish an image processing procedure including vertical WO 95/31869 2 1 8 2 5 1 4 r~

gradient, thresholding, morphological dilation and thinning. ~he result will be a set of linked binary edge pixels that trace out the trajectory of the laser scan. From this array of pixels, the system extracts the set of Y-~ ' for each pixel. A classical statistical analysis procedure known as linear 5 regression is then applied to this set of c~ ~. This procedure derives the parameters for a siraight line which best ~~ ~es the trajectory path ~vith the goodness of fit measured by a minimum mean-squared error criteria.
This procedure also slllt-)m~ti~lly accounts for any rotation that may be present in the reference film at the time it is scanned. As a result, a statistical 10 quantity "standard error" which is an E~ (root mean square) measure of the total de~iation in the sa nple population from the best ~ ,...,.~i.,... This value can then be teste~ against a preset threshold value to give a go/no-go decision as to the ~. rll~hl~ y of any laser beam wobble that may be preSent.
The same technique is applied to the start/end of scan geometric metrics æ well as the density response linearity metrics during the acquisition and printin~ functions of the electronic imaging system. The application of statistical measures for the other two geometric metrics differs only from the image quality metric analysis only in the quantities used for the 20 regression analysis, such as pixel coordinates or pixel values.
Other metrics, described more fully below, utilize some for~n of statistical measure such; as mean, standard deviation, or variance. In all cases, the result is a small set of numbers wh ich can be used in a simple go~no-go test decision to perform image quality assessment on the operation of the 25 overall system.
Tm~P Acquisiti--n ~~ M~rir ~n:llySie Referring once again to Figs. 2 and 3, image acquisition device 10 is used to input a digitized image into the system. The reference films 404 30 and 405 are scanned by digitizer 408 to produce digital san~le images 409 and 410, respectively. The digital sample images 409 and 410 are analyzed by the image quality rnetrics software to locate specific regions of interest and WO g5/31869 ~~ 2 1 8 2 5 1 4 P~ 5.C

to compute ~ L~ for each region of interest. The measurements are then compared to stored threshold values for each region of interest to determine if the system is ~ u""illg at the proper 1~ r~."..~ Ievel.
An e7~dmple of a geometric/MTF reference film 404 is shown S in Figure 5 as reference film 500. This film contains a number of regions of interest, the locations and d~ iùl~ of which are described in description file 406. The image qudlity metrics analysis software 411 first locates the ~;oi~LldLiull target 508 to compute an offset vector to locate the bounding rectangles 3nd other regions of interest in the scanned digital sample image 10 409. The analysis software 411 kno~vs to begin looking for the small black square ~oi~LldLiull target 508 in the center of the image using standard feature,, O routines. A large region of interest is æsigned to the center of the digital image 409 and a histogram is extracted and searched to locate the valley in whdt should be a bimodal histogram. The valley indicates the 15 density which optimally separates the dark l~oi~Lldi~ll target from its ~UIl~ ' g light l,d~,k~uu,ld. This density of the valley is used as a threshold to .l;~. . ,",;"..~ registration target pixels. The result of the threshol&ng is a binary image ~vhich is further analyzed to extract the centroid of the target 508. This is ~ ,l using either repeated erosion 20 or IUW/~I~I)II slicing.
The X and Y UUUI~'' of the centroid of target 508 in the binary image of the &gital sarnple image 409 is used to map onto the true center in the original image 404 as described m description file 407. This mapping first tells the analysis software 411 how much the image is shifted 25 left or right and allows the analysis software 411 to assign an offset vector(skew) for the location of the other regions of interest in image 409. Locating the other regions of interest is a prelude to computmg metrics of the scanner - 408 1~ r.. " "~".`P for each region of interest to allow the metrics of Table I to be computed.
There is a likelihood that the physical reference film 404 was slightly rotated when it was scanned by film digitizer 408 resulting in a rotation of the irnage 500 as it was digitized into digital sample image 409.

Ihis rotation is of little import since the metrics are designed to operate in the presence of rotation and to compute their respective results ;I lllr~ lrl ll of rotation. For exarnple, one of the statistical techniques in measuring the ~,, ru~ c~ of the film digitizer or sc~nner 408 is to measure the linear S sc~nning ability bæed an lines 502, 503 and 504. The endpoint pixels of the scan lines 502, 503 and 504 are located in the image file 409 and a best fit linear regression is performed on the pixel locations. For example, for vertical line 503, a best fit linear regression is performed on an array of the Y-coordrnates of the piYel locatior~s. The metric analysis soft-,vare then 10 performs a best-flt line to the data set which rnakes the analysis rotation For the analysis of run lengths in the in-scan or cross-scan direction, features 510 .~nd SOg are used l~~ ,ly. A very long and narrow region of interest is defined within the bar spacing regions so that the 15 region of interest stays completely within these features. Wlthin these regions of interest, there is no concern for rotation since only the peaks and the standard deviation between the peaks I~I~IItilIg the lines within features 510 and 509 are analy7ed. ~he standard deviations will in&cate the linearit~
of the scan velocity in the &rection arlalyzed.
The mn~ ti--n trar~sfer function within coupons 505, 506 and 507 are also rotation ;~ "1 ,I These coupons 505, 506 and 507 are used to test the resolving a~ility of the film digitizer 408 and are analyzed sirnilar to the technique described above for the arlalysis of run lengths. The bounding region of interest is placed completely within the coupons to do the histograrn analysis on the resolution bars of coupons 505, 506 and 507. The spaces between the bars 511, 512 and ~13 for coupons 505, 506 and 507, ly, are purposely placed to allow small regions of interest to be placed between resolution patterns. For example, resolution pattem 514 is designed to meæure 0.2 Irne pairs per mm. ~ertical resolution in coupon 505.
Resolution pattern 516 is designed to meæure 0.4 line pairs per mm vertical resolution in coupon 505. Space 515 between resolution patterns 514 and 516 is designed to allow a region of interest to only cover a portion of the W095131869 , ` 2 1 8 2 51 4 r~"u~

horizontal bars of resolution pattern 514 without including any portion of resolution pattem 516 or the edge of coupon 505. ~hus the test pattern features of reference film 500 is specifically designed to be rotation lrlll Ihere are, however, features 501, 502 and 503 which can be S used to measure the amount of rotation from the defrnition file description 406 if 111~;111~;;11~ of rotation is a metric of interest.
Str~A r ktP~'t;On P~A~A
As a pre-cursor to tne pixel value integrity tests that are 10 performed upon the gray scale step wedge reference filrn 405, a streak.
detection process is first undertaken. An example of a gray scale step wedge reference film for digitizer or laser imager testing is shown in figure 6 as reference film 600. This film object corltains a number of l~ul~ul~
aligned uniform density steps arranged with increasing density down the IS vertical axis of the film. The streak analysis process is used to detect the presence of any vertically oriented ,I;~il".l IAI 11 ' ~ in acquired or printed images.
These 1~ arise due to several problems:
I) C~ 11 of the folding mirrors used in a swept beRm laser film digiti~er or computed Iddiu~a~lly system;
2) ~1~11IA II- IA1;--11 or obstruction of line illllminRt ~ used in CCD
based film digitizers;

3) Mis calibrated or failed prRel sites on CCD's used in a CCD-based film digitizer,

4) C~- " I~ 1 ;1111 of rollers used in transport of film throu~h either a film digitizer, laser imager, or film processor;
S) Mechanical damage (scratching) of a film due to burrs or out of alignment m~hRnirRI guides or deflectors used in a film transport path.
All of these sources of streaks prc,duce a sensed density 30 anomaly in the form of a continuous or periodic vertical (or nearly vertical) line which may be darker or lighter than the ~ uullLI~s background. The purpose of the process is to determine if such streaks are present (detection), W0 95~31869 2 1 8 2 51 4 r~
.

and if so, to classify them in terms of their width, position on the imag~ and whether they are above or below the backgrourld level. The occurrence of any stre_ks will usually be construed by the procedure which invokes the streak analysis to be a fital error, in that subsequent pixel value tests or

5 calibration should not be carried out until the cause of tbe streaking has been removed The ;"r." " IAI ;(')1~ used to classify a streak can be used by higher level software (such as a rule based Al prograrn) or by skilled t~llll;~iAll:i in locating the source of the streaking.
Two &ff~rent approaches to streak detection are utilized, 10 depending upon the type of acquisition modality. Both of these use the concept of l,~k~uu..d estimation and subtraction to produce an image object which is then analyzed for the presence of vertical anomRlies having some minimum extent along ~he y-axis of the image. The b~ht; u~ d exlraction algorithrns utilize cullV~ tiulldl linear convolution and/or 15 filtering.
The first streak detection algorithm uses a two-pass 2-D
bækground estimation, while the second one utilizes colurnn summation to reduce the problem to l-D, thereby saving; 1~ A~ processing time. The later approach, while inher~ntly faster, could fail to detect a streak if there is 20 any substantive rotation in the film under test as it is being digitized. This approach is restricted t~ use with a film digitizer where it is knov~n in advance that the rnaxir~um rotation in the acquired image is small. For CR, or systerns with less control of the rotation during R~liciti~n then the slower~but rotation i, ~. Irl .. . Illrl ,l 2-D background subtraction process must be utilized.
The anamalies are detected first within each step of tne gray scale step wedge, usin~ a test for either the pixel value deviation above boa~h~uulld ar the ill~hlllAl~PlJl~`i slope (horizontal gradient) to dete~rnine candidate anomalous regions. The total y-extent of each candidate region is measured, as well as the width of a potential streak, and its "color" or polarit~
30 relative to the image b~,h~u~-ld. All of tbis streak rlRccifi~Rtir~n ;nfOrrnRt;i~n~
including the total number of streaks encountered is saved as a result for the purposes stated above. Anomalies which do not satisf~ a minimum length ~ W11)95131869 ~ 2 1 ~ 2 5 1 4 P~ rO
criteria are considered to be discrete anomalies. A count of these is also maintained as an indicator of the overall cleanliness and random scratch content in the reference film so that reference quality can be monitored and the reference replaced at such time as the discrete anomaly count eY~ceeds a S pre- irt~min~i threshold.
FY~mpll~ of ~r~ictration of an ~I Tm~
When the acquisition modality is one of the common 3-D
volume~ic imaging systems such as MRI, CT or the like, a lC~ LIdliUII
10 process to identify any rotation in the coordinate space is especially critical to the successful application of pre-~i~t~min~i region-of-interest processing. To ~ 1 "" ,~ 1, this task, one can rely either upon the eYistence of j~ i ri,~1 .1rtargets in culllulclc;~llly available phantoms (such as those from Cone L~ ) or preferably, to design into custom phantoms 3-D objects which 15 when imaged produce pixel sets which are amenable to the f~m of automatic image processing and feature extraction discussed here. A typical MRI
phantom used for MTF and geometric distortion Ill~UlClll~,lll will usually have a number of circular inserts spaced radially around the aY~is of a cylindrical volume. Each insert will have a number of vanes or target pins 20 spaced so as to simulale a different spatial frequency. The unknown in such an acquired image will be the angular rotation of the cylinder. Ideally, ar object to be used for IC~ LI~:IiUll would be embedded in the plane of the volume of interest which has a structure tbat is siglliL~llly different from thetest objects. For instance, if the MTF test patterns utilize circular elements, 25 than the l~ LIdliull target should be a lc~ ulal (or at least rectilinear) object. It should also be a size or density difference so that when an ll~lU,UI;~, pixel value or motl~hrli~-gi-~l (shape sensitive) filter is applied to - the resulting image, a clear ~ . .",;" ,I l.. , between the l~;i~dldliUll target and the test features is obtained. Once the pixel coordinates of the registration 30 target have been ~' 1, then the orientation of the test features can be derived using a prion knowledge of the spatial leLl~iUll~ J between all of the features in the phantom. The output of the lC~ iUIl process would again WO 95131869 ` 2 1 8 2 5 1 4 r~

be an offset vector, now a 3-tuple, which is used to adjust the application of any preset regions of irlterest for further f~ality testing.
!m~ H~lrd Corv ~ ty Metric Ar~ ic S Referring once agairl to Figures 2 and 3, image hard copy device 40 is used to output a digitiæd image of the system. The image quality metric analysis tf~StS the . l ~u, Irl ;~ , of the hard wpy de~ice 40, - which in the preferred ~ 1,. ' of the present invention, is a laser imager 207. Referring to Figures 4A and 4B, the testirlg of the ~ rv.l..dll~ of the 10 laser imager 417 (W~ lUllLlillg to laser irnager 207 of Figure 2) is a closef~
loop analysis ref~uiring the use of film digiti~er 408. For this reason, the film digitizer must be calibrated and its 1~' r~..,.,~".~ measured before the p~ru~l~-~ of the laser imager 417 can be measured.
To test the output of the laser imager 417, stored digital 15 reference images W ~ IL-g to the e~ected images defined in files 401 and 402 are retrieved from the image server 413 (w -c~ to the disk storage system 204 of tlle image storage device 30 of Figure 2). The læer irnager 413 will produce printer sample filrns 420 and 421 of the expected reference images defined rn definition files 401 and 402, ~ lJ~. The 20 læer imager exposes x-ray film or the like and processes the film through film processor 418, which is an integral part of the image hard copy device 40. The printer sample film 420 allows for the Ill~c;lll~l~ of the MTF and geometry ~]I,I.,I ~lr~ of the læer imager. Printer satnple film 421 allows for ~ of gray-scale ~ I rl 1~1 ;( `. of the læer imager 417. Sarnple images 420 and 421 are scarmed back into the system using film digitizer 408 in a closed loop fæhioll shown in Figure 3. Film digitizer 30, scans in printer sample film images 420 and 421 for input into the irnage quality metric analysis 411. The results of the arlalysis are stored in the analysis results file 412.
The values of the irrlage quality metrics are tested against ul~-llr~ threshold values to deterrr~ine if there has been any ,l~
in the hardcopy generation process. The .~ ,.", of the digitized images ~ W095/31869 .' ~ 2182514 r~l,u~ c~, 409 and 410 Wll~ ' ~ to printer sample files 420 and 421, respectively, due solely to the film digitizer is ~ for during the image quality metric analysis 411 if the digitizer wæ first calibrated and its 1~, r." " ,~".
was measured before testing the p~,rulll~l~e of the læer irnager 417.
S
Im~.P ~i.Crl7~V (~:lI;f,Y MPIr;-C Ansllysic The testing of the CRT morlitor of image review ,ctation 415 CUII~IVIIV~ to the image review stations 212a-212c of Figure 2. The Ill~iUlCIII~ of the pclrulll~lce of a CRT morlitor is difficult due to the 10 wide variations in phosph~r Ar~fl~fif!n, linearity, hùrizontal phase, width, vertical sizirlg, pincushion patterns and ringing Wlth the wide variation in geometric ~ r(~"~ of CRT monitor 415, a complete ~,lrulll3~ul~
Illewu~ is difficult. Wlthin the realities of the use of an electronic imaging system, it is more important to meæure the gray-scale ~.~, r, ." "~". P
15 of such CRT monitors 415 than it is to meæure the geometric ~,~, r. " " ,~". ~
L~ `C Thus, in the preferred ~ ,lIL of the present invention, only gray-scale I h .~ r. " " ,~"~ ~ of the CRT monitor is performed. This is primarily due to the fact that the geometric display ~1,,,,,~ Irl ;~ of CRT
monitor 415 do not degrade in a slow linear fashion. Most often, the 20 .~ ;.,., of a CRT monitor, and the geometric .~ is on a grand scale such that L~ lldu~ls shrinking or distortion occurs all at once. This is typically due to a failure of an electronic componcnt within the CRT monitor.
More subtle changes in the geometric ~ rl ;~ of the CRT monit~rs are not as much of a concem, especially in the medical industry, since the 25 monitors are rarely used to infer actual physical size of the image that theyare displaying Even relative sizing of the portions of the image are typically not relied upon. For relative sizing of portions of an image, a hard copy x-ray image or the like is produced.
Photometer 414 is used to meæure the gray-scale .l ,,-".~ ;rc 30 of CRT monitor 415. The ~ vLull~ 414 is attached to the face plate of the CRT monitor at a preAPci~,n~fPA location in order to measure the gray-scale p~lrulll~ll,c;. The result of the photometer readings from IJlwLu~ t~. 414 are wo gS/31869 -- 2 1 8 i~ 5 1 4 r~
quantifled and stored in a luminance sample data file 416 for use by the image quality mf tric analysis routines 411 for ~ , against a " ,;, If.1 digital value-to-luminance function. The results of this 1 can be used to determine the ~Ir~ ~l~ of CRT monitor ~15 in 5 tbe gray-scale domain and the results of this analysis is stored in analysis results file 412. The res~lts are also used to perform calibration of the display s)~stem by 1 "~, l,~.. l1..l i. ", of the display LUT (Look-Up Table).
H~rclw~rG Look Up T~hlf ~errirtiong Look Up Tables are utilized within the electronic imaging system of the present in~ention to perform conversion of an incoming pixel data set to an outgoing pixel data set so as to acbieve a desired transfer function for a particular system ~r~mronPnt In so doing, multiple objectives are achieved including calibration, matching and adaptation to a variety of 15 pixel value l~ g For example, the CRT display LUT (Look Up Table) is used to match the CRT contrast and brightness to the film density of the original film. The printf r CLUT (Contrast Look Up Table) is used to match the printed film density to that of the original film. The digitizer LUT
is typically used to fne tune the il~ calibration and remove any non-20 linearity in the (usual) density-to-density transfer function~
Figure 7 sho~vs the relevant portions of the autonlated image quality control system for the electronic imaging system of the preferred ~.",11l..l;.,Irl11 of the presfnt in~ention syste~n where LUrs are used to transform pixel quantities. There are three LUrs in the basic system: LUT A
25 601, LUT B 60~ and LUT C 606. Tl1e I ll ll l lrl ll ~ G used to describe the LUT input and output values is shown in Table 2.

WO 9S/31869 ' -~ ' ; 2 1 8 2 5 1 4 r~l,u~ ~o .
3~
T~hlr ~ T,IJT T~ n ~nrl O~ n N~ .c MOD[0:11] LUTA MPV[0:11] FihnDigiti~er Output Stage MPV [0:11] LUT B DPL [0:71 CRT Graphics Controller MPV[0:11] LUT C IPE [0:11] Imager Input Module LUT A'601 is il l~ 1 in the Lumisys filrn digitizer 210.
LTJT ~ receives a 12 bit quantity MOD[0:11], ~Ull~VllV~lg to the Meæured Optical Density output by the A/D converter of the Lumisys apparatus. This 15 value is calibrated by Lumisys to represent directly the film density æ an equivalent nurnber of milli-OD (.001 OD) units, with a m~ximum usable output of 3600, Culll r ~' ~ to 3.6 OD (see Figure 8). As a 12 bit unsigned integer, values for MOD up to 4095 are possible. The output of LTJT A is also a 12 bit integer, which is denoted æ MPV[0:11], for Measured Pixel 20 Value. Images consisting of an array of MPV integers are what are then stored within the Image Server 30 of the systenL It should be noted that LUT
A may also be e~lui.~ ly a part of any of the other irnage acquisition devices, such as a CR or MRI machine.
LUT B 608 is located in the display ~ buffer 25 hardware of the Image Review Station 212. Its input is the MPV[0:11] array.
from the innage server, and its output is the Display Pixel r ~, which is denoted æ DPL[0:71. LTJT B 608 is typically updated by the control CPU
213 of Figure 2. LUT B simply compresses the 12 bit input data to 8 bits of CRT T ~mlin~nrP interlsity. The content of LUT B will determine the window 30 and level of displayed data, and how that data is mapped to CRT gamma and perceived brightness levels.
Other input modalities can be Llsed in the preferred t;lllbUVl'llCllt of the present invention which utilize a LUT table. For example, input modality 610 may be magnetic resonance imaging (~I), computed 35 ~II~V~ / (CT), computed lddiu~aylly (CR), ultræound (US), and others.

The LUT (A) 611 used ~vith these modalities is as described above and operates m tbe same maf.~rfer through the modality control CPU 612.
L~lT C 6fr~6 typically resides on the Laser Imager 207. It is used to map a particu~ar digital image modalities' pixel values irlto the 5 ~u~u~ , fi m exposure values. Thus, the firput to Lfl,fT C is the array MPV~0:11], while its ou~put is L-fnûther 12 bit value of Irnaged Pixel Ex~fosuredenoted lPE[0:11]. It may also be ;",l.lr "r~ 1 either in bardware o}
soffware as part of control CPU 201 of FigL,fre 2.
It should be noted that the Lumisys Film Digitizer (selected for 10 use in the preferred ~,.,l ,ù~l' ,, .L of the present mvention) can sustain some calibration drift resulting in a somewhat nonfiinear film derfsity response.
Figure 8 shows a typical density response curve for the Lumisys model LS150. The unit fnas slightly excess gain for densities up to about +l~
percent llUl~ill~iLy. LfvfT A 601 in the fiflm &gitizer 408 is loaded with the 15 values shown FigLIre 9 f~o provide ~I)",~ A1;.~" for the residuafl nu~ ~;Ly in any giverf unit. The L~fT A 601 is caicuflated duf;ing caflibration (as described more fully below) and has the general forf n shown in Fig~re 9, lhis version of LUT A in FigLlre 9 preserves the modaflities' native prxel I~I~CllLilLiull (milli-OD units).
f~n cnnjl~fletinn with LUT A of FigLIre 9, LUT B would be required as shown in Figure 10. LUT B of FigL re 10 is caflcl~lated during CRT cafiibration based upon the measufred CRT ~rrma .1,~,Al .L. . ;~ , the pixel vaflue l~.c~lLALion of the source and the desired CRT lufninance~to-source perception llAll~r,ll l ~ . ", The general form for Ll,T C is depicted m FigLIre 11 for the Film Digitizer as the i,lpUt modaf ity. Note that LUT C is primarily used to map tfile source prxel ~alues to pf inted fiflm density. In its most gene al forn, Ll,T C also ~l""~ Al~ for diffefent fiflm spe~d and contrast ul~A~t~ Lics, reduced fiim density d~namic range and for fifim-processor-induced deviations 3û from tfhe desired tf~s~er ff~nction.

WO 95/31869 . . r~ s Soft~re FlmrlinnAI I~P~rr~ptinn The present invention mcludes sof~ware operating on a computer to measure the ~ r.... "~ of all ~ of the electronic digitAI imaging system. The present invention monitors irnage quality S throughout the system to measure a wide variety of potential image quality ., . sources. Ihe use of an automated system to measure image quality throughout the electronic imaging system allows for a far greater accuracy in measuring system ~-clr~ ~lcc above and beyond traditional visual or interactive diagrlostic procedures. By illWI~ul~lLillg image quality 10 metrics within the electronic imaging system, automatic assessment of the current level of irnage quality throughout the system is effected. A
level of image quality is obtained through the use of a set of consistent statistical ir~age quality metrics a~l.l.ll,.l;.-~lly computed with very little operator assistance. The result of this sofiware system is a completely 15 objective, repeatable process that can be invoked by the users of this systemat a much higher frequency than what a field engineer could provide during periodic visits to a customer site.
The image quality control software of the preferred I ."I,oil;."~"~
of the present invention applies a set of WII~ IICI~ , irnage quality metrics 20 to both the image acquisition and image print functions (laser imager) of theelectronic imaging system. Only a subset of these metrics are applied to the image displays smce only lumrnance III~LllClll~.~lt~i are ~,..,..." ~lly feasible using automatic data collection methods. Thus, such CRT geometric aberrations such as focusing, distortion, or spatial resolution cannot be 25 measured. It is assumed that most CRT displays are ~u~~ ly reliable that their quality as to geometric or spatial resolution will be stable betwecn relatively infrequent visits by a field person.
The most critical location withm the automated ir~age quality control system of the electronic digital imaging system shown in Fig~res 4A
30 and 4B is at the site of the image A~li~itir)n such as at filrn digitizer 408, which CUllC>~JUlllb to the digitizer 210 of innage acquisition device 10 of Figure 2. The most logical point of locating the quality control hardware and W095131869 ~ 2 1 825 1 4 r~~ o software functions is where the digitized images are fLrst available in memor~.
The film digitizer station 210 of Figure 2 is, of course, attached to a host computer 209. The host computer in the preferred c; l~ud;.--.l.~ of the presentinvention is a Unix-based workstation with floating point hardware support for 5 image processing operations. Suf~lcient memory in computer 209 is rçquired to maintain the sample images, 409 and 410, as well as temporary image processing buffers for processing image metrics. This software system also has ..~ which execute on the image server 201 which is also a Unix-based wulh~ iu~ addition, a data collection rernote procedure would be 10 required on the image review station 422 to collect luminance data from the monitor under test 41~ during luminance testing using pl.u~u.~ l 414.
The software in the prçferrçd r~ ;.". .. ,l to the present invçntion is dçveloped in C/C~ and is intçnded for operation using the UNIX opçrating systçm which is platform ;,I.~ (with the çxcçption of 15 the floating point hardwarç support). Those skillçd in the art will readily recogniæ that sincç t~le sourcç code of the preferrçd, I 1 of the present invçntion is i~ )lrll~- ~t 1 in "C" code, this providçs platform I l Illrl Ic~ 1r~ ,rP for usç with othçr operating systçms such as DOS, ~iaclntosh, etc.
A ltnm~tPA Tm~ ~-~ ontrol Soflwarç Cnny nn~nt~
Figurç 12 is a ~lobal flow diagram of the automated image quality control software " I~ I II N ~ for the prçfçrrçd ~ of the prçsent invçntion. The image quality control usçr mterfacç 701 lI~
25 with the user at a high Içvel. This manager componçnt of the software handlçs procçss control, paramçter setting, usçr query, and notifir~tion of status. A database is mairltained to save all rçsults 711 from the autornated quality control procçss marlaged by the image quality control usçr interfacç
componçnt 701 of th~ software 702.
The user has direct access to the automated image quality control sofrware 4 11 1 11 1~ Il` through the image acquisition station softwarç715. The evçnts/rç~ourcç managçr 713 is a lowçr level of rçsourcç managcr WO95131869 .~ .. ~ `` 21 82514 r~l,u~,5,~c- , used by the image acquisition station software 715 to control resources 703, 704, 705, 706, 707, 708, 70g, 710 of the software system 702 at a lower level Thus, i"î..""~l;."~ received from the film digitizer station software 715 which controls the input modality, such as an image digitizer, is under irldirect 5 control of the image quality control user interface 701.
The key component of the preferred cl.l ' of the present invention is the image quality metric arlalysis core software 702 which is written in C code and which contains the low level control to perform the variet~ of image quality metrics analysis requited for system mtegrity and 10 pclrulll~l~ testing. Reference object descriptor file 703, printer test imagedescriptor file 704, (~nnfi~l~til-n data file 705, and test parameters file 706 are inputs to this software analysis core 702 as constants to be used in measuring the actual values observed from the electronic digital imaging system.
The results of the image quality mettic analysis are a vatiety of output files such as the CRT display LUT (Look Up Table) file 707, the ptintet CLUT (Conttast Look Up Table) file 708, the modality output, such as a digitizet, LUT file 709 and the etror file 710. The results of these mettic analysis are stored in an image analysis rcsults database 711 for later use and 20 review by qualified field persorlnel in locating and identifying the sources of lr~ l l in the system.
The image quality mettic analysis core 702 also relies on some third party image processing macro libtaties 712 such as DIP Station or IP
Lab Spectrum which ate well-known to those skilled in the att. As desctibed 25 above in, , with Table 1, a wide vatiety of image quality mettics can be performed by the pteferred rll,l,~ of the present invention to analyze features and statistical quantifiets as to the ~lrulll~lcc of the system.
Each of the metric analysis software routme3 used in the preferred ;;lllLl(lllilllUI~ of the present invention ate described below.

WO 95131869 - 2 1 8 2 5 1 4 . ~ o Alltnm~tf~i Tm~ Ifllity Cnntrol Prnf~f ~ St~tff Fig_re 13 is a state diagram showing the automated image qualit~ control process of the preferred f-~ lrlll of the present invention.
This quality control process begins with the user of the system making a S quality f ontrol menu selection at the image quality f ontrol user interfacf 701 of Fig~re 12. A qualit~ control menu selection is made which brings up a screen which inff~rms the user about the proposed sequence of test or calibration defined by tfle f~nnfi~lr~tinn file. The only option wouffd be to continue or, had the self~ction been made f~llolleuLbly or work flow demands, 10 it could be cf ncelled. The complete set of procedures that rnay be executed by an operator include film digitizer fcalibration 852f film digitizer ~1r~ y test 853~ film digitizer geometric test 854~ laser irnager f alibration 8557 laser imager .l~ ..-'~-..-,. ;Iy test 856~ laser imager geometric test 8577 and image storage and ~ ..1.. test 858. The execution sequence of these procf~dures is controlled by a v3riety of factors describef f more fully below.
The CR~ calibration and test procedure is ordinarily performed by a field service engirfeer using a Ifll~du~ to complete the test. This test is described below in f~f~njlmf~tifln with Fig~re 22.
Figure 13 shows the complete set of automated image qualit~
f~ontrol procedures for execution on the film digitizer station 209. Those skilled in the art will readily recognize tbat a computed l~lifj~lly interface for the electronic imaging system of the preferred rl l Ih~ f 1;1 l l~ of the present invention would ref~uire a similar process state diagf~am for testing the quality of that image acquisition device. Figure 13 shows state transition paths ~hich may occur between thf~ various steps of the complete set of procedures. The bold lines 801~ 802~ 8087 811~ 8147 819~ 8237 8267 828, ~fnd 829 of the process state diagram of Figf re 13 show the successful result paths between the various states of the pl~cedf~re for a system operating within acceptable lifnits.
30 lhe remaining paths of the state diagram are taken in the event that errors occur in the process (such as invalid calibration results) or image quality ~ W095131869 ~ ' 2182514 ._I/U..,_.'O~ /

metrics have exceeded the "fatal" level thresholds, thereby ~ the process.
Figure 13 also defines a philosophy of result reporting wherein each procedure stores its o~Yn results in the log files and additionally 5 generates waming or fatal level error messages in an e~ror file as they are cll~ uu~ cd. The approach shown generates only one type of error message (waming or fatal) for each metric type. A warning level ~- ~,-1;..,-~ of a metric is not considered a negative result except that it could be used by higher level software to generate an automatic service nr,tifir~ti(-n that a 10 problem may occur in the near future for this component in a particular category (predictive servicing). Nor~nal users of the automated image qualit~
control :~rrl jr~tir~n would never see a warning level ~ ;. "1. Rather, this category of error ~vould appear on a system o~ t~ l~ console for ;"f~ " ", .l ;", . .l purposes only. For each test ~ l, the retum status is 15 updated ~ U~ ly.
At the conclusion of each procedure, messages are displayed on the film digitizer station screen indicating the interpreted result status (wamings are concealed as successful results). At this time, results are also saved. ~ther results are saved for IJr~ r. " " ,~". ~ tracking and æ an aid to 20 Lluul~lc~llouLillg component failures. After the completion of each procedure, the user's screen indicates the next procedure to be executed and control then passes to that procedure.
Beginning at tbe start state 850, control is passed 801 to the startup routine 851. The startup routine notifies the user as to which test is to 25 be performed bæed upon prestored test r )nfigll~tinn inf~)nn~tir,n (which canbe changed as necessary). Usually, the digitrzer calibration test ~52 is selected frst. The user has the option, howe ~er, of directly selecting the digitizer density test 853, the laser imager density test 856, or the laser imager calibration 855.
- 30 Usual control is passed 802 to the digitizer calibration procedure 852. Normal completion of the digitizer calibration passes control 808 to the digitizer density test 853. A failure of this test will pass control , WO 95/31869 ~ 2 1 8 2 5 1 4 F~~ ., 5'0: / ~

either to the image storage and ~ i. ,., test 858, or to the completion of the test 859 upon a fatal error.
The digitiz~r density test 853 iS next performed. ~gain, a failure of this test passes control either to the image storage and s 1~ ;( 11 1 test in the case of a waming message, or to the completion routine upon fatal fr~Tnin~tinn of this test.
~ Ith a normal completion of the digitiær density test 853, - control is passed 81 1 to the digitizer geometric test 854. Again, a warning message Ir~ l l l;l l~ll ;~1~ this test ~uld pass control to the image storage and 10 ~ test8580r~inthecaseofafatallrlll~ controlispassed 818 to the completion rolltine $59. Under certain conditions, the digitizer geometric te~t 854 Will skip the laser imager calibration test 855 and pass control to the laser imager llrl.~;l..,l,~ Iy test 856, or to the laser imager geometric test 816 if the user so selects to ignore the laser imager calibration15 test 855.
Normal flow of the battery of tests would pass control from the digitizer geometric test 8~4 to the læer imager calibration procedure 855.
This is the ~ 1l l ll l l. . I~lrJ1 path, since calibrating the laser imager is an important prelude to the laser imager 1, . I~;ll ll l l- I Iy test procedure 856 and the 20 laser imager geometric test procedure 857. Again, a warning level ... ,.,;" ~ .., of the laser innager calibration procedure 855 sends control to the image storage and ~'JIIIIIIIllll;' " ;1lll test procedure 858. A fatal f~ ;--" of the laser imager calibratii:)n procedure 855 passes control to the completion procedure 859. Normal ~ i- -" of the laser imager calibration procedure 25 855 passes control to the laser imager llrll~ y test 856.
The laser ;mager lrll.. '~..llrl~y test procedure 856 could terminate with a warning message sending control to the image storage and ..;1~1~;1111 test procedure 858. A fatal t~rrninsltinn of the laser imager (Ir"~;l.",.rl,y test procedure 856 sends control to the completion procedure 30 859. Norn~l completion of the laser imager llrl~ llr11y test procedure 856 passes control to the laser imager geometric test procedure 857.

WO95131869 '~ ;' ' 2 1 825 1 4 ,~ C' The laser imager geometric test procedure 857 could terminate the test with a fatal error, sending control directly to the completion procedure 859. Normal completion of the laser imager geometric test procedure 857 passes control to the image storage and ~ ."-."";- A~ test procedure 858. In all cvses, Ir~ 111 l 1 IA1;~ 111 of the image storage and . I ll . ,.. , .;. ,.; ;/ ." test pr~v~cedure passes contrral to the cnmrlPtinn procedure 859, which finally terminates the process bæk at the start procedure 850.
~PtAilP~ Pcr-~pti--n of Flowch~s From the use~s p-"~Live an initial screen is presented which contains a listing of the proposed execution path for the test procedures shown in the state diagram of Figvre 13. Ihe list will be based upon an initial evaluation of the ~ A~ I I and status variables without regard to the result variables. The purpose of this list is to remind the user of the 15 sPIIllPnrin~ of the tests. The user is given a choice to either cc~ntinue or to c~ncel the automated image quality control process altogether. If the 'continue' option were chosen, the startup procedure pas~ces cordrol to one of the five next procedures described above. Each of the procedures described m Figure 13 are described more fully below in the form of flowcharts.
~lltnmAtPA Tm~r (~lA~ / PrnrPcc ('nnh~l Flnw Figure 14A is a process flowchart for any of the procedures 852, 853, 854, 855, 856, and 857 of Figure 13. Eæh of the pre~cedures of Figure 13 compute a number of metricc as shown in Figure 14A. Thus, each 25 procedure of Figure 13 computes a plurality of metrics as described above in r~njlmrtinn with Table 1.
Each prc~cedure may compute a different number of metrics as shown as metric I ~rougb n in Figure 14A. A~er the .~ AI;,.,~ of eachmetric 902, 904, and 906, a metric Test, Log, and Status Update (TLSU) 30 pr~v~cess 903, 905, and 906 is performed, ~hich is shown m more detail in Figure 14B. For example, atter the fust metric is computed 902, TLSU :~
process 903 is execlded. The TLSU process performed after each WO 95/31869 - 2 1 ~ 2 5 1 4 1 ~I~U~,'IC: ~

~1~1~ '1" 1~;-11-111 of each met~ic determines which type of message is to be reported in the log fil~. As shown in Figure 14B, if the metric calculation exceeds a fr~st threshold level 909 but not a second threshold le~el 910, a ~arrLng message is generated 911 and placed rn the message log 912. If a 5 second threshold 910 is exceeded for the computed metric, the TLSU routine 903 will put a i~al error message 913 in the log 914. Thus, the log is updated with each of the computed metrics and the results of the metric calculation compared against threshold levels.
After the rompletion of all metrics, and the completion of all 10 TLSU procedures, a result messaging and next state processrng (RMNS) routine 908 is executed to determine which state in Figure 13 is to be executed next. The result messaging and next state processing (RMNS) routine 908 is shown in Figure 14C. If, for example, a fatal error is uul~tltd in any of the computed metrics, the RMNS routme will return control to ihe ftnish or completion procedure 859 shown in Figure 13. If a warning message is computed, the next state in Figure 13 is computed and corltrol is passed to that routine.
Film Di0ti7pr (~llihr~tion Figure 15 is a flowchart for the film &giti~r calibration procedure of F;gure 13. The purpose of the digiti_er calibration is tû ensure that the measured densi~y error is kept to within a E~ r. . " ,;. ,~1 tolerance.The approach taken is to use the referenced step wedge film in ~ ", ith a linear full scale digitiær output LUT to obtain the raw digitizer 25 transfer function. From this, a new LUT is computed which1 when loaded into the digiti~r, should produce the expected linear trarlsfer function bet~veen actual and measured de3sitv. For the Lumisys digiti~r, this CUllC~)Ull~b to pixel values having a least significant bit of 0.001 optlcal density and a hard limit for inputs of greater than 3.6 optiral densitv.
This pro,r,ess is shown in detail in Figure 15. First, a prestrJred line~r full scale output LUT ~Le., without the usual clipping function at 3.6 over 3600) is (lu~vllload~l to the digiti_er 1002. Then the system prompts the , j,,. ,~!, " ~ , ~
WO 95/31869 = , 2 1 ~ 2 5 1 ~ P~ .. C'C ' user to insert the step wedge film correctly at 1003. Upon .l."l~""~ l" that the film has been entered 1004, it is digiti7ed at 1005. Af~er film rli~iti7~ti~n at 1005, the step wedge digital sample image is next checked at 1006 to verify that it is both the correct film and in the correct orierltation. A failure 5 sends control to 1007. The user is given a second chance to correct the situation at 1007. If it is not second failure, the user is given a message at 1008 and the process sta~ts anew at 1004. If the second failure of the correct orientation or correct film is indicated, the failure is logged at 1009, a fatalflag is set at 1010, and the result massaging and next state processing routme 10 at 1011 is mvoked to complete this procedure.
Assuming the correct orientation and colrect film has been entered and digiti cd, the digital sample image is checked for the presence of streaks at 1012. This is necessary to ascertain that both the digiti_er optics are clean, and that no significant scratches have been introduced into the 15 referenced film. Control is then pa~sed to the metric test log and status update routine 1013, and the number of st~eaks are counted at 1014. If any streaks are found, a fatal error message is logged at 1010 and the procedure completes at the R~hNS routme 1011. If no stre~ks are found, control is passed to 1015, where the system generates a ne~v, but not necessarily linear, 20 output LUT. If there were no errors in this calibration routme of Figwre 15, a new OLUT (Ou~put Look Up Table) is both stored for later use and ~luw~lluad~d to the digiti ær.
If an error had occurred or, if abort were selected by the user, then the error condition would be logged and the most recent digiti~r output 25 LllT would be restored in the digiti_er. The next procedure to execute would depend on the state of the results of this test as shown in the state diagram ofFigwre 13. Note that in this procedure, and in the laser imager calibration procedure described below in ~ l ;. ", with Figures 18A and 1 8B, there isno smgle metric computed from ~vhich to assess the efficac~v of the result.
30 Ihese are merely calib}~ion algorithrns.

WO95/31869 `: 2!825~4 I~I/rJ.. ~_~C /

Film D~iti7~r Dengi~ Test prorPAIlre Figures 16A and 16B describe the film digitiAr density test procedure. The ~Ir~ ' test of the filtn digiti_er is ordinarily exercised follo~ving a succe3sful calibration procedure. In this case, the system is 5 verifying that the just-computed output LUT produces the desired result. This procedure ci~n be invoked by skipping the digiti~er calibration and using the existing LUT. ~
In begirn~ing the film digiti7er density test at 1101, the user is prompted to insert or possibly reinsert tbe referenced step wedge film at 1102.
10 After the usual digiri7~rion and v~lir~wliull cycle, the streak check is performed at 1111, as w~s done for the digiti_er calibration. Again, if no streaks were detected, a series of four tests are conducted: absolute density error 111~, best fit density linearity 1116, contrast resolution analysis 1118, and large area uniformitJ analysis 1120.
Film D~ti7Pr Geom~h~c T~ re Flr.w Figures 17A and 17B describe the process for the film digiti_er geometric test. If the geometric tests are enabled by the user, then this procedure will begin flow at 1201 as shown in Figure 17A. To reach this 20 procedure requires that the digiti_er density test procedure h~s been executed ~ucu;~lly as shown irl Figures 16A and 16B. As is common in these routines, the user is prompted to insert the geometric test film 404 in the digitiAr at step 1202 .~ similar proccss of ~;rl~Liull with one retry is allowed as described above in ~ ", with Figure 15.
The geo`metric tests begin with a step to register the image obtained by fli~iti7~tion 1210. Because many of the tests to be performed involve the placement of small regions of interest in precise locations as shown on the geometric/MTF reference image of Figure 5, the system needs to know if there are an~ x or y L~ liul~S of the im ge as it ~as digitiæd.
30 From the lC~l~t~ iUII arlalysis using the registration target 508 of Figure 5, an offset is determined that is used by all subsequent tests.

WO95/31869 ~ ~ ~ 2 1 825 1 4 r~ rc AftQr registration, a sequence of eight image feature extractions and pQru~ ~ metrics are computed as shown in Figure 17B. These eight areas are: in-scan velocity uniformity analysis 1212, cross-scan velocit~
uniformity analysis 1214, pixel si_e and aspect ratio analysis 1216, nf1rmSII;7P(~
5 glare area analysis 1218~ hori_ontal arld vQ-tical MTF and a number of frequQncies at 1220, pQ~iphery urliformity analysis 1221, start and end of scan position urliformity (jitter) analysis 1223, and scarl;wobble (laser beam wobble) analysis 1225. While the analysis will fnd the M~F at a number of r~u~ ,i~, in practice, the system will likely only test the MrF value at a 10 single frequency as the lJr~ r( " " ,~"- P metric. In the preferred embodiment of the present invention, the value at or near 1.0 Ip/mm will be tested.
T ~Pr lm~Pr ('~lihr~if~n Pm~PA~lre Flrw Figares 18A and 18B iS a flowchart showing a lasQr imager 15 calibration procedure. In the prefQ~red ~ ."I,o~l,",. .1l of the presQnt mvQntion, the læPr imager is selected to be Model No. 959 or 969 læer imagQrs available from Mannesota Mining and M~ " 1 ~- ~ ", ;"~ Company, the æsignee of the presQnt mvention. The Model 969 imagers ~ r a closed-loop density and contræt control ".~ IIA.I;~III anternally. This feature elimalates the 20 need for periodic calibration for the sake of .1~ udu,liclll accuracy. However, the earliQr Model 959 machines are "open-loop" and are ,J to . ~ ;(lr"~ P density and contræt changes as a result of chemistry, film and laser power va~iations. Ihus, depending on the type of læer imager used in an electronic imaging system, the automated image 2~ quality control r "- I ;- ,"~ y of the preferred ~,IlIL ' of the present invention is essential for calihrating and ~ proper operability of læer imagers due to the excessive variability withan and between different Iaser imagers.
The feedback device for obtaining printed film densities from a 30 sample film is the fi~m digiti~r 408 of Figares 4A and 4B. Howe~Qr, some systems may not have a film digiti7er 408 as the primary input modality. For example, a computed ,~io~lly input modality may be substitated for the W095/31869 21 8251 4 I~,l/U.,,.~O' I
~ 0 digitizer. In this instance, a self-scanning spot-type df~ncit~mrter will be rA in the system as a means for llr ~ (""r~,y feedback.
The irnager calibration process is shown in Figures 1 8A and 18B. First, a nominal density setting and a linear CLUT are duw~lu~l~l to S the irnager to be calibrated at 1302. I'nerl, a print reouest is issued at 1303.
The sample film to be printed will depend upon whether or not there is a digitizer present in the s~rstem of Figures 4A and 4B. If there is, a step wedge will be printed at 1309. Otherwise, a sheet of film with an embedded test pattern suitable for rr ading by the ~ Ir~ will be made. ~he user is 10 prompted to pror-ess or retrieve the resulting sarnple film at 1310 and plaræ it in the ~J~JIU~ reading device. If a step wedge has been made at 1309.
then it is digitized and verified for c~rrect orientation at 1312 and 1313, justas in the &gitizer calibration and test procedure described above in (~njlm, ti-)n with Figure 15. After the film is veri~led, a streak analysis is 15 performed at 1314. If tlle system is using the .l~, ,~;l. ,,, ,~ t~, at 1304, an orientation test would be performed at 1308. However, the streak testing and analysis at 1314 would be skipped, since it cannot be done with a spot reading device~
Next, the new CLUT for the imager is generated at 1317. This 20 process will e{ne the printed film D,~ (maximum density obtained) and decide if any density adjustment nræds to be made. At this time, a check is rnade to sfæ if the desired density can be obtained at all, or if a Wllll~JlUllli 7~
must be made. If it is found that the density change can be made, the CLUT
generation process will ternninate without setting its DONE 'dag to be true at 25 1322. It will also output the new density to be used. At this poirlt, a new sarnple filrn would be produced arld the process would be repeated.
The CLUT generation algorithm will signal completion by setting DONE to be t~ue. At this time, it will also set the appropriate warning or fatal error flags inriicating the type of calibration result achieved. A
30 waming in this case would be interpreted to mean ~at the imager calibration was sucræssful only after a culll~)lollr~ ~ to the desired D"",~ had been made. A

W095/31869 j ~ ' 2 t ~2~ 1 4 P~ C

fatal error occurs when it is found that not even a f~ll~ulullfi~l D~r could be obtained.
Finally, tne new CLUT is saved to tne default LUT file at the image server for future print requests. It is also saved as a named result file 5 for later reversion in case of future errors. Other results obtained during the calibration process are also saved at the RMNS routine 1323. The next step in the process is then flPtPrm;nPA by the RMNS routine to move control flow as shown in Figure 13 from the laser imager calibration prf cedure 855 on to the next step.
T :~cPr Im~Pr Dencilv TPct Pro~PA~re Flow Figures 19A and 19B describe the laser imager density test prf~cedure flow. The laser imager density test may be either as a p..~ l;l",,l;nn verification of the just computed CLUT or it may be entered IS mto directly from the &gitizer testing sequence as shown in Figure 13. In either event, this procedure of Figure 19A and l9B begins exactly like the laser imager f~alibration as shown in Figures 18A and 18B. After the process of printing the ~~ sample film, scar~ning and .~ ;".~ , and ~,~;L~Lioll, the actual ~ il(-lll- hy testing begins in Figure 19B. The only 20 &fference betweP~n the testing in Figure 19B and the testing done in tne &gitizer flrl ,~ y tests of Figures 1 8A and 18B is that a &fferent set of descriptor files and test thresholds are used. The descriptor files for the imager tests account for the ",;";,'i. ~ .,. of the images that occurs on printing, as well as a moAifir~tif\n of the expected density .1., ,, Ir~ in the laser 25 imaged fiLms. The test results file will be different to account for the fact that the sample images, once &gitized, will cor~tain flP~,..I,II;I..I~ due to both the imager and the digitizer. Thus, the thresholds used to test the results will be relaxed enough to account for the cascade effect incurred. Note also that if a llrll~;l(.", ~ . were being used to obtain the data, that the last two processes of 30 urliformity 429 and contr2st resolution testing 427 would be skipped.
T ~cPr Im~Pr ~IPnmPtnc Tpct Prr)f Pfl r e W0 95131869 ~ ~ 1 ~ 2 5 1 4 r~ o , Figures 20A and 20B describe the laser imager geometric test procedure flow. This pr~cedure, when enabled, can only be performed in ~nnjlmrtinn with a film iigitiær, and will proceed in the same manner as for the digitizer geometric test procedure with the exception that a different 5 descriptor file for the geometric sample film will be referenced. Also, the test thresholds for the various MTF and geometric metrics will differ from the digitrzer case because of cascaded ~Iry~ , differences rn film color, and generally different ~clru~ ~ behavior between the laser imager and the film digitizer.0 lm~- St~ge an~ ,ll",.,..~ Tr~t Pr~ r~ re Flow As described above, another form of image quality .1~.,..1,,1;....
is due to faults in the ~ and storage paths as image data moves through the system. In the preferred C~ ' of the present invention, 1~ irnage ~."",.",.: -,1;..., errors may occur between the digitizer, and the film digitiær host, from the film digitizer host to the image server over Ethernet, and from the image ser~er to the review stations over fiber. Wlthin the image review station, faults could a]so occur affectmg only imagery. Image storage errors may occur on th~ disks in the film digitizer host and the image server.
One way to test for these kinds of errors is to store multiple copies of the same test image file at each point in the system arld then move copies from point to point. At each poirlt, a cnn~ri~nn is made betwee~ the irnage trarlsferred and the one stored. The use of an image is not strictly required, but it may be oonvenient from a standpoint of software overhead.
25 Wh$ is important is that the volume of data be similar to or greater than th$of any image file and that the da~a set trarlsferred exercise in a predictable fashion all bits and all possible birlary values typical of image files.
lhe IC~IllilCll~ for moving image-siæd volumes of data stems from a desire to test for random or pseudo-random faults in the 30 . i ." ", ., ll ,; ,.~ , o} storage p$hs. By rncreasing the time over which a transfer or storage operation takes place, the higher the probability that random error faults will be disoovered.

2 1 8 2 5 1 4 . ~ o~
Figure 21 shows a process for perforrning the kind of test described above. ~he process would propagate copies of stored images up the chain from the film digitizer to the point of display. At each node, an image "" ,~-" would be performed and errors recorded. All errors would be S harvested into the system where they would be entered into the results log file. This form of testing will provide an added confidence check for all users of the system that image quality control has been addressed at all levels. It isnot meant to supersede additional diagnostics to test the buses, CPUs and memories within the host computers used throughout the system.
Alltnm~t~l Tm~ ity Control Finich T~mrPAllre Upon completion of all the procedures called for and described in Figure 13, or upon an abort operation from within any of the procedures described above, the frnish procedure is invoked. Tn this procedure, a 1~ summary screen is presented to the user which shows the completion status of all other procedures. This screen would be a sunple forrrlat showing the final status of all ofthe ~ rl1~ I procedures.
Tm~ Review St~tinn PrnrPcc St~t~
Figure 22 is a state &agram showing the automated image quality control systern applied to the image review stations. As described above, the system is only attempting to provide control of the luminance ..1.~.~. ~rl 1~ of displayed images. This is done to ensare image quality in two main areas.
In the frrst area, the system will adjust the transfer function from acquired pr~el values to displayed luminance such that the sol[ copy rmage seen by the users is "matched" to what would be seen by that same user viewing an original film on a light box. Of course, the absoluteluminance from the CT~T displays will only be a fraction of that from a light 30 box, but the aim here is to preserve the relative contrast irl the viewed image.
This concept of so~ copy to source matching can be extended to other modalities of the preferred ~l .,l "~-l;" Irl11 of the present invention mcluding WO 95131869 2 1 8 2 5 1 4 ~ o images from a computer Iddiu~d~ y unit, ~I or CT scan. In these cases, the objective ~vill be to either match the image originally presented to the operators of these syster~s once they have window/leveled images, or to the hard copy generated durr~g the normal use of such systems. In the preferred 5 r l "l .o.l;" ,~"1 of the presellt invention, only the matching of CRT to original x-ray images is described.
In the second main area, lurninance ~I IA~ r~ c bet~veen all of the monitors used in the ele~tronic imaging system w~ll be rnatched. This feature is especially critical where users have dual headed displays. It is 10 essential that paired irndges be shown to the user with identical contrast and brightnesc for CUIll~dli:~Ull. This process will also remove the variation in gamma .l.,,.AI lr. ,~ b~veen monitors rn a pair, and optionally, between all monitors m a given inct~llAtil-n To perform these two kinds of matching, a series of procedures 15 are executed as shown in Figure 22. There are three rnain functional procedures, including CRT calibration 1703, CRT test 1704, and monitor matching 1707. The C~T calibration procedure acquires the ~.l,A,A~1r"~ of a given monitor and uses this data in i~, j. l. .. l ,. ." with ~ . " .:. .~l source-to-viewed image ~,A"~r. .,,AI;.,,, function to generate anew display LUT.
Once the new display LUT is generated, it can be .luw.llod 1~1 and tested against a seri~s of known images to verify the hardware r, .". l ;. .. ,~ . In both Or these procedures, the luminance data is obtained by rl ll ;l l~ the system on a ternporary basis with a ~JlluLu~ t~l having serial digital data output. One such unit, from Tektronics, Model J17, has an RS232 25 port which can be used with a host computer to command readings and upload luminance readings to the host. In the preferred ~ 1 of the present invention, a fiel~ engineer would connect the IJlluLu~ l to the image review station serving ~le monitor to be tested. These two procedures prompt the user as to where to place the ~IIULU~ I read head, and t~nereafter, all data30 is taken without user ill.~ iull.
Figure 22 sho~vs the se~uencmg of these calibration and test procedures for both one~ and two-headed verSions of the image review station.

=; :
WO95/31869 ~ ; 21825 ~ 4 r~ ;o~ I
Ihe system provides the option to skip the test and verification of the display LUT hardware.
Once the calibration phase is completed, the monitor matching procedure 1707 is started. In this procedure, the system would attempt to 5 rescale the display LUTs just generated for one or botb of the current image review station monitors either to match each other or to all of the rest of the monitors in the system. The exact approach is dictated by additional control - variables which specify whether to match at all, and another which says, "match only pairs." In this way, the user has complete flexibility as to how 10 the system monitors will be matched. Note that if the system is matching all monitors, this procedure will act globally upon the display LUTs of all monitors to rescale them to match the upperAower luminance ~ Al't~
just .ll~t~rrnjn~l Ihis is a highly recursive process.
(~T (~Alihrti~m Pro~ re Figure 23 is a flowchart showing the CRT calibration procedure. The S~l vi~ ull would be prompted with an onscreen dialogue and graphics showing where to place to the ~ u~ read head on the monitor to be tested at 1802. Once the placement is confirmed, tbe raw CRT
20 lurninance data would be acquired at 1803. This process performs a series of full screen fills of known digital value followed by a sampling of the luminance reading from the rhrt~mrt~r This produces an array of r~l value to act~lal lurninance that is a ~ Al t ;~I ;1 of this monitor's garnma, phosphor, and min/max luminance settings.
After lurninance data is harvested, a check would be made against preset mirAIimum and maximum levels to determine if any further manual adjustment of black level and full scale lurninance is required. If none are, then a new display LUT is computed. The LUT generation process returAs a single metric, the minimum detected slope of the ne~ LUT. This is 30 a measure of the "goodness" of the new display LUT, and if found acceptable, the LUT data is made available for subsequent testing and/or matching with other monitors. The system also allows a forced override of the cutrently WO gS/31869 2 1 8 2 5 1 4 ~ v o established limits on luminance. This feature allows for calibration of a marginal monitor in the event that it cannot be adjusted due to a weak power supply or an aged CRT.
S ('RT Tc~ t Prn~ lre Figure 24 describes the CRT test prooedure flow. Following the display LUT generation is the verification step where luminanoe error statistics are found and tested as in other ~ r. ,. ~ . r metrics. The user is prompted again to ensure that the ullu~u~ . is correctly plaoed on the CRT
10 under test at 1,02. After c-,. ,r" 1 "~ ,. ,; the display LUT hardware is loaded with the LUT data just calculated at 1903. Next, an acquisition process is started which feeds imag~s of known value to the selected monitor. This prooedure is exercising t~le LUT hardware to obtain the II,..l~rlllllrJ1 lurninanoe values. Since testing of the LUT hardware may be performed 15 using other, less time-collsuming t~ nf)~ti~ an option is provided to bypass this form of display LUT verification.
M/-nif~-r M~t~hi~ Pr~
Fig~re 25 is a flowchart showing the monitor matching 20 prooedure. This proced lre is only invoked if the user desires to globally match all CRTs. Once this procedure is started, a check is made to see if only matching of monitor pairs is desired. The fr~st step is to verify that therc are at least two monitors to match. This is clone either by looking at the status of the ~lihr~tinn~ just executed, or by inspection of the status from all25 other monitors in the system. Once the verification is done, the match monitors prooess is started at 2005. In this prooess, the monitor ~ith the lowest maximum lumrnanoe is found. Then the display LUTs of all monitors with higher luminance are rescale~ so that their maximum luminanoe will match the previously determined minimum upper limit. During this process, 30 access is required to the previously obtained ra~ L~ rl ,~ data for a given CRT obtained at some earlier time. The matching process itself does not generate any errors. R~her, it returns a count of the number of rescaled WO 95/31869 , ~ 2 1 8 2 5 ~ 4 p "~
display LUTs along with a set of CRT j~irntifir~tir,n numbers indicating which were rescaled and which were used as the master. The total monitor match procedure only generates a fatal level error when it cannot find at least two monitors upon which to act.
Tm~Ee Review ~t~tir,n Fir,ich ProrrAl~e As described above irl CO~ LiUIl with the automated image quality control FDS :~rrljr~ti~n, all other procedures will pass to the finish procedure 1708 for the image review station process of Figure æ. The only 10 action at this point in the process is to confirm the screen and exit the test routine. At this time, message are passed and displayed as to the status of this image review station and any other affected image review stations. The summary screen of the fiu~ished routine would simply display the final status of the procedures shown in Figure 13 æ either bemg successful, failed, not 15 rlrerAtir,n7ll aborted or not executed.
CONCLUSION
Although specrfic ~",1~.7;".~ have been illustrated and described herein, it will be ~ JIC~;~i by those of ordinary skill in the art 20 that any ,", ,,- ,~c" Irl IL which is calculated to æhieve the same purpose may be substituted for the specific CllllJ;)d;lll~ shown. This application is intended to cover any 7~ or variations of the present invention. Therefore, it is manifestly intended that this invention be limited only by the clairns and a-G eq~lival~:nts a ~:of

Claims (10)

I claim:

1. A method of automatically measuring the performance of the functional components of an electronic digital imaging system, comprising the steps of:
storing metric thresholds for known good system components;
acquiring a sample image from a reference object to produce a set of pixel data representative of the reference object;
measuring the selected attributes from the set of pixel data and producing therefrom a set of feature statistics;
comparing the set of feature statistics against the metric thresholds; and indicating if any one of the set of feature statistics fall below any one of the metric thresholds.

2. The method according to claim 1 further including the step of locating a specific functional component which may cause any one of the set of feature statistics fall below the metric thresholds.

3. The method according to claim 2 further including the step of indicating the cause of failure of the specific functional component which caused a particular one of the set of feature statistics fall below a corresponding particular one of the metric thresholds.

4. The method according to claim 1 further including the steps of storing a data set of reference features corresponding to the set of feature statistics; and producing the reference object from the data set of reference features for use in measuring the output modality of the electronic digital imaging system.

5. The method according to claim 1 further including the step of calibrating the functional components of the electronic digital imaging system by adjusting values in look up tables each corresponding to each of the system components.

6. The method according to claim 1 further including the step of acquiring the sample image from a reference film as the reference object.

7. The method according to claim 1 further including the step of acquiring the sample image from a three dimensional phantom as the reference object.

8. The method according to claim 6 further including the step of producing the reference object from a stored data set of reference features to measure the output functional component of the electronic digital imaging system.

9. The method according to claim 8 further comprises measuring the linearity of a line in the reference object, including the substeps of:
performing a region-of-interest calculation on the pixel data within the subset of pixel data to produce a one-pixel width line;
calculating a best fit line to the one-pixel width line; and measuring the maximum deviation of the one-pixel width line from the best fit line as a measurement of the linear geometric performance of the functional components of the electronic digital imaging system.

10. An automated image quality control system, comprising;
an input modality;
output modality;
image storage memory;
a processor connected to the image storage memory, the input modality and the output modality and having means for executing the following steps:
storing metric thresholds for known good system components;
acquiring a sample image from a reference object to produce a set of pixel data representative of the physical reference object;
measuring the selected attributes from the set of pixel data and producing therefrom a set of feature statistics;
comparing the set of feature statistics against the metric thresholds; and indicating if any one of the set of feature statistics fall below any one of the metric thresholds.