WO2001011445A2

WO2001011445A2 - Machine vision analysis utilizing a spreadsheet interface

Info

Publication number: WO2001011445A2
Application number: PCT/US2000/022383
Authority: WO
Inventors: John Mcgarry
Original assignee: Cognex Corporation
Priority date: 1999-08-09
Filing date: 2000-08-09
Publication date: 2001-02-15
Also published as: WO2001011445A3; WO2001011862A9; EP1230787A4; JP2003527658A; WO2001011862A3; JP2003535385A; WO2001011862A2; EP1221101A4; EP1221101A2; EP1230787A2

Abstract

A user interface is provided for programming machine vision applications that includes an image window for displaying an image to be analyzed (41), and a spreadsheet (46) for analyzing the image displayed with the image window. A hand-held control pad (44) is used to navigate over the spreadsheet (46), thereby selecting cells of the spreadsheet. The control pad is additionally used for menu item selection, and for spreadsheet input. The spreadsheet is semi-transparent, and is adapted to perform conditional cell execution, as well as operate on time interval data. When the control pad indicates a specific cell of the spreadsheet, a change occurs in the image to be displayed (47). Buffered outputs are utilized for circular references or recursive operations. Complex vision objects are also instantiated as a single cell in the spreadsheet.

Description

Machine Vision Spreadsheet

Cross-references to Related Applications

This application is a continuation-in-part application of application No. 09/370,705, filed 8/9/99, application No. 09/370,706, filed 8/9/99, and application No. 09/370,808, filed 8/9/99, and claims the benefit of priority of provisional patent application No. 60/160,958, filed 10/22/99, and provisional patent application No. 60/169,514, filed 12/7/99.

Field of the Invention

This invention relates generally to software user-interfaces and programming environments, particularly as applied to industrial automation.

Background of the Invention

In the field of factory automation, the high cost of implementing machine vision technology represents a significant barrier to the wide-spread adoption of industrial vision. Traditionally, a large part of this cost has been attributed to the expense of software development. Programmers with specialized machine vision experience are often required to accomplish ostensibly simple machine vision tasks. Although many attempts have been made to simplify application development, there is no clear solution to the machine vision ease-of-use problem.

In general, the ease-of-use problem is concerned with the natural tradeoffs between flexibility and simplicity. As any computer interface becomes more flexible, it becomes more complex and as such more difficult to use. Conversely, simplifying the interface generally limits its range of applicability.

With the advent of windows-oriented operating systems and graphic user interfaces, most machine vision companies have made vision algorithms accessible from interactive graphical interfaces implemented in industry- standard shared libraries such as DLL, VBX, COM, and ActiveX. Other machine vision suppliers have developed dedicated menu-driven interfaces. Still others have created elaborate graphical programming environments based both physical and abstract connection metaphors. None of these paradigms have proven completely satisfactory.

Library interfaces, even those offering active graphical components, are essentially programming accessories. The user must still construct an intricate sequence of instructions linking the various vision primitives into a single application. This process requires a skilled software developer.

Menu-driven systems do not require conventional programming but are too inflexible to be considered generally purpose. These systems are limited to narrow application domains where all aspects of an application setup and configuration can be anticipated.

Graphical programming seems like an improvement over conventional programming, but in reality the arcane nature of the metaphor represents a substantial challenge to an unskilled user. Connecting wires to boxes may be a more playful way to generate code, but the true ease-of-use advantages are negligible.

The superior ease-of-use characteristics of conventional electronic spreadsheets are well-known, especially in the field of office automation. The use of electronic spreadsheets in other fields is also known, although in the past, certain computer applications have been unable to take full advantage of the spreadsheet metaphor. One reason for this is that a large class of computer applications require algorithms that operate over time-based intervals of a continuous data stream, and conventional spreadsheets provide no intrinsic support for continuous data processing over more than one processing interval.

An electronic spreadsheet is, essentially, a means of graphically representing a set of expressions as a grid of cells. Each cell in the spreadsheet grid represents a parenthetical expression that can, in turn, be a function of some number of other cellular expressions. A spreadsheet program updates its grid, as necessary, to maintain the programmed relationship between cellular expressions. Electronic spreadsheets are intended to provide immediate response to any modifications of the programmed expressions. However, in conventional electronic spreadsheets, cellular expressions have no means for accessing previously evaluated results from prior processing intervals. Absent this capability, it is impossible for a conventional spreadsheet to process a continuous data stream of incoming data on a time-based interval of a duration greater than one processing interval.

Although methods for implementing algorithms that operate on continuous data streams are well-known in computer programming, no methods are known for implementing this class of processing within the context of an electronic spreadsheet.

Machine vision programming tasks typically involve the manipulation of images and other large data sets, a task poorly suited to conventional electronic spreadsheets.

A standard spreadsheet consists of a rectangular grid of cells, each cell representing a single value that is, in turn, a function of some number of other cells in the grid. The standard electronic spreadsheet is suited for implementing formulas that require a relatively small number of input values and generate a single scalar output. In contrast, certain machine vision functions are parameterized by arguments representing a million or more pixel values. Since there is no facility in a standard electronic spreadsheet that enables a single cell to represent multiple values, the standard spreadsheet model is clearly not suited to support functions that operate on images and other large data sets.

In the fields of scientific visualization and computer graphics, the term "spreadsheet" has been loosely applied to certain programs that manage multiple images and other data representations in a two-dimensional rectangular grid within a bounding window. These programs, typically designed for graphical editing and array processing, essentially, allow individual data cells to be formatted as data arrays. However, in order to accommodate data sets of varying size, these interfaces generally abandon the strict two-dimensional grid relationship between cells that is the essence of the spreadsheet paradigm. In fact, most of these programs are unrecognizable as spreadsheets, and are generally unsuited for machine vision.

In the past, many computer applications have been unable to take advantage of the spreadsheet metaphor. One reason for this may be that computer applications are often required to exhibit conditional statement execution behavior. However, conditional statement execution behavior is beyond the capability of conventional electronic spreadsheet programs.

In the field of pattern recognition, for example, it is common for a single program to execute one set of instructions during training, and another set of instructions during classification, the instructions for classification being dependent on exemplar data extracted during training. In this field it is also common for programs to dynamically adapt their order of execution based on external control parameters. Consequently, known electronic spreadsheets cannot adequately be used for programming applications in the field of pattern recognition.

In conventional programming, positional order of program statements and conditional branching within program statements drive the order of execution of the program statements. In the BASIC programming language, for example, a program statement at line 20 in a program listing is always executed after the program statement at line 10, unless a branch statement is encountered and a jump instruction is invoked.

Unlike a programming language, an electronic spreadsheet is essentially a means for graphically representing a single expression by distributing the expression across a rectangular grid of cells (or any regular array of cells), each cell being a parenthetical expression that can be a function of some number of other expressions in cells across the grid. As in typical expression evaluation, precedence and dependencies drive the order of execution. Since the order of execution is not explicitly programmed, there is no spreadsheet equivalent to a jump instruction in conventional spreadsheets. This limitation of conventional spreadsheets makes programming certain kinds of behavior difficult, if not impossible.

Conventional electronic spreadsheets are known to support limited conditional behavior, e.g., conditional argument value assignment using the standard IF functions.. An IF function consists of three arguments: A, B, and C, where argument A is a logical switch expression, and arguments B and C are the two possible expressions to be evaluated. If the expression in argument A evaluates TRUE, the expression in argument B is evaluated and returned. If the expression in argument A evaluates FALSE, the expression in argument C is evaluated and returned.

Like IF, the CHOOSE function simply evaluates and returns one of N expressions. The evaluation of the expression in the first argument of the CHOOSE function determines which of the N expressions will be evaluated and returned, which is essentially a generalization of the IF function. However, the functions IF, CHOOSE, and similar functions, are only capable of selecting among result values, and are incapable of controlling the order of execution of expressions associated with a cell or a set of cells within the spreadsheet.

Summary of the Invention

Overcoming the problems of the known ease-of-use approaches, the invention introduces a spreadsheet environment and programming model, powerful enough to be applied to a wide range of problems, yet accessible to one who typically does not have programming experience or expertise, such as a typical manufacturing engineer.

Although spreadsheets are widely used in manufacturing, they are not available as a framework for use in industrial computer vision.

Spreadsheet ease-of-use derives from the almost purely modeless character of its easily mastered intelligent worksheet metaphor. Unlike conventional programming, the development environment, run-time environment, and user interface are all one and the same. In the spreadsheet, there are no abstract modes; what you see is truly what you get. Change any number or any formula in any cell and everything else updates automatically.

The challenge met by the invention was to develop a spreadsheet- programming environment for industrial machine vision. Ideally, this would have been accomplished by simply finding an existing spreadsheet engine and adding vision functions. Unfortunately, standard spreadsheets are inadequate for industrial machine vision. Achieving the desired result required substantial extensions to the fundamental spreadsheet model.

One aspect of the invention is a spreadsheet interface for programming industrial machine vision applications. The machine vision spreadsheet of the invention offers many key improvements over conventional spreadsheets, as detailed herein below.

Semitransparent Worksheet Superposed on Image

The user interface of the invention seamlessly integrates the worksheet and the image. The worksheet grid appears to be a semitransparent sheet situated on top of an image display. This technique lets the user simultaneously view a full screen image and the associated worksheet. Alternative techniques include displaying the image and the worksheet side- by-side or supplying a means to flip from one display to the other. Both of these alternatives have significant drawbacks.

Locating the image beside the worksheet creates display size limitations. On small monitors, common to most production environments, both the image and the worksheet would be small and difficult to see. Such an interface would probably need to scroll the image window or scale it into a low- resolution format. Scrolling techniques that hide large sections of the image tend to cause confusion, since it is never quite clear which section of the camera's field of view is being displayed. Scaling the image into a low- resolution window is also undesirable, since graphics overlays such as crosses and region boxes cannot be positioned with single pixel precision.

Making the image and worksheet displays mutually exclusive presents an important ease-of-use limitation. This technique makes it difficult to reinforce the relationship between the contents of the image and the contents of the worksheet. In the ViewPoint user interface, changing the contents of the image can lead to changes in the contents of the worksheet. Similarly, changing the contents of the worksheet can lead to changes in the image. When the image and the worksheet display are mutually exclusive, the relationship between cause and effect becomes detached. In other words, it becomes difficult for the user to associate actions taken in the worksheet with results appearing in the image.

In preferred technique — the one employed by the user interface aspect of the invention — a semitransparent worksheet is superimposed on a full resolution image display. This approach supplies immediate feedback for user-initiated actions executed in either domain. When necessary, the image can still be displayed exclusively, but this can be reserved for times when the image demands system modal focus.

Remote Input Device Suitable for a Manufacturing Environment

Most spreadsheet user-interfaces run on business workstations, typically personal computers. The manufacturing environment for machine vision is far different and far less accommodating than the typical office environment. Users are often standing, with no desk for a keyboard and mouse. In addition, operators may need to be several feet away from the computer monitor to adjust the trigger, the camera, the lights, and other remote components.

The user interface aspect of the invention better accommodates the manufacturing environment. Its pendent remote interface device lets operators control the vision system while standing at a distance from the monitor, even while wearing gloves. Operators navigate using a combination of four direction buttons and four action buttons. Unlike any other spreadsheet interface or environment, an entire application can be constructed using a total eight buttons. Although it somewhat less familiar than the keyboard/mouse combination, it is, nonetheless, well suited for its intended purpose and far superior to conventional input devices, given the practical constraints imposed by the manufacturing environment.

Worksheet Focus Coordinated with Image Display

As explained above, the user-interface aspect of the invention strives for the seamless integration of the spreadsheet and the image. It achieves this partly by simultaneously displaying image and worksheet data. Beyond simple overlaying data, the invention coordinates browsing in the spreadsheet with presentation of the image. The spreadsheet cell that has focus — that is, the cell selected — can automatically cause a change in the image display consistent with the associated worksheet formula. The feature is also referred to as "flyover", in that as the point of focus "flys over" the cells of the spreadsheet, the associated image changes in some way. The image is typically visible through the spreadsheet displayed as a semi-transparent (translucent) image. For example:

When a cell with an image-related input parameter is selected, the associated parameters are graphically depicted in the associated image.

When an image-processing cell is selected, the invention automatically displays the processed image.

When a graphics cell is selected, the associated graphic component will be activated in the associated image. If it is already visible, its color will be changed so that it can be uniquely identified.

2D Image Integrated in Dependency Tree

In most programming languages, positional order and conditional branching drive the order of execution. In BASIC, for example, line 20 is always executed after line 10, unless a branch is encountered and a GOTO LINE# statement is invoked. Unlike a conventional programming language, a spreadsheet is essentially an elaborate representation of a single expression. As in expression evaluation, precedence and dependencies drive the order of execution.

For example, in the expression X=A*(B+C), the parenthetical expression (B+C) is always executed first. This is true even if the order with respect to the factor A is reversed, as in X=(B+C)*A. The parentheses enclosing B+C indicate to the expression evaluation engine that adding takes precedence over the multiplication. Taking the example a step further, in the expression X=A^*((B+C)^*(D+E)), (B+C) and (D+E) are executed first, but the order of execution is unspecified and unimportant. In a spreadsheet, any expression contained within a cell is the functional equivalent of a parenthetical expression. So, if a cell Z contains B+C, cell Y contains D+E, and cell X contains A^*Y^*Z, then cell X is functionally equivalent to the expression X=A*((B+C)*(D+E)), since cells Y and Z must be satisfied before cell X can be evaluated. Spreadsheet execution is driven by dependency, not by the order or position of the active cells. The spreadsheet aspect of the invention makes important contributions to the known spreadsheet paradigm by introducing, among other aspects, two-dimensional image data, complex data structures, and objects into the dependency graph.

Most known spreadsheet interfaces allow users to insert bitmap image data in the worksheet, but this is mostly for simple display. The vision spreadsheet interface of the invention is unique in its integration of two-dimensional images with worksheet formulas. Unlike most spreadsheets, which deal with images by simply overlaying bitmaps on top of cells, the invention represents an entire image with a single cell. According to an aspect of the invention, an image acquisition initiates a chain of execution starting with those cells that reference the acquisition buffer as an input data. If an image feature extraction operation depends on some form of image preprocessing, then a reference to the preprocessed image cell can be passed as an argument to the image feature extraction function.

More concretely, suppose the objective is to inspect the contrast of a preprocessed image region. The end result is pass/fail flag, where 1=PASS and 0=FAIL. One could write the pseudo-expression:

OUTPUT(Bit#,CONTRAST(HISTOGRAM(IMAGEPROCESS(ACQBUF FER)))>Thresh)

As in the previous example, this expression could be split into individual cells.

CELL A= IMAGEPROCESS(ACQBUFFER) CELL B= HISTOGRAM(A) CELL C= CONTRAST(B) CELL D= Thresh

CELL E= OUTPUT(Bit#,OD)

When the system of the invention receives an acquisition trigger, the new image data drives the update of (Cell A). (Cell A) creates a processed image represented by (Cell A). These events force the update of (Cell B), which represents the grayscale histogram of (Cell A). After the histogram is computed, the image memory associated with (Cell A) can be automatically released, since all dependencies associated with the processed image have been satisfied. (Cell C) reduces the contrast to a single value. (Cell E) compares that value to a threshold stored in (Cell D) sending the Boolean result to a specific bit of discrete I/O.

In addition to simple cell dependencies, the spreadsheet of the invention understands and manages regions of interest within image buffers. Data written to a specific section of an image buffer represented by a single cell will update only those cells dependent on the pixels written. If in this example, an acquisition were directed to a region of interest that doesn't intersect with the image-processing region, the chain of execution would not be initiated. This is because there would have been no actual change of pixel input data with respect to the image-processing operation in (Cell A).

Expressions can Depend on Clock Input

As described above, spreadsheet execution is driven by the need to update dependent expressions. According to the spreadsheet of the invention, however, expressions may be dependent on external clock input. Therefore, broadcasting a clock input to the spreadsheet causes clock dependent cells to update. A clock signal of the invention can be the result of a manual trigger action, a free running clock, a timed internal clock, or an external trigger. The presence of this clock dependency gives the spreadsheet of the invention unique advantages, allowing the extension of spreadsheet techniques into the time domain. Expressions Respond to Coded Triggers

As described above, all cells and expressions respond to global external clock input. Cells respond to this clock signal in one of three ways: Run, Define, or Refine.

The Run response is the normal spreadsheet response: cells simply execute their nominal function.

The Refine signal tells cells that react to Refine to take the image and worksheet data existing from the previous clock step and use it to update their internal state. For example, a search might take the most recent POSE result coordinates and use them isolate image data for the purposes of updating a stored model template.

The Define signal tells cells that react to a Define to execute a Define process. For example, the search function might respond by automatically bringing up the search property sheet and prompting the user to train a new model template.

This unique ability to respond to coded triggers allows the spreadsheet of the invention to react much as if it were a single vision tool. The user of the spreadsheet of the invention, using this capability, can develop not only the run-time sequence of operation but also specify the pattern of both manual and automatic training behavior.

Special Cells Buffer Outputs for Recursive Operation

As described above, all cell expressions implicitly depend on clock input. This statement is not intended to imply that a clock signal is required to achieve output. The clock is simply one of, potentially, several inputs that must be satisfied.

A conventional spreadsheet does not employ the concept of a globally synchronized system clock. Outputs occur as data propagate freely along a chain of dependencies. Unlike conventional spreadsheets, the invention supports the concept of clocked cells. A clocked cell acts in two phases of the global clock: In the first phase, inputs are satisfied and an output is generated. In the second phase, the output buffer is copied into the internal buffer.

This architecture makes it possible to construct items such as counters and accumulators — something impossible in a known spreadsheet.

For example, suppose one wanted to count defects. Let the defect be asserted by a (1) TRUE output in (Cell A). Let the counter be in (Cell B). In the spreadsheet of the invention, (Cell B) could contain the expression B=B+A — that is, take the contents of (Cell A), add it to the contents of (Cell B), and leave the result in (Cell B). This syntax is legal only because (Cell B) is a clocked cell. On the first phase of the clock signal, (Cell B) copies its contents into a local buffer. On the second phase, (Cell B) accesses its own buffer and (Cell A), sums the two values, and places the result in (Cell B)'s output. Since clocked cells update only on the external clock signal, the circular reference is interrupted by the clock input signal parameter, as illustrated below.

Defect Defect Counter

Cell A Cell B

Clock A Bo Bb

0 0 0

0 phase 1 1 1 =A+Bb 0 phase 2 1 1 1=Bo

1 phase 1 0 1 =A+Bb 1 phase 2 0 1 1 =Bo

2 phase 1 1 2=A+Bb 1 phase 2 1 2 2=Bo

3 phase 1 1 3=A+Bb 2 phase 2 1 3 3=Bo

Table 1

In this example, three (3) defects occur in four (4) clock cycles. A conventional spreadsheet cannot duplicate this simple behavior. Clocked cells are essential for time domain integration.

Special Functions Support Data Buffering over Time

The above discussion describes a simple clocked cell. The spreadsheet of the invention supports another variety of clocked cell by way of the FIFO function, which allows spreadsheet users to create pipeline buffers of arbitrary depth. The FIFO function clocks data through a first-in-first-out shift register. The output is the data from N clock periods ago, where N is the depth of the pipeline. The pipeline can also be tapped at any depth necessary. This represents a unique and powerful function with respect to spreadsheet machine vision. Consider three examples:

To compute a Pass-Fail output based on the average of the last N feature responses, the FIFO function could be tapped to access the previous N responses, which could be averaged and compared to the current value.

To delay the output of a Pass-Fail signal — a common problem on assembly lines, since rejects can rarely be handled at the vision inspection station--the Pass-Fail can be fed into a FIFO of the appropriate depth. The pipeline can also be tapped to support actions at any stage between the inspection station and the final reject station.

To compute the velocity of a feature coordinate, three FIFO pipelines could be created. As the feature is located, the first FIFO inputs X coordinate, the second inputs the Y coordinate, and the third inputs a real-time clock. By appropriately tapping the three FIFOs, it is possible to compute the time- based derivatives of the image feature.

These three tasks are impossible using conventional spreadsheet operations.

A Coherent Hierarchy of Data Reduction

The spreadsheet function interface of the invention has many examples of cells that represent more than one value. For example, template search not only returns the number of templates found, it also buffers the POSE coordinates, the score for each result, and internal state data including the trained template. This kind of cell, which we call a complex cell, is actually more like an object than a function. The spreadsheet of the invention preserves the standard single-value return paradigm by providing data-access member functions. For example, the POSE row coordinate of a search results in A1 is available through the access function PoseRow(A1 ,0). An elegant feature of the spreadsheet vision architecture of the invention is that image buffers, like vision tools, are treated as complex objects. Image- buffer access functions are vision tools. The image acquisition buffer is simply a complex cell accessed by vision tools that are in turn accessed by increasingly higher-level access functions. Image data is, therefore, systematically reduced to scalar values that can be manipulated by conventional means.

Sine the spreadsheet paradigm forms the basis of many of the worlds most success computer applications, e.g., Lotus 123 and Microsoft Excel, literally millions of users are already familiar with the concept of worksheet based computer programming. Consequently, another important advantage of the machine vision spreadsheet of the invention is that users in an industrial environment will probably already be familiar with the basic aspects of the user interface of the invention. Remarkably, despite the acceptance of conventional spreadsheets in manufacturing, industrial machine vision products have never successfully adopted spreadsheet-programming techniques. This is due, in large part, to the inability of conventional spreadsheet products to manage the dimension and complexity of the machine vision application domain.

Vision applications introduce many complications including an inherent time dimension, complex data types, and special human interface constraints. The invention addresses these complications with innovative extensions including:

Clocked cells and FIFO functions to extend cell dimensionality in time;

A coherent methodology for manipulating and reducing complex data; and

Special human-interface input means and graphical display techniques.

By adapting the well-known spreadsheet-programming paradigm, the enhanced spreadsheet of the invention succeeds in bringing new flexibility and intuitive ease-of-use to a significant number of industrial automation problems, such as machine vision problems, that could otherwise only be solved through complex conventional programming techniques. The invention also provides an electronic spreadsheet adapted for programming the processing of continuous data streams. The invention also provides means and methods that, to a high degree, preserve the fundamental characteristics of conventional electronic spreadsheets. The invention also provides means and methods, which are uncomplicated and intuitive to use.

These benefits and features are accomplished by extending the spreadsheet paradigm to include apparatuses and methods of clocked data buffering, such as by using shift registers, delay lines, FIFOs, pipelines, and even random access memory. The use of shift registers is preferred because shift registers do not require addressing. Delay lines objects are instantiated (memory is allocated for each delay line object) and assigned to spreadsheet cells (each allocated memory is associated with the coordinates of a cell). A delay line object includes a data buffer configured as a variable length shift register, and a method for clocking data through the buffer. In a preferred embodiment, input data streams supply the clocking signals, and "tap functions" support random access of elements, and other analysis over the buffered interval.

The invention extends the conventional electronic spreadsheet paradigm to include the processing of continuous data streams on finite intervals.

The invention further provides methods for adapting electronic spreadsheets to applications in the field of industrial machine vision, while also substantially preserving the fundamental characteristics of conventional electronic spreadsheets. The invention also makes efficient use of a given display area and resolution, using for example, a semitransparent spreadsheet superimposed over an image of the contents of a data buffer. Further, the invention facilitates and simplifies the manipulation of images and other large data sets, while preserving the ease-of-use, and other advantages and benefits, of the spreadsheet paradigm.

The invention exploits a new class of object-oriented spreadsheet functions that incorporate internal data buffering. In addition, the invention provides methods by which the graphical representation of buffered data sets may be displayed simultaneously with a spreadsheet grid, preferably as a graphics layer positioned beneath a variably transparent spreadsheet grid. A split-screen and dual screen approach can also be employed for displaying the buffered data sets alongside the electronic spreadsheet.

In a preferred embodiment, a conventional spreadsheet is extended to include data buffering functions, herein referred to as "single method objects", because from the perspective of the spreadsheet programmer, there is only one public member function, i.e., only one public method. For example, in accordance with the invention, the familiar "search" tool of machine vision is the public member function of the search object, which includes a search function, as well as data associated with the search function.

A single method object can be instantiated and assigned to a spreadsheet cell within a variably transparent grid superimposed on a graphical display buffer, the contents of the graphical display buffer being determined in accordance with the currently selected cell. If an empty cell, or a cell containing a conventional spreadsheet formula is selected, the image last acquired is stored and displayed in the underlying graphical display buffer. If a cell containing a single method object is selected, a graphical representation of the object data members is stored in the underlying buffer and displayed.

The foregoing methods selectively extend data dimensionality to meet the practical requirements of machine vision processing, while still retaining the fundamental characteristics and ease-of-use of a conventional electronic spreadsheet.

The invention also provides methods and apparatuses for programming conditional execution behavior in an electronic spreadsheet, and therefore methods of programming the order of execution of expressions associated with cells in an electronic spreadsheet. While providing such methods and apparatuses, the invention preserves the fundamental characteristics of conventional electronic spreadsheets to a high degree. Moreover, the methods and apparatuses of the invention for programming conditional behavior in electronic spreadsheets are easy and intuitive to use.

The invention includes implicit conditional wrappers associated with each cell of the electronic spreadsheet, whereby the execution of each cellular expression is conditioned upon the state of an individually assigned logical switch expression included in an associated implicit conditional wrapper. In a preferred embodiment, every expression in every cell is enabled or disabled by an implicit conditional wrapper. This implicit conditional wrapper uses a logical switch expression to control the execution of the cell's expression, i.e., to enable or disable the execution of the cell's expression based on the value of the logical switch expression.

In a preferred embodiment, the logical switch expression is entered into the implicit conditional wrapper of a cell using a pop-up dialog box and an associated pop-up window, for example. The dialog box can then be closed by the user, causing the pop-up window and its entered logical switch expression to disappear along with the dialog box. Closing the dialog box makes the logical switch expression appear to no longer be associated with the cell, thereby making the logical switch expression invisible or "implicit", because the existence and nature of the logical switch expression is not visible to the user solely by looking at the contents of the cell. However, the effects of the logical switch expression become manifest upon execution of the electronic spreadsheet.

The logical switch expression can be any legal spreadsheet expression that can be evaluated to logical TRUE or FALSE. If the logical switch expression evaluates to TRUE, the cell's corresponding expression is evaluated, a new result value is returned, and the cell's internal result buffer is updated accordingly. If the logical switch expression evaluates FALSE, the cell's expression is not evaluated, and the cell's result buffer is returned unmodified.

In a preferred embodiment, individual switch expressions are initialized to a constant logical TRUE. In another preferred embodiment, the spreadsheet user interface supports the interactive assignment of a switch expression to any cell or range of cells within the spreadsheet, using dialog boxes and popup windows, for example. A user selecting a "cell state" display mode from a menu after selecting a cell will cause a dialog box to appear, along with an associated window that displays the associated logical switch expression.

The default behavior of the improved electronic spreadsheet is unaffected by the extensions and improvements described above if all individual logical switch expressions are initialized to a constant logical TRUE. In this case, a user can ignore the implicit conditionals. Alternatively, the user can mark arbitrary sets of cells, and then selectively assign logical switch expressions to program desired electronic spreadsheet behavior beyond the capacity of known electronic spreadsheets, such as providing various orders of cell execution.

The invention efficiently and elegantly extends the conventional spreadsheet metaphor to meet many of the fundamental requirements of conditional expression execution in computer programming.

BRIEF DESCRIPTION OF DRAWINGS

The invention will be more fully understood from the following detailed description, in conjunction with the following figures, wherein:

Fig. 1 is an example illustrating methods of electronic spreadsheet programming for processing continuous data streams of the invention;

Fig. 2 is an example output sequence illustrating the response of the example program of Fig. 1 to a specific input sequence;

Fig. 3 is a diagram illustrating the components of a delay line object; and Fig. 3A is a diagram of the parameters of the delay line object of Fig. 3.

Fig. 4 is a block diagram of a computer system for suitable for practicing the invention;

Fig. 5 illustrates a superposition of graphics layers to obtain the user- interface graphics display of the monitor of Fig. 4 ;

Fig. 6 is a depiction of the graphical user interface of the invention with cell A1 selected of a semi-transparent spread sheet that is superimposed on the image corresponding to the single method object instantiated in cell A1 ;

Fig. 7 is a depiction of the graphical user interface of the invention with cell A2 selected of a semi-transparent spread sheet that is superimposed on the image corresponding to the single method object instantiated in cell A2;

Fig. 8 is a depiction of the graphical user interface of the invention with cell A3 selected of a semi-transparent spread sheet that is superimposed on the image corresponding to the single method object instantiated in cell A3; Fig. 9 is a depiction of the graphical user interface of the invention with cell A4 selected of a semi-transparent spread sheet that is superimposed on the image corresponding to the image last acquired;

Fig. 10 is a sequence of four instances of a portion of an electronic spreadsheet incorporating the improvement of the invention, each instance including a dialog box and a pop-up window making explicit the logical switch expression of the conditional wrapper associated with each cell of the improved spreadsheet of the invention;

Fig. 11 is a listing BASIC computer program that is the functional equivalent of the spreadsheet of Fig. 10;

Fig. 12 is a flow chart illustrating a method of the invention for programming conditional execution in an electronic spreadsheet;

Figs. 13-16 are flowcharts representing the logic of an unimproved spreadsheet;

Fig. 13 is a flowchart of the main loop of a spreadsheet without the extensions of the invention;

Fig. 14 is a flowchart of the DisplaySheet() module of Fig. 13;

Fig. 15 is a flowchart of the EvaluateSheet (Row, Col) of Fig. 13;

Fig. 16 is a flowchart of the EvaluateCell (u,v) of Fig. 15;

Figs. 17 - 29 provide a hierarchically structured flowchart of an example program supporting methods of the current invention, wherein:

Fig.17 s a flowchart of the main processing loop; Fig.18 s a flowchart of a function to display cell contents; Fig.19 s a flowchart of a function to input new formulas; Fig.20 s a flowchart of a function to update the spreadsheet; Fig.21 s a flowchart of a function to respond to a trigger; Fig.22 s a flowchart of a function to display the extended data ; Fig.23 s a flowchart of a function to display a data cell; Fig.24 s a flowchart of a function to execute a single formula ; Fig.25 s a flowchart of a function to execute dependent formulas; Fig.26 s a flowchart of a function to execute conventional formulas; Fig.27 s a flowchart of a function to execute extended formulas; Fig.28 s a flowchart of a function to allocate cell data; and Fig.29 s a flowchart of a function to de-allocate cell data. Detailed Description of Preferred Embodiments

Referring to Fig. 1 , a sequence of user interface screens 1 , 2, 3, 4, 5, and 6 illustrates how a spreadsheet processes a continuous data stream according to the invention. Displayed is a spreadsheet program that computes the square root of the sum of the squares of the last three data elements of a continuous data stream. The spreadsheet program of Fig. 1 is constructed in 6 expressions entered in cells B1 , B2, B3, B4, B5, and B6.

Cell B1 contains an external input function "INPUT(1)". The input function "INPUT(1)" accesses a data source "1" external to the spreadsheet. In this example, we assume that cell B1 updates periodically as a function of some unspecified external event, e.g., in response to data on the lines of a parallel port whenever a line is toggled, or in response to a pixel value at specified coordinates in an image at a particular time.

Cell B2 contains an instance of a delay line object, and is parameterized with four parameters: the input data stream (B1*B1), the clock source (B1), the number of elements in the shift register (3), and the clock divider (1).

Referring to Fig. 3, a delay line object 30 consists of a data buffer 32 configured as a variable length shift register (or as a delay line, FIFO, pipeline, etc.), and a method for clocking data through the buffer 34. The clock signal 34 indicates when data is valid for processing in the spreadsheet. Input data streams supply the clock signals 34, and "buffered interval" spreadsheet functions support the random access of elements and other analysis over the buffered interval. For example, parallel random access of data elements is performed using "tap functions" 36 that independently access single values from the delay line buffer 32 at particular specified locations by calling methods 37 of the delay line object. Analysis over the buffered interval includes any vector operation on the data buffer, including summing the buffer, and performing a standard deviation or other statistical analysis on the buffer. Further, other sources of clocking signals include a timer (e.g., a clock edge every 20 seconds), an image acquire signal, or a parallel port, or any external event, for example. Other spreadsheet entities can also provide clock signals. The delay line object also includes a plurality of parameters 40, as detailed in Fig. 3A.

Referring to Fig. 3A, the delay line object is parameterized by four arguments. The first argument is the input data stream 31 ; the second argument is the clock source 33; the third argument is the number of elements in the shift register buffer (delay line) 35; and the forth argument is a clock divider 39.

In the example of Fig. 1 , the data source is defined by an expression that squares the input data stream B1 ; the clock source is referenced to cell B1; the number of data elements in the shift register is three; and the sample rate is one per cycle. So, when data is received in B1, it is squared and input to a three-element delay line.

Cells B3, B4, and B5 access the delay line instantiated in B2 to return the values of all three elements in the delay line data buffer. The tap function TAP(B2,0) in cell B3 accesses the last data item to be input to the delay line in cell B2. The tap functions TAP(B2,1) and TAP(B2,2) in cells B4 and B5, respectively, access data that was input on the previous two clock cycles 1 and 2.

Cell B6 completes the processing by referencing B3, B4, and B5 in a computation of the square root of the sum of the three values B3, B4, and B5 returned from the delay line's buffer.

Fig. 2 shows the final four spreadsheet display states that result when the spreadsheet program of Fig. 1 processes the input data stream {1.00,2.00,3.00,4.00,5.00,6.00}:

At time (T-3) cells B3, B4 and B5 display the values 9.00, 4.00, and 1.00, respectively, resulting from the squaring of the first three input data elements 3.00, 2.00, and 1.00;

At time (T-2) the value 16.00, the square of the input data value B1=4.00 (B1^*B1), shifts into the delay line;

At time (T-1) the value 25.00, the square of the input data value B1=5.00 (B1^*B1) shifts into the delay line; and

At time (T-0) the value 36.00, the square of the input data value B1=6.00 (B1^*B1) shifts into the delay line. For every cycle, the square root of the sum of the delay line contents (Mag) is computed by the expression assigned to cell B6: (B6=SQRT(B3+B4+B5)).

The input function INPUT(1) receives data from an external source and updates the value assigned to B1. The delay line object in B2 (B2=DELAY(B1^*B1 ,B1 ,3,1) is clocked by the input function, and a new value is shifted into the data buffer upon each new input. The three tap functions TAP(B2,0), TAP(B2,1),and TAP(B2,2) are dependent on the contents of the delay line B2, and update in an unspecified order. Finally, the expression in B6 (SQRT(B3+B4+B5)) is dependent on B3, B4, and B5, and is evaluated, the result being assigned to B6. At this point, all dependencies have been satisfied, and the contents of the spreadsheet remain unchanged until the next input event occurs.

In the forgoing example, the delay line expression in B2 behaves in a way very different from a conventional spreadsheet expression. First, the delay line is not a function; rather, it is an object instance, i.e., there is data storage (memory) allocated. Delay line class objects combine data storage with a member function that implements the shift register, for example.

Second, unlike a conventional spreadsheet expression that is evaluated only as necessary to update the spreadsheet, according to the invention, the evaluation of the delay line member function is executed only upon triggering by a clock signal. The value of the input to the delay line member function does not need to change to force the evaluation of the delay line member function. The state of the source data argument is irrelevant; the shift register action can only be activated by a signal from a valid clock source, such as the input function in the example.

Referring to Fig. 3, the delay line object 30 consists of a data buffer 32 configured as a variable length shift register (or as a delay line, FIFO, pipeline, etc.), and a method for clocking data through the buffer 34. The clock signal 36 indicates when data is valid for processing in the spreadsheet. Input data streams supply the clock signals 36, and buffered interval functions support the random access of elements, and other analysis over the buffered interval. For example, parallel random access of data elements is performed using "tap functions" 38 that independently access single values from the delay line buffer 32 at particular specified locations by calling methods 37 of the delay line object. Analysis over the buffered interval includes any vector operation on the data buffer, including summing the buffer, and performing a standard deviation or other statistical analysis on the buffer. Further, other sources of clocking signals include a timer (e.g., every 20 seconds), an image acquire signal, a parallel port, or any external event, for example. Other spreadsheet entities can also provide clock signals.

The delay line object 30 also includes a plurality of parameters 40, as detailed in Fig. 3A, including, for example, the input data stream 31 , the clock source 33, the number of elements in the shift register, and the clock divider 39.

Large Data Sets

Referring to Fig. 4, a preferred spreadsheet user interface for machine vision applications includes a monitor 41 connected by cable to a computer 42. The computer 42 includes image acquisition, processing, and graphical display capability. A camera and lens assembly 43 has an object surface 45 within its field of view, and is connected to the computer 42. Also, connected to the computer 42 is a user input device, such as a game controller 44, or a standard keyboard (not shown).

The screen of the monitor 41 depicts a semitransparent spreadsheet 46 superimposed on an image and graphics layer 47 to form a composite display of the invention. The transparency of the semitransparent spreadsheet 46 can be adjusted by the user. The semitransparent spreadsheet 46 can also be displayed in a split-screen arrangement wherein the image and graphics layer 47 is displayed on one portion of the screen of the monitor 41 , and the semitransparent spreadsheet 46, which may be adjusted so as to no longer be transparent, can be displayed in another portion of the screen of the monitor 41. Alternatively, the image and graphics layer 47 can be displayed on the screen of the monitor 41 , and the semitransparent spreadsheet 46, which may be adjusted so as to no longer be transparent, can be displayed on the screen of a second monitor (not shown). Fig. 5 shows the superposition combination of the contents of data display buffer 51 and the semitransparent spreadsheet overlay 52 to form a composite display 53. Not shown are the split-screen combination, and the dual-screen combination of the contents of data display buffer 51 and the semitransparent spreadsheet overlay 52.

Figs. 6, 7, 8, and 9 illustrate the effect of spreadsheet cell selection on the underlying data display buffer. The contents of the underlying data display buffer changes in accordance with the particular cell that is currently selected. Also, the contents of the formula line changes in accordance with the particular cell that is currently selected. Note that only the data contents of the selected cell is displayed; all of the other data associated with the objects underlying the other cells remains invisible to the user. By contrast, all of the data in a standard spreadsheet is visible to a user, including a large standard spreadsheet, notwithstanding the need to scroll over the large spreadsheet.

In monitor 61 of Fig. 6, spreadsheet focus (highlight) 63, indicated by the frame border (or any selection highlighting), is on cell A1. Formula line 62 indicates that cell A1 is assigned the single method object ACQUIRE, which contains an image buffer resulting from its image acquisition method. The underlying data display buffer 64 shows the content of the image buffer associated with ACQUIRE.

In monitor 71 of Fig. 7, spreadsheet focus 73 is on cell A2, indicating that cell A2 is selected. Formula line 72 indicates that cell A2 is assigned the single method object PROCESS which contains the image buffer that results from the processing a rectangular region 75 in the image associated with cell A1. The other four arguments of the object PROCESS represent the coordinates of the upper-left corner, and the height and width, of the rectangular region 75 in the image associated with cell A1. The underlying data display buffer 74 shows the content of the image buffer associated with the processed image.

In monitor 81 of Fig. 8, spreadsheet focus 83 is on cell A3. Formula line 82 indicates that cell A3 in this example is assigned the single method object HISTEXTRACT which contains a histogram array extracted from a rectangular region 85 in the processed image assigned to cell A2. The other four arguments of the object PROCESS represent the coordinates of the upper-left corner, and the height and width, of the rectangular region 85 in the image associated with cell A2. The underlying data display 84 shows the content of the input image buffer plus a graphical representation of the histogram array associated with the content of the input image buffer.

In monitor 91 of Fig. 9, spreadsheet focus 93 is on cell A4. Formula line 92 indicates that cell A4 in this example is assigned a contrast value computed from the histogram array assigned to cell A3. The underlying data display buffer 94 shows the content of the acquired image.

The forgoing illustrates how single method objects co-operate with a variably transparent spreadsheet overlaid on the contents of a data display buffer so as to efficiently and elegantly extend the conventional spreadsheet paradigm to accommodate images and other large data sets.

Conditional Cell Execution

Referring to Fig. 11 , a sequence of four instances 101 , 104, 7, 110 of a portion of an electronic spreadsheet incorporating the improvements of the invention is shown. In the first spreadsheet instance 101 , cell B1 is selected by a user, and is seen to be equal to the logical constant FALSE. After a user selects a "cell state" display mode from a pull-down menu, selecting a cell or plurality of cells causes an enable dialog box 102 to appear as well as causing an associated logical switch expression pop-up window 103 to appear (also see Fig. 12, step 115). The enable dialog box 102, superimposed on the spreadsheet grid, shows that the enable switch expression is equal to the logical constant TRUE, indicating that the expression in B1 will be executed. It is also possible for the user to change the enable switch expression by entering a new enable switch expression 115, as shown in Fig. 12.

Thus, the value of B1 will be FALSE, provided that the enable switch expression that controls the execution of cell B1 is set to TRUE. The implicit conditional expression (including the enable switch expression) that controls the execution of cell B1 is shown in the pop-up window 103 that appears with the enable dialog 102, and contains the logical expression: IF(TRUE,B1=FALSE).

In spreadsheet instance 104, cell B2 is selected and conditionally set equal to the results of the expression evaluated in B3, based on the conditional expression in the pop-up window 106. Enable dialog 105 shows that the enable switch expression is equal to the result of expression evaluated in cell B1. Since B1 is known to evaluate to FALSE, in this case, the expression in B2 will not be executed. The implicit conditional expression associated with cell B2 is shown in the pop-up window 106.

In spreadsheet instance 107, cell B3 is selected and set equal to the result of an image measurement function performed on an external image data buffer. Enable dialog 108 shows that the enable switch expression is equal to the logical constant TRUE, indicating that the expression in B3 will be executed. The implicit conditional expression associated with cell B3 is shown in the pop-up window 109.

In spreadsheet instance 110, cell B4 is selected and set equal to the result of a logical operation performed on the results of the expressions evaluated in B2 and B3. Enable dialog 111 shows that the enable switch expression is equal to the logical constant TRUE, indicating that the expression in B4 will be executed. The implicit conditional expression associated with cell B4 is shown in the pop-up window 112.

Referring to Fig. 11 , a BASIC program fragment 113 shows the BASIC program steps that together are equivalent to the electronic spreadsheet program shown in Fig. 10. One skilled in the art can observe that this BASIC program will exhibit the same behavior, i.e., evaluate expressions, assign values, and transfer program control, so as to achieve the same result as the execution of an improved electronic spreadsheet according to the invention, yet via an entirely different user interface. Fig. 11 emphasizes and illustrates that the improved electronic spreadsheet of the invention can achieve functionality that was previously not possible in electronic spreadsheets, and prior to the invention, was possible only using computer languages like BASIC, COBOL, FORTRAN, and C, for example.

In the forgoing description of Fig. 10, a spreadsheet is programmed to behave in one of two possible ways, depending on cell B1. If the expression in cell B1 is set to TRUE (thereby indicating that the spreadsheet is in training mode in the example of Fig. 10), the spreadsheet executes a training sequence whereby the results of an image measurement executed in cell B3 are saved in cell B2 for future comparisons.

If the expression in cell B1 evaluates to FALSE (thereby indicating testing mode in the example of Fig. 10), the training step associated with the expression in cell B1 is not executed. The expression in cell B4 goes on to perform a test which compares the current image measurement performed in B3 to the measurement saved on the most recent training cycle in cell B2. Figure 13 depicts the spreadsheet main loop. At the start of processing, as indicated by the start bubble, the function calls the DisplaySheet function to display the contents of the spreadsheet. It then waits for one of six input conditions: Move Up, Move Down, Move Left, Move Right, Edit Cell, and Escape. It responds to any of the first four conditions by testing the current row or column count and incrementing or decrementing the respective one of these, all as indicated in Fig. 13. The program responds to an Edit Cell input condition by awaiting an input, testing the validity of the command, and if the comand is valid, then calling the Update CellCmd and EvaluateSheet function, as shown. The program responds to an Escape input by quitting.

Figure 14 depicts the DisplaySheet function. As shown here, the program initializes and increments the indices i and j in order to loop through the rows and columns, respectively, of the spreadsheet. For each combination of those indices within the spreadsheet, the program calls the Print Value function in order to display a cell at the location p][j]. A test is performed during the inner loop to determine whether a location i and j points to the current row and column respectively and, if so, to highlight the cell (via the function Highlight Cell) prior to invoking the Print Value function.

Figure 15 depicts the EvaluateSheet function. At the outset, this function call the EvaluateCell function. Thereafter, as in Figure 14, the EvaluateSheet function initializes and increments the indices i and j in order to loop through the rows and columns, respectively, of the spreadsheet. The inner loop here, rather then invoking the aforementioned Print Value and Highlight Cell functions, tests CellCmd[i][j] for dependency on CellVal[u][v]. If so, it calls the EvaluateSheet function. Otherwise, it proceeds to the next row and column, all as clearly shown in Fig. 15.

Figure 16 depicts the aforementioned EvaluateCell function of Fig. 15 which, at the onset, includes invoking the Split CellCmd[i][j] with the indicated command and arguments. Thereafter, the function evaluates arguments Arg1 -> VaM , Arg2 -> Val2, and so forth, as indicated. The function then executes one of the five respective commands: SET, ADD, MUL, SQRT, and DEL, as shown. The SET command results in the cell value being set to Val1 , as shown. The Add command, results in the cell value being set to the sum of VaM and Val2. Likewise, the MUL command results in the call value being set to the multiplicative product of VaM and Val2. The SQRT command results in the cell value being set to the square root of VaM . The DEL command results in the cell value being set to zero and command being set to NULL. Following execution of the indicated command the function returns.

Spreadsheet with Extensions

Spreadsheet processing begins in the Main Loop, shown in Fig. 17. The Main Loop allocates storage and assigns variables to represent data storage of dimension N by M where, N is the number of rows in the spreadsheet, and M is the number of columns. The array element Formula[u][v] identifies storage for the spreadsheet formula assigned to the (u,v) coordinate of the grid. The array element Data[u][v] identifies storage for data associated with the output of Formula[u][v]. The local variables Row and Col identify the active cell. The active cell is the spreadsheet cell that has the current user interface focus and is the implicit target of editing and special display operations.

After entering at the start bubble, the DisplaySheet function is called to displays the contents of the spreadsheet. The program then waits for one of four input conditions: Move, Edit, Trigger, or Escape. The Move condition indicates that the user wants to change the coordinate of the active cell. The Update function updates the row and column coordinate of the active cell within the spreadsheet grid. The Edit condition indicates that the user wants to modify the formula associated with the active cell. The EditCell function is called to assigns a new formula to the active cell. The ExecuteCell function is called to updates the data associated with the active cell and all dependent cells. The Trigger condition indicates that an external event has occurred. The ExecueTrigger function is called to locate and execute all cells within the spreadsheet dependent on that event.

After handling Move, Edit, or Trigger conditions the main processing loop is restarted. The Escape condition indicates that the user wants to terminate the main processing loop and exit the spreadsheet program.

The DisplaySheet function, shown in Fig. 18, is called from the Main Loop to displays the contents of the spreadsheet. Special emphasis given to information associated with the cell at focus coordinates (u,v). On entering DisplaySheet the DisplayData function is called to handle special display requirements associated with any large data sets that may be assigned to the cell at the focus coordinates. The conventional spreadsheet grid is then displayed by looping through the cell array and, for each cell, calling the DisplayCell function. When the temporary grid coordinates (i j) are equal to the focus coordinates (u,v) the function HighLightCell is called to highlight that position in the spreadsheet grid. Before return, the DisplayFormula function is called to display the formula assigned to the focus cell.

The DisplayData function, shown in Fig. 22, is called from DisplaySheet to manage the graphical display of large data sets contained within the cell at grid coordinate (u,v). The function compares the cell formula with formulas that output large data sets. If a match is found the corresponding data display function is called. For example, if the cell has been configured to acquire an image, the Displaylmage function is called to display the acquired image data. Similarly, if the cell has been configured to compute a histogram, The DisplayHist function will be called to display the accumulated histogram.

The DisplayCell function, shown in Fig. 23, is called from DisplaySheet to control the representation of cell data in the conventional spreadsheet grid. The function compares the cell's formula to determine the type of data stored at (u,v). If the data is a conventional scalar value, that value is displayed at the cell's (u,v) grid location. If the cell contains extended data, the name of the stored data type is displayed cell's (u,v) grid location. For example, if the cell has been configured to acquire an image, the text string "IMAGE" is displayed. Similarly, if the cell has been configured to compute a histogram the text string "HIST" will be displayed.

The EditCell function, shown in Fig. 19, is called from the Main Loop to inputs formulas to be assigned it to the cell at coordinates (u,v). The formula is first tested for valid syntax and argument type. If valid the DeleteCell function is called to release storage associated with the old formula. The old formula is then replaced by the new formula and the AllocCell function is called to allocate and initialize the necessary data structures.

The AllocCell function, shown in Fig. 28, is called from EditCell to manage the allocation and initialization data. The function compares the return type of the formula at (u,v) with extended data types. If a match is found the corresponding data allocation function is called and Data[u][v] is assigned a reference to the allocated storage. For example, if the cell's formula returns image data, the Alloclmage function is called to allocate and initialize memory storage locations associated with image data. Similarly, if the formula is determined to return histogram data, the AllocHist function is called to allocate and initialize storage appropriate to the storage of histograms.

The DeleteCell function, shown in Fig. 29, is called from EditCell to manage the de-allocation of cell data. The function compares the return type of the formula at (u,v) with extended data types. If a match is found the corresponding data de-allocation function is called to release the storage referenced by Data[u][v]. For example, if the cell's formula returns image data, the Deletelmage function is called to de-allocate memory storage location associated with image data. Similarly, if the formula is determined to return histogram data, the DeleteHist function is called to de-allocate data storage associated with the array. The ExecuteCell function, shown in Fig. 20, is called by the Main Loop and, recursively, by ExecuteDependentCells. The ExecuteCell function calls the ExecutueFormula function to update the contents of Data[u][v], and the ExecuteDependentCells function to force the update of all cell dependent on Data[u][v].

The ExecuteFormula Function, shown in Fig. 24, is called by ExecuteCell to manage the evaluation of the cell formula at spreadsheet grid coordinate (u,v). The function views the formula in three parts: the switch expression, the command instructions, and the arguments. If the switch expression evaluates TRUE, the values of arguments are resolved, and a determination is made as to whether the cell is conventional or extended. If the cell contains a conventional spreadsheet formula returning a simple scalar the ExecuteConventional function is called to complete the evaluation of the cell's value. If the formula results in a large data set, such as an image or a histogram, the ExecuteExtended function is called to complete the evaluation of the cell's internal data structure.

The ExecuteConventional Function, shown in Fig. 26, is called from ExecuteFormula to manage the evaluation of conventional spreadsheet commands. The input command is compared with a list of valid commands, if a match is found the corresponding operation is executed updating the cells data value. For example, the conventional spreadsheet conditional command "IF" interprets the first argument as a condition by which to decide whether to set the cell's Data[u][v] to either the second or third argument. Similarly, the command "ADD" causes Data[u][v] to be set to the sum of the first and second arguments.

The ExecuteExtended Function, shown in Fig. 27, is called from ExecuteFormula to manage the evaluation of extended spreadsheet commands. The input command is compared with a list of valid commands, if a match is found the corresponding operation is executed updating the cells data value. For example, the "ACQUIRE" command invokes the Acquire function, which causes a new image data to be copied from an external source to the storage location identified by the cell's Data[u][v] variable. Similarly, the "HIST" commands invokes the Histogram function which causes a histogram of the image data identified by first argument to be computed. The histogram array is stored in the locations identified by the cell's Data[u][v] variable.

The ExecuteDependentCells Function, shown in Fig. 25, is called by ExecuteCell and by ExecuteTrigger. The ExecuteDependentCells function locates and executes spreadsheet cell whose output is a function of the data stored in the cell located at spreadsheet grid coordinate (u,v). This function searches the spreadsheet for cells whose input expressions reference the cell at (u,v). If a match is found, a test is made to determine is the reference is circular, that is, if the formula is ultimately dependent on itself. If the reference is determined to be noncircular, the ExecuteCell function is called to execute the dependent cell. Note that the call to ExecuteCell is recursive, since ExecuteCell will call ExecuteDependentCells to evaluate subsequent dependencies.

The ExecuteTrigger function, shown in Fig. 21 , is called from the Main Loop to search the spreadsheet for a trigger formula. If a matching trigger cell is found, its Data[i][j] variable is set TRUE and function ExecuteDependentCells is called to update any dependent cells. After all dependent cells have been executed the trigger cells value is set back to FALSE, to prevent the asynchronous execution of cells with explicit trigger dependencies.

Further understanding of the illustrated embodiments may be obtained by reference to the Appendix filed herewith, which discloses preferred methods and apparatuses for machine vision interfaces and environments.

Other modifications and implementations will occur to those skilled in the art without departing from the spirit and the scope of the invention as claimed. Accordingly, the above description is not intended to limit the invention except as indicated in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A user interface for programming machine vision applications, comprising: an image window for displaying an image to be analyzed; a spreadsheet for analyzing the image, displayed with the image window.

2. The user interface of claim 1 , further comprising: a control pad, at least for navigating over the spreadsheet.

3. The user interface of claim 1 , further comprising: a plurality of menus having various software tools.

4. The user interface of claim 1 , further comprising: a palate of alpha-numeric characters for input into the spreadsheet.

5. The user interface of claim 1, wherein the spreadsheet is semi-transparent, such that the image can be seen through the spreadsheet.

6. The user interface of claim 1 , wherein the spreadsheet is adapted to perform conditional cell execution.

7. The user interface of claim 1 , wherein the spreadsheet includes functions that can operate on values stored over an interval of time.

8. The user interface of claim 2, wherein the control pad indicates a particular cell of the spreadsheet, thereby resulting in a change in the appearance of the image displayed.

9. The user interface of claim 1, wherein the spreadsheet includes cells adapted to buffer outputs, thereby providing circular reference for recursive operation.

10. The user interface of claim 1 , wherein the spreadsheet includes cells that can be assigned arbitrary depth in a time dimension.

11. The user interface of claim 1 , wherein complex vision objects are instantiated as a single cell in the spreadsheet, the single cell having data that is accessible through a plurality of member functions.

12. The user interface of claim 1 , wherein the spreadsheet includes cell that are adapted to store two dimensional image arrays that result from image processing operations.

13. The user interface of claim 1, wherein the spreadsheet includes image buffers used for intermediate processing, the image buffers being automatically released once dependent operation are satisfied.

14. The user interface of claim 1 , further comprising: a spreadsheet mask for obscuring portions of the spreadsheet, leaving portions unobscured that can be used to control operation of the spreadsheet.

15. An electronic spreadsheet for processing a continuous data stream, the electronic spreadsheet having a plurality of cells, the improvement comprising: a clocked delay line object, the clocked delay line object being instantiated and assigned to a cell of the electronic spreadsheet; an external data input function adapted to provide a clock signal to the clocked delay line object; and a spreadsheet function adapted to provide random access to data stored in the clocked delay line object.

16. The electronic spreadsheet of claim 15, wherein the clocked delay line object is parameterized by a plurality of arguments, including: an input data stream; a clock source; a number of elements in a shift register buffer; and a clock divider.

17. The electronic spreadsheet of claim 15, wherein the clocked delay line object further includes: a method for clocking data; and a method for access to data stored in the clocked delay line object.

18. The electronic spreadsheet of claim 15, wherein the clocked delay line object is adapted to trigger the execution of the shift register function of the delay line object upon receipt of a clock signal provided by the external data input function.

19. The electronic spreadsheet of claim 15, further including a spreadsheet function that performs analysis of a data interval stored in the clocked delay line object.

20. A method for using an electronic spreadsheet for processing continuous data streams over an interval of size greater than one, the method compπsing: instantiate and assign a dockable delay line object having data storage to at least one cell of the electronic spreadsheet; provide a clock signal to the delay line object; randomly access the data storage over an interval of size greater than one; and process the data so-accessed.

21. A method for processing continuous data streams over an interval of size greater than one using an electronic spreadsheet having clocked delay line objects instantiated in the cells of the electronic spreadsheet, each delay line object having a delay line buffer, a source of clock signals, and spreadsheet functions that support selectable access to data stored in a delay line of the clocked delay line object, the method comprising: assign to a first cell an external input function adapted to access a data source external to the spreadsheet; assign to a second cell an instance of a delay line object; assign to a plurality of cells a respective plurality of tap functions, each tap function adapted to access the delay line buffer of the delay line object at a selectable clock cycle to provide a respective plurality of data elements stored in the delay line buffer; and assign to at least one cell a function adapted to process the plurality of data elements upon each clock cycle.

22. An electronic spreadsheet having a plurality of cells, the improvement comprising: a single method object, adapted to be instantiated in at least one of the spreadsheet cells, and adapted to provide internal storage and member functions, the member functions being adapted to access the single method object and return a single value; a data display buffer, the data contents of which are displayed under a variably transparent spreadsheet grid; and means for selectively displaying the data content of the single method object in the data display buffer corresponding to a selected spreadsheet cell.

23. A method for selectively displaying large data sets in an electronic spreadsheet having a plurality of cells, the method comprising: instantiating a single method object in each of a plurality of the cells of the spreadsheet, each single method object being adapted to provide internal storage for storing a large data set; displaying the large data set of the single method object corresponding to a selected cell of the spreadsheet; and displaying in superimposed relationship with the large data set a variably transparent spreadsheet including the selected cell.

24. A user-interface method for selectively displaying machine vision images stored in an electronic spreadsheet having a plurality of cells, the method comprising: instantiating a single method object in each of a plurality of the cells of the spreadsheet, each single method object being adapted to provide internal storage for storing a machine vision image; selecting a cell from the plurality of cells; displaying the machine vision image stored in the single method object corresponding to the selected cell; and displaying in superimposed relationship with the machine vision image a transparent electronic spreadsheet including the selected cell.

25. The user-interface method of claim 24, wherein the transparent electronic spreadsheet is adjustably transparent.

26. The user-interface method of claim 24, wherein the selected cell is selected using a hand-held controller.

27. The user-interface method of claim 24, wherein the selected cell is selected using one of a standard keyboard and a mouse.

28. The user-interface method of claim 24, wherein the machine vision image includes a superposition of an object image, and a graphical representation of an analysis of the object image.

29. The user-interface method of claim 28, wherein the analysis of the object image is a histogram of the object image.

30. An improved electronic spreadsheet having a plurality of cells, each cell having an executable expression, the improvement comprising: for each cell of said plurality of cells, a cooperative implicit conditional execution wrapper having a logical switch expression, the implicit conditional execution wrapper combining the executable expression with the logical switch expression so as to enable or disable the executable expression of the cell in accordance with the value of the logical switch expression.

31. The electronic spreadsheet of claim 30, further comprising; a user interface for selecting a cell or range of cells.

32. The electronic spreadsheet of claim 30, further comprising: a user interface for assigning a logical switch expression to an implicit conditional execution wrapper.

33. The electronic spreadsheet of claim 30, wherein the logical switch expression is a legal spreadsheet expression that can be evaluated to logical TRUE or logical FALSE.

34. The electronic spreadsheet of claim 33, wherein if the logical switch expression evaluates TRUE, the cell's corresponding expression is evaluated, a new result value is returned, and the cell's internal result buffer is updated.

35. The electronic spreadsheet of claim 33, wherein if the logical switch expression evaluates FALSE, the cell's expression is not evaluated, and the cell's result buffer is returned unmodified.

36. The electronic spreadsheet of claim 33, wherein individual switch expressions are initialized to a constant logical TRUE.

37. The electronic spreadsheet of claim 31 , wherein the user interface includes an interactive assignor that interactively assigns a switch expression to any cell or range of cells within the spreadsheet.

38. The electronic spreadsheet of claim 30, wherein the user interface includes a cell selector that selects an arbitrary set of cells, and an interactive assignor that selectively assigns a logical switch expression to each set of cells so-selected, so as to program a desired behavior.

39. A method for programming conditional execution of program statements in an electronic spreadsheet having a plurality of cells, the method comprising: associating an implicit conditional execution wrapper with each cell; selecting at least one cell; and entering a logical switch expression into the implicit conditional execution wrapper associated with the selected at least one cell.

40. An apparatus for programming conditional execution of program statements in an electronic spreadsheet having a plurality of cells, the method comprising: means for associating an implicit conditional execution wrapper with each cell; means for selecting at least one cell; and means for entering a logical switch expression into the implicit conditional execution wrapper associated with the selected at least one cell.