US20040073546A1 - Data exploration system and method - Google Patents

Data exploration system and method Download PDF

Info

Publication number
US20040073546A1
US20040073546A1 US10/629,125 US62912503A US2004073546A1 US 20040073546 A1 US20040073546 A1 US 20040073546A1 US 62912503 A US62912503 A US 62912503A US 2004073546 A1 US2004073546 A1 US 2004073546A1
Authority
US
United States
Prior art keywords
output
data
input
query
column
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/629,125
Inventor
Michael Forster
Lindsey Spratt
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hedron LLC
Original Assignee
Hedron LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hedron LLC filed Critical Hedron LLC
Priority to US10/629,125 priority Critical patent/US20040073546A1/en
Publication of US20040073546A1 publication Critical patent/US20040073546A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99931—Database or file accessing
    • Y10S707/99932—Access augmentation or optimizing
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99931—Database or file accessing
    • Y10S707/99933—Query processing, i.e. searching

Definitions

  • RDM relational data model
  • B+ tree an access method that requires multiple accesses to storage (e.g., memory or disk) to obtain a given record.
  • Some of the accesses are to reference structures that are used to effectively “point to” the data of interest (e.g., indices or hierarchies of linked lists).
  • reference structures can get so large that portions of the structure must reside on disk.
  • a given request for a database record may involve multiple disk storage requests.
  • the database operation algorithms are tightly bound to the access method. That is, the algorithm itself has been optimized to the access method and is thus dependent on the existence of the access method. Much of the literature on database performance explicitly or implicitly assumes the existence of such access methods.
  • Preferred embodiments of the invention provide a system for, and method of, exploring relationships in data stored in a computer readable medium.
  • a query is received having at least one operator chosen from a set of operators that includes relational operators and having at least one input and output associated with the operator and defined as a table having at least one domain having a type associated therewith.
  • the query is transformed into a set program having at least one operation structure, corresponding to the operator and having logic for type-independently performing an operation, corresponding to the operator, and having a data relation structure, cooperating with the operation structure, for handling all data access and storage associated with the operation.
  • FIG. 4 shows query evaluation logic according to a preferred embodiment of the invention
  • FIG. 5 shows outer language evaluation logic according to a preferred embodiment of the invention
  • the query constructed is a graph of operations in which “tables” (i.e. depicted sets either DSets or DPSets, discussed below) flow between operators.
  • the outer language follows a functional, applicative paradigm that, among other things, supports parameterized queries and recursion.
  • the query (or program) may include one or more constraints between columns within a table, columns of different tables, or a column and a scalar. As will be explained below, these constraints imply row selection if they are within a table, or a join operation (e.g., inner or theta join) if they are between tables.
  • the primitives (and particularly their semantics) and the algorithms used by the internal engine to execute the primitives have been found to be particularly efficient. This efficiency is observed both in direct performance (e.g., the algorithmic efficiency of an Inner Join implementation) and in the system's amenability to both local and global optimizations, exploiting certain characteristics of the primitives and algorithms. For example, some optimizations exploit the language's and algorithm implementation's characteristic of preserving the “sortedness” of data across an operation. More specifically, this preservation of sortedness may be exploited in detecting and removing redundant Sort operations within a query.
  • the server may send status messages to the client, indicating the progress of the solution process. Once it is solved, the user may cause the client to present the results.
  • a preferred embodiment of the server 104 uses the following data structures for organizing data in server memory and/or on attached storage 106 , such as disks or tape.
  • An “XSet” is used directly by the server as a unit of access and manipulation and also in the construction of compound structures.
  • An XSet is a type-less array of data storage units (e.g., bytes). Each XSet corresponds to a file that provides persistence for that XSet's data. Each XSet is arranged as shown in FIG. 2. (There is an exception to this arrangement when an XSet is included in a compound structure called a DPSet, described below.)
  • the header includes information indicating the number of elements in the XSet (i.e., the “cardinality”); the width of each element in data units; the size of the header; the identity of the XSet; the version number of the XSet; and other state information.
  • a “Depicted Set” (or “DSet”) is a compound data structure that combines an XSet and a Depictor.
  • DSets correspond to the entities used by users. That is, these are the structures users typically use as inputs to queries and that they view.
  • a user typically wants to impute some interpretation on the data being explored and might specify, as a first analysis, that the Depictor should effectively mirror the interpretation supplied by a database schema of the data being explored.
  • the separate structures allow users to easily modify a Depictor (and consequently a DSet) and thus cause the system to analyze the data differently, e.g., consider a first column as text, and then later as an integer.
  • an exemplary embodiment has a given column in a DSet having the same width in each row of the DSet.
  • Depictors are also constructed by the software logic.
  • an output of an operation will be a DSet, the Depictor of which will be formed either from user input or from software logic using default Depictor forming rules (e.g., output type equals left most input as viewed on screen.)
  • a DSet may be represented in either “extensional” form or in “intensional” form.
  • An extensional DSet is implemented as contiguous storage in which DSet members are stored sequentially, preferably without gaps.
  • the first column of the DPSet holds the byte offset of the start of the corresponding row in the XSet file.
  • the rest of the columns specify the width in bytes of the columns of the DPSet on a row-by-row basis.
  • the Depictor is a DSet which has one row for every column in the DPSet. Each row in the Depictor DSet defines the properties of the corresponding column in the DPSet, including column name, column type identifier, column type width, column type format, and storage information about a given domain.
  • DPSets are advantageous in that they allow conversion and use to be postponed so that only the data that is actually needed will be read from the file.
  • a “QRSet” is a DSet representing a query.
  • the Depictor specifies the entities used to represent a query, for example, including the part element type, its identification number, and the like.
  • the Depictor for a QRSet has information about the parts list which represents a function (more below).
  • the XSet portion of the QRSet's DSet holds the corresponding values, e.g., the value of the identification number.
  • a QRSet is an “intensional” form of the DSet eventually solved, or derived, from the query. QRSets, being DSets, may be queried.
  • the tables are maintained, or “cached,” and available to be used in certain optimizations during query evaluation and/or solving.
  • the workspace managers consider the amount of memory available to the workspace and allocate that memory to tables. Preferably sets and tables are kept in “extensional” form; that is, the data itself resides in the workspace.
  • extensional form that is, the data itself resides in the workspace.
  • dynamic run-time usage can require the workspace managers to effectively “swap out” the extensional form to storage but retain the “intensional” form, i.e., the query sub-expression that resulted in the data.
  • usage histories are maintained and considered by the workspace managers in determining which sets to swap out or delete.
  • the query evaluation system 310 transforms the user-created query into a semantically equivalent form, constructed of operation “primitives.” These primitives are operations that are executable by a server engine. In short, the primitives generally have more restricted operation assumptions than the user query (e.g., input arguments must be in a prescribed order) and are set-theoretic, rather than relational.
  • the outer language though having relational model characteristics, also contains set theoretic aspects. Thus queries may be expressed in a form emphasizing set membership. As a consequence, the outer language has both algebraic aspects and more powerful calculus aspects.
  • logic 502 to normalize the outer language expression
  • Logic 504 also detects when the constraints used in a query refer to DPSets and in such cases injects into the kernel program a corresponding Select Column and Select Row operators having the DPSets as input. Any other operators having a DPSet as an input are converted so that a DSet is formed from the DPSet and so that the kernel operator refers to the DSet.
  • Logic 504 also ensures that certain input ordering is maintained (e.g., that table constants come after variables), transforming the operator if necessary. Logic 504 also transforms nested disjunctions (i.e., ORs).
  • the outer language allows queries to be created that have one or more constraints between (1) columns within a table, (2) columns of different tables, or (3) a column and a scalar value. These constraints imply row selection if they are within a table, or some kind of join (e.g., Inner or Theta) if they are between tables.
  • logic 504 compares elements in the parts list with pre-programmed patterns (not to be confused with the patterns sent to client to describe how an operator block should be drawn). The recognition of such implied operating use conventional algorithms.
  • Logic 504 also transforms function calls (i.e., Evaluate) and expands “macro” operations. The actual instantiation of a function call is not performed until the query is being executed by the internal engine (more below), at which time the function call is identified by a query name.
  • macro expansion one exemplary embodiment implements Aggregate-By-Group and Concatenate Strings as macros.
  • the macro is transformed into corresponding kernel operators that select rows for each group of constraints on the Aggregate-By-Group operator; Cross Products the name for that group with the selection; Unions the named selections; and Aggregates-By-Key the union adding the group name as the first key column.
  • the Concatenate String example the operator is expanded into a graph of Binary-Concatenate-String operators.
  • the logic 506 to normalize the kernel version of the outer language expression analyzes the “data flow” of the query to verify that each operation potentially has enough information to be executed. This is done to determine whether the query is legitimate from a data flow perspective.
  • a new QRSet is created and organized according to the data flow dependence as suggested by the query.
  • the new QRSet also has its identifiers and names normalized to facilitate further processing.
  • the logic 508 to convert the normalized kernel version to an “inner program” performs another set of optimizations and transformations to produce an inner program, or QRSet (this is not yet the set of operation primitives).
  • Each operator in the outer program has at least one corresponding inner program operator.
  • the operation assumptions between the outer and inner operations are not necessarily identical.
  • the Inner Join operator of the inner language assumes that its inputs are sorted, whereas the same operator in the outer language makes no such assumption. Consequently, the conversion logic 508 detects operations that do not have semantically equivalent inner forms and makes the necessary changes.
  • this transformation would include the analysis to determine whether the inputs were necessarily sorted (e.g., because of a prior operation) and if the inputs were not necessarily sorted, the logic would insert the necessary sort operations into the QRSet.
  • Logic 508 then analyzes the inner program to detect operators that correspond to primitive operators that require their inputs to be sorted and upon such detections inserts the appropriate sort operation according to the corresponding keys. More specifically the logic 508 detects instances of Inner Join, Intersect, Relative Complement, Symmetric Difference, Union, Group-And-Count, Sum over Groups by Key, Max, Min, Average, and Standard Deviation, Row Mode, Median, Count Unique Rows, Substitute By Key, and Relative Complement by Key and injects into the QRSet, before the operators themselves, the appropriate sort, corresponding to the keys in these operators.
  • the sort takes all keys in key order as sort key inputs and then uses all other domains on the operator needing sorting as the non-sorted inputs to the sort by key operator.
  • the outputs of the sort by key operator will go to the inputs of the questioned operator (such as sum over groups) both keyed and unkeyed as appropriate.
  • Logic 508 detects and removes redundant Select Columns operations, i.e., selection operations that perform a selection already being done by a necessarily preceding section of the program.
  • logic 602 to transfer the inner language expression to an optimized form
  • Logic 602 performs yet more optimizations on the query. These optimizations may alternatively be performed in the outer evaluation system 402 .
  • Logic 602 combines sorts. More specifically, it finds pairs of sort operators that sort the same input table such that the column sequences defining the sort orderings are equal up to the length of the shorter of the two column sequences defining the key orderings.
  • Logic 602 then replaces these sorts by deleting the second sort and substituting, for the first sort, a Sort by Key that has the longer of the two column sequences and it includes all of the data columns present in either of the two sort operators. This sort subsumption is valid because the implementation of all operation primitives preserve the “sortedness” of data.
  • the most restrictive of the less restrictive forms could be used as a starting point and be substituted into the query and operations may be injected to complete the intension, i.e., to further restrict the substituted form into the form sought.
  • a table may exist having “all males in Kansas.” This table may be used (provided appropriate other data exists) in the expression and further restricted. Finding all males in Wichita from this table would be less costly than finding all males in Wichita Kansas from the original data sources. The optimization logic finds such tables and uses them and inserts the appropriate subsequent restrictions.
  • the logic 606 receives the oper set and invokes the DSet operation execution system 406 , as the operators become “ready” to execute.
  • “ready” means from a data flow scheduling perspective of which the inputs are valid.
  • This logic 606 invokes the DSet system 406 to schedule the primitives for execution according to a data flow scheduling algorithm, in which an operation is “fired” when all of its necessary inputs are detected as being “ready”. Inputs are ready when they exist in extensional form in the workspace.
  • a given primitive operation is “fired” to execute as its own execution thread on the server, thus increasing the execution parallelism further.
  • the scheduling of opers within an oper set are implemented with a separate event queue. This event queue has specific logic to handle the control structures associated with an oper and to detect when an oper is ready to be executed.
  • the DSet operation execution system 406 is shown in more detail in FIG. 7.
  • the DSet operation execution system 406 includes set up logic 702 ; internal engine 704 ; workspace interaction logic 706 ; depictor system 708 ; RShip construction logic 710 ; type access system 712 ; and memory utilities 714 .
  • the set up logic 702 ensures that the DSets involved are ready, i.e., with open files. Under one embodiment this logic will determine whether the input DSets exist in extensional form and if not will cause them to be so. Another embodiment however performs the above just before global optimization is performed by inserting the appropriate operation blocks before the blocks requesting the DSets.
  • the set up logic 702 cooperates with the outer evaluation system 402 , as outlined above.
  • the set up logic 702 detects the existence of Evaluate opers and sends a solve event to the outer evaluation system 402 so that it may evaluate the instantiations of the corresponding query.
  • the outer evaluation system 402 will instantiate and submit it to full outer and inner language evaluation, as necessary. If a QRSet exists, that will be the starting point of function instantiation.
  • the set up logic 702 is also responsible for re-expressing a given oper from an operation with input and output DSets to a primitive operation that operates in conjunction with a corresponding RShip and in which the data is in XSet form.
  • RShips are used to control all data access and data storing in result sets. Under a preferred embodiment they are also used to convert input data (as needed) from one type to output data of another type; that is, conversion during transfer of elements. Under a preferred embodiment they are also used to convert data in relator operations, both in a comparative sense and in an arithmetic sense; for example, when relating a floating point an integer.
  • the RShip will abstract the algorithm (for the most part) from the data types of the data involved. More specifically, the set up logic 702 invokes the depictor logic 708 to initiate the construction of an RShip for a corresponding oper.
  • the depictor logic 708 analyzes the Depictors for input and output DSets corresponding to the oper and also analyzes the oper to determine whether it is for a relator and constructs the corresponding RShip structure to map the data conversions involved with data transfer and/or comparison aspects involved with the operation.
  • An operation may require more than one RShip for its execution. For instance, the Add operator has an RShip that describes the domain relationship between two operands and another RShip for transferring the results.
  • an inner join involves both data transfer and comparison operations.
  • the RShip in addition to containing mapping information, will contain corresponding control structures involved with the conversion. For example, the RShip may contain high-level control logic for iterating over the domains involved. The RShip structure thus has the mapping information and control logic. It relies on type access logic 712 to perform the actual conversions of data.
  • a select may have an input of a first type and an output of a second type.
  • the RShip structure in such case will construct n-tuples indicating the corresponding mapping.
  • the operation involved may be a relator operation such as Equal To and may involve inputs of disparate types.
  • the RShip in such case, will map the appropriate conversions needed for the comparison operation, as outlined above.
  • loop unfolding techniques may be used to pre-load RShips and delimeters.
  • the engine 704 includes oper algorithm logic for each of the inner language operations.
  • the implementations follow standard expressions of the algorithms (e.g., by Knuth) but are preferably type-less, avoid conditional execution, and are maximally abstracted from the data.
  • the algorithms use a single pass over the data.
  • the algorithms are implemented to tell a corresponding RShip to compare and transfer data as needed by the operation. In some instances the data will be transferred to a buffer so that it may be further manipulated by the algorithm logic. In other instances, the algorithm logic will not need to do any further manipulations (e.g., Inner Join and Select).
  • the set handling engine 704 includes logic allowing multi-threaded access to files. This logic is included because the operations are handled by multiple threads. Some operating systems do not permit transparent multiple access to a file from different places in multiple threads. To address this, logic references Xset files and includes access counters to interface file access. The multiple threads access the engine's interface logic rather than the underlying file system to see if files are open and to cause the corresponding file operations. (Mutex controls are used to maintain coherency of the interface logic.)
  • the engine memory utilities are used for accessing and allocating memory. For example, they may be used to allocate contiguous physical chunks of memory. The buffer allocation is mapped to table rows and the services may be used to access the desired row of a table, while hiding the mapping details. Other memory utilities (lower level) are preferably used to abstract type conversions, such as endian conversions for floating point data.
  • the workspace logic 708 cooperates with the workspace system, and is used both in allocation and committal of DSets. When an operation is being executed, the logic needs to allocate DSets for outputs. The workspace logic 708 will make such requests to the workspace system. When an operation has finished execution, thus defining the data for the corresponding output DSet, it is “committed” to the workspace indicating it is valid. At this point it may be used for viewing and/or global optimizations, and as inputs to operations, among others.
  • the data types for both the outer and also the inner programming languages include Int8; Int16; Int32; Int64; Uint8; Uint16; Uint32; Uint64; FloatS (32-bit floating point number); FloatB (64-bit floating point number); Float precision (64-bit floating point with specified precision) StrFA (common string type); StrXA (DPSet string type for “delimited” strings); Byte ( basic string type used to implement both StrFA and also StrXA); Time; Date; and DateTime.
  • non-kernel operators are “macro” operators that are converted to program fragments built out of kernel operators.
  • the non-kernel constraints are replaced by a combination of kernel operators and modifications of existing portions of the query program.
  • Some of the non-client-visible kernel operators are used by the server to implement some of the non-kernel operators and constraints (e.g. Binary Concatenate Strings is used to implement Concatenate Strings, Evaluate is used to implement Query, Generate Strings from Regular Expression is used with Inner Equijoin to implement the Regular Expression Match constraints).
  • Each cell of Output contains the absolute value of a corresponding cell in Input.
  • In1 is “void” then Output is derived from In2. If In2 is “void” then Output is derived from In1. If In1 and In2 are both non-void, then Output is union(In, In2).
  • Output has the same number of columns as Input. If Key is empty, then Output has one row where each cell of the row is the average value of the cells for the corresponding column of Input. If Key is not empty, then the rows of Input are grouped according to Key and Output has one row for each key group where each cell of a row is the average value of the corresponding group of cells for the corresponding column of Input.
  • Output contains one column. Output contains the same number of rows as the implied input table of Left and Right. Each cell in Output contains 1 if the conjunction of Left/Relator/Right is true for the corresponding row of the input table, otherwise it is 0.
  • Input contains two or more column reference items, not necessarily from the same table. Output has the same number of columns as the items in Input. Output has the same number of rows as the smallest of the input tables associated with the items of Input. Each cell in a row of Output is copied from the cell of the corresponding Input column reference.
  • Input contains one or more scalar items. Output contains one column. Output contains one row for each item in Input. Each cell of Output is the value of the corresponding item from Input.
  • Input contains one or more column references, all from the same table.
  • Output contains one column for each item of Input.
  • Output contains one row for each row of the input table.
  • Each cell of Output contains the lower case representation of the string representation of the corresponding input column and row cell's value.
  • In1 has one column reference.
  • In2 has one column reference. If In1 and In2 are different tables then they are “coerced” to a single table (by simulating a concatenate columns of the two tables).
  • Output contains three columns: Pearson's correlation coefficient, the confidence level, and Fisher's Z. Output contains one row with the three columns calculated from the two input columns.
  • Input contains a column reference. Output contains one column. Output contains one row. The cell of Output is the count of the number of rows in column referred to by the item of Input.
  • Input contains a column reference. Output contains one column. Output contains one row. The cell of Output is the count of the number of unique values in the column referred to by the item of Input.
  • In1 and In2 contain one or more column references.
  • Output contains one column for each item in In1 and In2.
  • N 2 be the cardinality of the table of In2.
  • Output contains one N 1 *N 2 rows.
  • the rows of Output are all possible combinations of rows of the two input tables, with only the columns identified in In1 and In2 and ordered and typed as specified in the Output definition.
  • Data has one or more column references.
  • Key has zero or more column references from the same table as Data's column references.
  • Output has the same number of columns as Data plus one “count” column.
  • InA has one or more column references.
  • InB has the same number of column references as InA.
  • Output has the same number of columns as InA.
  • Output has the one row for each combination of rows from InA and InB such that InA equals InB.
  • the cells in a row of Output have the same values as the corresponding cells in InA.
  • InA has one or more column references.
  • KeyA has one or more column references to InA's table.
  • KeyB has the same number of column references as KeyA.
  • Output has the same number of columns as InA. Output has the one row for each of row from InA such that there does not exist a row of the table for KeyB where KeyA equals KeyB. The cells in a row of Output have the same values as the corresponding cells in InA.
  • Input has one or more column references. Output has the same number of columns as Input. Output has the same number of rows as Input. Each cell of Output is the length of the string representation of the corresponding Input cell's value.
  • Input is one or more items. In a function instantiation, these items must all be column references. In a function definition, these items may be a mixture of column references and parameter items.
  • Output has the same number of items as Input.
  • the Output table has the same number of rows as the Input table. Each cell in a row of the Output table has the same value as the corresponding cell in the Input table.
  • the only differences between the Input and Output tables are the column order, column names, column data types, and table names.
  • the cells of Output are calculated as above, but the single scalar row of Source is reused for each Search row. If all of the items of Search are scalar, then Source must contain column references. Output has the same number of rows and columns as Source. The cells of Output are calculated as above, but the single scalar row of Search is reused for each Source row.
  • In1 has one or more column references. In2 has the same number of column references as In1. If In1 and In2 are from different tables then the “effective input” table is the concatenate columns of In1 and In2. Output has the same number of columns as In1. Output has the same number of rows as the “effective input” table, minus 1.
  • InA has one or more column references.
  • InB has the same number of column references as InA.
  • Output has the same number of columns as InA.
  • Output has the one row for each of row from InA such that there does not exist a row of the table for InB where InA equals InB.
  • the cells in a row of Output have the same values as the corresponding cells in InA. See also Keyed Relative Complement.
  • Data has one or more column references. Size, Start, and Step each have one scalar item. Output has the same number of columns as Data. Output has rows selected from Data based on the values in Size, Start, and Step. Size can be specified as derived from the Start and Step values, as a percentage of the number of rows in Data, or as “as many as possible up to” a specified maximum value. Start can be specified as either to be picked randomly or using a given row index (a negative index counts backward from the end of Data). The Step value can be specified as either: calculated base on the Size and Start values, to be picked randomly, or else using a specified step value (a negative step value steps from the end of Data back toward the beginning).
  • Input has one or more column references.
  • Output has the same number of columns as Input has items.
  • Output's result table has the same number of rows as the table identified by Input's column references. Each row in Output is based on the corresponding row of the Input table.
  • Each cell's value in Output's table is the (converted if necessary) value of the corresponding cell in Input's table.
  • Output has the same number of columns as Data. Output has one row for every row in Data that satisfies the constraints of Left/Relator/Right.
  • Key has zero or more column references from the same table as Input. Same definition as Sort All Ascending, except if Key has one or more items in which case the sorted version of Input's table is created using the columns referenced by Key to define the sort order instead of the columns referenced by Input.
  • the processed row of InA takes the values of the (first) matching InB row. If there is no matching row of InB, then the processed row of InA takes the values of the original InA.
  • each Output cell value is the string representation of Source minus the first occurrence of the string representation of Search.
  • Control has one column reference.
  • Data has one or more column references. Only Out1 and Out2 are defined in the non-kernel/client-visible use of this operator. All Outi have the same number of column definitions as Data has column references. If there is a cell value convertible to integer N in the Control column, then OutN's table is defined the same as a select_columns(Data) ⁇ OutN. If there is a cell value convertible to integer 0 in the Control column, then this is interpreted the same as an integer of K (i.e. the last Out connector). For all J such that there is no cell value convertible to it (according to the above, including the special handling of 0), Outj's table is the special value “void”. The void table has no columns or rows.
  • InA has one or more column references.
  • InB has the same number of column references as InA.
  • Output has the same number of columns as InA.
  • Output's table has the one row for each row from InA's table and each row from InB's table. The rows of Output's table have the same relative ordering as the rows of InA's and InB's tables, sorted by the “keys” defined by InA and InB items, respectively.
  • Pattern is one or more scalar items.
  • Output's table is an intensional table of one column. Output's table has many rows as there are strings that can be “generated” from the regular expression given on the Pattern connector: a string is generated from a regular expression R if that R matches that string under the “regular expression matching” algorithm. Since there may be infinitely many strings that could match a given regular expression, Output's table cannot (in general) have an explicit “extensional” representation. For an intensional table only a few operations may be defined. One of these is always “is_member(X)”, where the intensional table “system” can always answer yes or no to such a question for a particular intensional table. This mechanism is currently used (through the RShip system) to implement inner-join restriction of a “normal” extensional table against an intensional table to produce a new extensional table.
  • Output has one column for each item in Data, plus a column for “group names”. If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to all of the cells in the column corresponding Data item of the selected data for that group. If Key has one or more items, then the selected data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups.
  • Data and StatOps have the same number of items.
  • the StatOp item corresponding to a Data item indicates the statistic (e.g. min, max, average) that is to be calculated for the column indicated by that Data item.
  • Output has one column for each item in Data. If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to the data of all of the cells in the column of the corresponding Data item. If Key has one or more items, then the data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups.
  • Input contains two or more items. All of the column references among these items must be to the same table. There may be any number of scalar items interspersed among column reference items.
  • Output contains one column Output contains the same number of rows as the input table implied by the column references of Input. Each cell of Output is the concatenation of the string representations of the values of the specified columns of the corresponding input table row, in the order given in Input and with the scalars if Input interspersed as specified in Input.
  • Set-Theoretic Programming Language a query in the preferred embodiments is a form of set language program.
  • the query language is diagrammatic, supports parameterized queries, recursive functions, and decision arbitration components.
  • Second Order Logic Theorem Proving a preferred embodiments accomplish a second order logic theorem proving system. This includes theorems for translating from a high level form into a low level form, theorems that recognize and properly set up the low level executable form, theorems that optimize sequences of instruction to ensure proper preparation prior to execution of certain types of instructions and to remove redundant instructions and perform local optimizations based on set-theoretic identities, and theorems for global optimization, such as identifying cache results that may be used for a current query.
  • Adaptive Physical Design the extensional form of sets forms the physical “building blocks” of the data base. As the extensional forms are added or deleted from the database through the use of the system, the physical design of the database changes. These changes reflect optimizations for disc space and set or structure utilization.
  • Types may be composed and decomposed to create new types or break up domains or compose domains within tables and between tables. For instance, a date may be decomposed into three integers, and likewise, three integers may be composed into a date. A name may be decomposed into three name part strings and three strings may be composed into a name.
  • All aspects of the server may be utilized in queries: for instance, a user may query the system concerning the current set of active clients, number of sets which are greater than three million records, or how many sets have been created within the last three weeks. Most important, queries may be used to question the system about the form of the database as it exists, the domains and sets present, and the relation of names, types and size of sets as a match for possible operations. The results of these query functions may be used to guide query execution, to post to the user such things as consistency and normalization information about tables and functional dependency information and possible join opportunities between tables and with end tables.
  • each serve may be a client to another server allowing databases to be shared among different machines and to exist on multiple machines.
  • Database composition this allows users to create databases and then share the results of analysis and optimization with other databases. The sharing occurs through inheritance. The sets that exist in a “super” database are available to all databases that are derived from the super database.
  • Each operator and function call includes table and column definitions for each of its output sets, eliminating the need for separate from the query schema definitions processing or storage; i.e., eliminating the need to maintain and work with schema definitions.
  • server logic is divided for execution on multiple, hierarchically-arranged servers.
  • server 104 may be implemented on multiple computer nodes to improve performance, distribute processing, improve fault tolerance, or the like. Skilled artisans will appreciate, upon reading the disclosure, that alternative data arrangements may be substituted.

Abstract

Preferred embodiments of the invention provide a system for, and method of, exploring relationships in data stored in a computer readable medium. A query is received having at least one operator chosen from a set of operators that includes relational operators and having at least one input and output associated with the operator and defined as a table having at least one domain having a type associated therewith. The query is transformed into a set program having at least one operation structure, corresponding to the operator and having logic for type-independently performing an operation, corresponding to the operator, and having a data relation structure, cooperating with the operation structure, for handling all data access and storage associated with the operation.

Description

    BACKGROUND OF THE INVENTION
  • 1. Field of the Invention [0001]
  • This invention relates to database, data warehouse, and data mart technology and, more particularly, to an improved system and method for exploring information relationships in data. [0002]
  • 2. Discussion of Related Art [0003]
  • Modern computing databases have extremely large quantities of data. Businesses often desire to discover information relationships in this data to make better informed business decisions. In this regard, “data warehousing” is used to describe computing technologies used to discover relationships within a database, and “data mart” is used to describe technologies for a subject-specific data warehouse. [0004]
  • To date, data warehousing and data mart tools have been undesirable because of their high cost, both in infrastructure and human capital. Modern systems are effectively customized database applications. Consequently, exploring relationships usually involves the creation of new, custom queries and typically requires a management information systems (MIS) professional, or other programming personnel, to implement the query. If a user, for example, in a marketing department, wishes to investigate a potential new information relationship, he or she is often forced to cross department boundaries and as a result almost invariably experiences undesirable delays. As a result, much of the data is under utilized because many relations are never explored because the delay outweighs the benefit. [0005]
  • Moreover, because modern data warehouse systems are effectively customized database applications, they often inherit inefficiencies from the underlying database. These inefficiencies may be information related (e.g., inherently precluding certain lines of questioning because the application is tightly-coupled to the database's schema) or performance related (e.g., the system may be optimized for a certain type of transactional access that does not perform well to the accesses involved in the data warehousing queries). [0006]
  • More specifically, concerning performance related issues, most systems rely on the relational data model (RDM). The performance of a RDM implementation is typically limited by its “access method.” Commercially-available systems, for example, have their software logic rely on an access method (e.g., “B+ tree”) that requires multiple accesses to storage (e.g., memory or disk) to obtain a given record. Some of the accesses are to reference structures that are used to effectively “point to” the data of interest (e.g., indices or hierarchies of linked lists). Sometimes, these reference structures can get so large that portions of the structure must reside on disk. Thus a given request for a database record may involve multiple disk storage requests. Moreover, the database operation algorithms are tightly bound to the access method. That is, the algorithm itself has been optimized to the access method and is thus dependent on the existence of the access method. Much of the literature on database performance explicitly or implicitly assumes the existence of such access methods. [0007]
  • Aside from the above limitations, most commercial systems are limited to the actual data within the database. The systems cannot query other important data elements such as the schema, the meta data, or the data dictionary without significant custom programming. Consequently, significant knowledge, e.g. useful queries, is not reported or available for use within such systems. [0008]
  • The above difficulties are exacerbated in the context of data residing on disparate databases. [0009]
  • Alternative approaches have been attempted. Childs, for example, discusses set-theoretic approaches in [0010] Feasibility of a Set-Theoretic Data Structure: a General Structure Based on Reconstituted Definition of Relation, Information Processing 68, Edinburgh, 1968; Description of a Set-Theoretic Data Structure, Fall Joint Computer Conference, San Francisco, 1968; and Extended Set Theory: a General Model for Very Large, Distributed, Backend Information Systems. He is believed to have developed a system (STDS and XTDS) in which a user may express queries directly from a small set of set operators.
    • SUMMARY
  • Preferred embodiments of the invention provide a system for, and method of, exploring relationships in data stored in a computer readable medium. A query is received having at least one operator chosen from a set of operators that includes relational operators and having at least one input and output associated with the operator and defined as a table having at least one domain having a type associated therewith. The query is transformed into a set program having at least one operation structure, corresponding to the operator and having logic for type-independently performing an operation, corresponding to the operator, and having a data relation structure, cooperating with the operation structure, for handling all data access and storage associated with the operation.[0011]
    • BRIEF DESCRIPTION OF THE DRAWING
  • In the Drawing, [0012]
  • FIG. 1 shows a preferred client-server embodiment of the invention at a high-level of abstraction; [0013]
  • FIG. 2 shows an Xset data structure according to a preferred embodiment of the invention; [0014]
  • FIG. 3 shows server logic according to a preferred embodiment of the invention; [0015]
  • FIG. 4 shows query evaluation logic according to a preferred embodiment of the invention; [0016]
  • FIG. 5 shows outer language evaluation logic according to a preferred embodiment of the invention; [0017]
  • FIG. 6 shows inner language evaluation logic according to a preferred embodiment of the invention; and [0018]
  • FIG. 7 shows server engine logic according to a preferred embodiment of the invention.[0019]
    • DETAILED DESCRIPTION
  • A preferred embodiment of the invention provides a data exploring system and method that allows users to express queries diagrammatically and with many declarative or calculus aspects. This expression is evaluated to a form used by a set-based server engine. The arrangement among other things allows queries to be expressed as a function of set membership and allows implementations to be proved correct. Among its many advantages, preferred embodiments allow users to query any available databases regardless of structure, origin, or size, singly or in combination with other data sources; to analyze any and all parts of the data including the schema and query information; and to query free-form text. Because customized database applications, queries and schemas are avoided the system may be setup and used quickly and new lines of questioning may be created with more flexibility than existing systems and without the necessity of MIS involvement. [0020]
    • Overview of the Preferred Client-Server Model
  • FIG. 1 shows a preferred embodiment of the invention at a high level of abstraction. The [0021] system 100 follows a client-server arrangement having a client 102, a server 104, and one or more data sources 106 a-n holding the data to be explored, which may be imported or attached as described below.
  • The [0022] client 102 and server 104 communicate over a computer network 108 according to a predefined protocol (e.g., TCP/IP). The communication is based on packets that encode various requests and that provide data to and from the client and server. The server communicates with a data source 106 over a link 110 according to a defined protocol, such as ODBC. The data sources 106 a-n provide the data to be explored, though the data to be explored may also include information stored in “workspaces” in server memory. Moreover, the “data” within the data source may include database schema information, queries, or the like and may be organized as individual files or organized as libraries.
  • Preferably, [0023] client 102 is “thin,” providing a graphical front-end that allows a user (1) to create a diagrammatic representation of a query, (2) to view query results and other workspace information, and (3) to submit queries to be solved. The client 102 has minimal intelligence concerning the query language (“outer language”). Instead, its intelligence is primarily directed to graphics. It has logic to graphically construct and manipulate boxes to represent outer language operators, or operations, and lines connecting boxes to represent data flow dependence. It also has logic to present workspace information such as query results.
  • The [0024] server 104 is preferably “thick,” having the intelligence to manage workspaces and to evaluate and solve outer language queries. (A later section “Operators and Types” describes the outer language operators of a preferred embodiment.) In this fashion, most of the system's intelligence is localized to the server 104. Consequently, the client 102 should not need updates in response to changes in the data exploration system or the underlying query language, and the server should not need updates in response to changes in the data presentation techniques or the system's graphical user interface (GUI).
  • The [0025] client 102 and the server 104 cooperate in the construction of a query in response to user actions. A user, for example, performs graphical actions such as the selecting, dragging, or dropping of an operator icon from a palette on to a query window. (The client, having no built in intelligence about the outer language or system, is informed of the set of operations implemented by the server through a collection of server-provided identifying codes.) In response, the client 102 sends messages to the server 104, providing the operator's identifying code and other information indicative of the user action. The server 104 analyzes the messages and responds by sending messages to the client, providing “pattern” and other information indicating how a corresponding operator block should be drawn (e.g., indicating the number of input and output connectors for the operator block and the number of items to be drawn on each connector). If a user connects an operator block to another operator block or to an iconic representation of a data source, the client sends indicative messages to the server, which updates a corresponding internal representation of the query (called a “parts list”) to reflect the connection. Sometimes the modification of a given block or connection can cause the server to propagate the change through the parts list.
  • In short, the query constructed is a graph of operations in which “tables” (i.e. depicted sets either DSets or DPSets, discussed below) flow between operators. The outer language follows a functional, applicative paradigm that, among other things, supports parameterized queries and recursion. The query (or program) may include one or more constraints between columns within a table, columns of different tables, or a column and a scalar. As will be explained below, these constraints imply row selection if they are within a table, or a join operation (e.g., inner or theta join) if they are between tables. [0026]
  • When the user is satisfied with the query, he or she may “submit” it to be solved by the [0027] server 104. More specifically, the user will cause the client 102 to send an indicative message to the server, instructing it to solve the identified query. The server 104 will then evaluate the query into a semantically equivalent set of operation “primitives.” (This process, as will be explained below, will generate intermediate expressions of the query. Moreover, through the implementation of the primitives' algorithms, this semantic equivalence is provably correct, not merely asserted as is common in the RDM art.) The server will then submit the primitives to an internal server engine to be solved. The primitives (and particularly their semantics) and the algorithms used by the internal engine to execute the primitives have been found to be particularly efficient. This efficiency is observed both in direct performance (e.g., the algorithmic efficiency of an Inner Join implementation) and in the system's amenability to both local and global optimizations, exploiting certain characteristics of the primitives and algorithms. For example, some optimizations exploit the language's and algorithm implementation's characteristic of preserving the “sortedness” of data across an operation. More specifically, this preservation of sortedness may be exploited in detecting and removing redundant Sort operations within a query. During the above process, the server may send status messages to the client, indicating the progress of the solution process. Once it is solved, the user may cause the client to present the results.
  • The [0028] server 104 also maintains “workspaces.” A workspace organizes data such as related tables (sets of data) and queries (called QRSets). A given workspace effectively provides (1) a namespace for the related data and (2) a user view into the system, including a history of prior queries (i.e., a knowledge base). The workspace also provides a collection of “managers” to facilitate the use of the related data and queries.
    • Preferred Data Structures
  • A preferred embodiment of the [0029] server 104 uses the following data structures for organizing data in server memory and/or on attached storage 106, such as disks or tape.
  • An “XSet” is used directly by the server as a unit of access and manipulation and also in the construction of compound structures. An XSet is a type-less array of data storage units (e.g., bytes). Each XSet corresponds to a file that provides persistence for that XSet's data. Each XSet is arranged as shown in FIG. 2. (There is an exception to this arrangement when an XSet is included in a compound structure called a DPSet, described below.) The header includes information indicating the number of elements in the XSet (i.e., the “cardinality”); the width of each element in data units; the size of the header; the identity of the XSet; the version number of the XSet; and other state information. A preferred embodiment maintains the following XSet state information to represent the XSet's access type and status information: the XSet is new, open, closed, read only, writable, the existing file should be replaced on new file operations, the set should be handled on disk and memory, or memory only, the XSet is unsavable, the XSet is marked for deletion and an indication that an Xset's data is stored externally (e.g., by an attached, non-writable file). A preferred embodiment mirrors the header information in the workspace managers. [0030]
  • Because under a preferred embodiment there is no inherent “meaning” (or type) to an XSet, meaning is imputed from separate entities, either a “Depictor” or an “RShip,” depending on the stage of query construction, evaluation or solution. A Depictor is analogous to a schema and includes data indicating how an XSet's data should be portrayed or understood (at least in a specific instance). A Depictor includes column name, column type identifier, column type width, and column type format, as well as possible data delimiters in the XSet (e.g., spaces, commas, special characters, etc.). Delimeters are used on partitioned data, for example, DPSets with attached data from outside the workspace. An RShip is used in the context of an operation's execution and is created dynamically during query solution. As will be discussed below, it is used to provide data abstraction so that the operation primitive algorithms may be implemented with minimal or no data type dependencies. [0031]
  • A “Depicted Set” (or “DSet”) is a compound data structure that combines an XSet and a Depictor. DSets correspond to the entities used by users. That is, these are the structures users typically use as inputs to queries and that they view. A user typically wants to impute some interpretation on the data being explored and might specify, as a first analysis, that the Depictor should effectively mirror the interpretation supplied by a database schema of the data being explored. However, the separate structures allow users to easily modify a Depictor (and consequently a DSet) and thus cause the system to analyze the data differently, e.g., consider a first column as text, and then later as an integer. For implementation simplicity, an exemplary embodiment has a given column in a DSet having the same width in each row of the DSet. Moreover, Depictors are also constructed by the software logic. For example, an output of an operation will be a DSet, the Depictor of which will be formed either from user input or from software logic using default Depictor forming rules (e.g., output type equals left most input as viewed on screen.) A DSet may be represented in either “extensional” form or in “intensional” form. An extensional DSet is implemented as contiguous storage in which DSet members are stored sequentially, preferably without gaps. An intensional DSet is implemented with a stored “intension.” The intension is an expression that when executed on the server yields the DSet in extensional form. (As will be explained below a preferred embodiment stores an expression called a “QRSet” as the intension.) [0032]
  • A “Depicted, Partitioned Set” (or “DPSet”) refers to a DSet in which the data portion resides on disk, not in server memory or in a server controlled area on disk. More specifically, a DPSet is a compound data structure that includes a data XSet, a DSet for holding partitioning information, and a Depictor. The data Xset preferably does not include the header information identified above for normal XSets but it could under alternative embodiments. The DSet for holding partitioning information defines a layout of the data in the XSet file. This partitioning DSet has one row in it for every row in the DPSet and it has the same number of columns as the DPSet, plus one. The first column of the DPSet holds the byte offset of the start of the corresponding row in the XSet file. The rest of the columns specify the width in bytes of the columns of the DPSet on a row-by-row basis. The Depictor is a DSet which has one row for every column in the DPSet. Each row in the Depictor DSet defines the properties of the corresponding column in the DPSet, including column name, column type identifier, column type width, column type format, and storage information about a given domain. DPSets are advantageous in that they allow conversion and use to be postponed so that only the data that is actually needed will be read from the file. [0033]
  • A “parts list” is a compound data structure that is a collection of n-tuples. Each n-tuple corresponds to a query element as it is drawn on the screen (e.g., input, connector, operation block). The parts list's n-tuples contain data identifying the input and output connectors and the operation, but preferably with just enough information to describe the query as drawn on the display. They also contain data showing the input and output domain on connectors (more below). [0034]
  • A “QRSet” is a DSet representing a query. The Depictor specifies the entities used to represent a query, for example, including the part element type, its identification number, and the like. The Depictor for a QRSet has information about the parts list which represents a function (more below). The XSet portion of the QRSet's DSet holds the corresponding values, e.g., the value of the identification number. A QRSet is an “intensional” form of the DSet eventually solved, or derived, from the query. QRSets, being DSets, may be queried. [0035]
    • Server Architecture
  • FIG. 3 shows the server architecture [0036] 300. The server 104 includes a message and event handling system 302, a query editing system 304, a workspace system 306, an input-output (I/O) system 308, and a query evaluation system 310. Each of the systems 302-310 is preferably constructed with software logic and associated data structures and is executable on server 104 as a separate thread.
  • The message and [0037] event handling system 302 is a queue-based system that receives messages and events from the client 102 and the other server systems 304-310. An event identifies the target of the event and the event itself, and it may include information corresponding to the event. Most of the inter-system communication is event based, though some is implemented with direct system calls. (As will be explained below, some systems, notably the evaluations system 310, also use intra-system event queues.)
  • The [0038] query editing system 304 receives the editing messages from the client 102, via event system 302, and constructs a “parts list” as an internal representation of a user's query. The parts list involves a collection of query operations in which each is associated with inputs and outputs that are (1) table constants, (2) table variables, or (3) scalars. A table constant is a table that exists at the time the query is defined, that is, attached or imported tables, or tables in a workspace, or more specifically tables the intensions of which already exist. A table variable, on the other hand, is one that does not yet exist at the time the query is defined. It may include, for example, the output of another operation in the query (e.g., one higher in the query from the perspective of data flow). A table variable may be considered as an intermediate result of a sub-expression of a query expression. In response to client editing requests, the editing system 304 sends back to the client 102 parts list information indicative of the updated, edited query. This information is used by the client to present the query diagram. A given user action may affect multiple defined operations in a given query. The editing logic detects such dependencies and ensures that query edit requests are propagated through the query to reflect the edit.
  • The [0039] workspace system 306 provides a namespace in which data and queries are related. A suite of workspace managers operates on, or manipulates, that data. In short, the managers are generally organized according to the manipulation's target. For example, some manipulations target a given kind of data, such as an Xset, DSet, or DPSet. Other targets are defined by how the data is used: Depictors are used to describe the column properties of a DSet; editable outer language programs are used to define a query; and executable inner language programs are used to solve a query. There are also managers that record inter-set dependencies, a DSet's intension (i.e., QRSet); progress message addresses; set openings; set access control; set interest; set usage; and a workspace's users. The managers, besides being responsible for the above, also monitor access to determine when a set is no longer needed and in response eliminates such sets.
  • The [0040] workspace system 306 also monitors and detects when sets have changed. One set's definition may rely on another set. This other set may change either because an input file has changed or because of an operation on the server. The workspace system 306 maintains data structures reflecting this inter-set dependence and indicates when a set has changed. Either manual (i.e., in response to a user action) or automatic propagation may be used to update affected sets.
  • Sets have associated access or viewing privileges. A preferred embodiment organizes access based on arranging privileges to local (i.e., the creator), Project (i.e., a set of related users), and All (i.e., anyone). [0041]
  • As will be explained below, the tables are maintained, or “cached,” and available to be used in certain optimizations during query evaluation and/or solving. The workspace managers consider the amount of memory available to the workspace and allocate that memory to tables. Preferably sets and tables are kept in “extensional” form; that is, the data itself resides in the workspace. However, dynamic run-time usage can require the workspace managers to effectively “swap out” the extensional form to storage but retain the “intensional” form, i.e., the query sub-expression that resulted in the data. Moreover, when there is no more interest in a set it may be deleted. Usage histories are maintained and considered by the workspace managers in determining which sets to swap out or delete. A preferred embodiment also factors the size of the set in determining a preference as to which set to keep in extensional form. In this fashion an XSet's cost is consider. Larger sets are more costly because more I/O bandwidth is consumed. Many other caching replacement algorithms may be added or substituted. [0042]
  • An intension manager is used to manage QRSets, for example, to match identical or semantically equivalent intensions during global optimization (more below). [0043]
  • The input-output (I/O) [0044] system 308 cooperates with the server's operating system to read and write data from storage. It abstracts the operating system from the rest of the server logic described in this application.
  • The [0045] query evaluation system 310 transforms the user-created query into a semantically equivalent form, constructed of operation “primitives.” These primitives are operations that are executable by a server engine. In short, the primitives generally have more restricted operation assumptions than the user query (e.g., input arguments must be in a prescribed order) and are set-theoretic, rather than relational. The outer language, though having relational model characteristics, also contains set theoretic aspects. Thus queries may be expressed in a form emphasizing set membership. As a consequence, the outer language has both algebraic aspects and more powerful calculus aspects.
    • The Ouery Evaluation System
  • The [0046] query evaluation system 310 transforms the server's internal representation of an outer language query into a semantically equivalent set of operation primitives. In the process, several intermediate expressions are formed. These primitives are then executed by an internal engine, under the control of the evaluation system 310. As show in FIG. 4, the query evaluation system 310 cooperates with the event and message handling system 302, described above, and includes an outer evaluation system 402, an inner evaluation system 404, and a DSet operation execution System 406.
  • The event and [0047] message handling system 302, as outlined above, receives and handles messages and events from the various server systems 304-310. In the instant context, these events include client-originated messages identifying a particular query and requesting that it be solved and client-destined messages indicating the status of a particular query's execution. These events also include messages to and from the workspace manager 306 to record interest in sets and their usage and to determine whether the workspace manager 306 includes a set or superset as expressed by a set's intension. These events and messages also include requests to read and write data to storage 106 via input-output system 310. These latter events and messages may be partially mirrored to the workspace system 306 so that it may update set status accordingly.
  • The [0048] outer evaluation system 402 transforms the query from an outer language expression to an inner language expression. The inner evaluation system 404 transforms the inner language expression into a second inner language expression and then into a set of operation primitives and corresponding control structures (an “oper set”). An oper set is a collection of “opers” in which each oper identifies the operation primitive to be performed (e.g., Inner Join with inner language operation assumptions) and which describes the operation's input and output DSets. This oper set description includes control structures to identify the state of inputs and outputs (e.g., valid or invalid). These control structures are used to data flow schedule any “ready” opers (i.e., ones in which all inputs are valid, and outputs are defined). (Data flow scheduling is known, though not in the combination described.) The operations include a “switch” operator, which creates special “void” sets as outputs to “force” non-selected output dataflow paths to be skipped. This skipping occurs because all operators except the Accept operator degenerates into simple pass-through of void sets without attempting any calculation. The set handling logic that checks for ready opers sees that one of the inputs is void and marks that oper as if it had completed execution and copies void sets to outputs as necessary. The “Accept” logic checks if all of its inputs are ready (including void sets) and does a union of all inputs to generate an output. The oper set is used in invoking and is executed by the DSet operation execution system 406. The DSet operation execution system signals completion status to the inner evaluation system 404 to inform it of DSet operation execution system's status. The DSet operation execution system also invokes the outer evaluation system 402 to solve parameterized queries (or functions). A parameterized query has table variables, not table constants, as inputs. If a function is called (through the Evaluation operation) in a query, its definition must be instantiated, and in this regard, the outer evaluation system 402 is the entity responsible for evaluating such instantiation. The DSet operation execution system 406, however, is responsible for the actual execution of the overarching query and only it can determine when a function is to be called and it signals the outer evaluation system to evaluate an instance of the function definition. This allows for a recursive evaluation of the function definition. Thus, the DSet operation execution system must signal the outer evaluation system to instantiate a function.
  • The [0049] outer evaluation system 402 is shown in FIG. 5 in more detail. The outer evaluation system includes
  • [0050] logic 502 to normalize the outer language expression;
  • [0051] logic 504 to convert the normalized outer expression to a kernel version;
  • [0052] logic 506 to normalize the kernel version; and
  • logic [0053] 508 to convert the normalized kernel version to an “inner program.”
  • The combination of logic [0054] 502-508 uses a multiple pass approach, in which the expression is continually processed in passes until a pass results in no more normalizations or conversions. Items within the parts list are marked and unmarked to indicate whether a certain normalization or conversion has occurred to improve the performance of the processing.
  • The [0055] logic 502 to normalize the outer language expression processes the query to place the expression in a better form for conversion. The purpose of the normalization is to handle user “shorthands,” i.e., expressions that are convenient to specify but which have certain implied meaning that needs to be made explicit.
  • More specifically, the [0056] normalization logic 502 ensures that all operations have properly defined input sets and if not it adds them. Additionally, logic 502 ensures that all “relator” operations (e.g., Less Than) have an associated type specified in the internal representation. This is done because the relator may have inputs of different explicit or implicit types, e.g., one being an integer and another being a floating point. The default relator type is assigned to that of the type of the “left” input, as connected in the user's query. The logic 502 also ensures that the various items in the query (i.e., input and output connectors, operators, etc.) have names and identification numbers (IDs) in sequence (e.g., 1 . . . N with no gaps) to facilitate further processing. It also ensures that the parts lists includes information indicating the sort key ordering associated with a given DSet. This way the sort key ordering information is maintained for subsequent conversions and optimizations.
  • The [0057] logic 504 creates a “kernel version” of the query which is a simplified form of the outer language. In short, the kernel version does not support constraints. Thus, any use or implication of constraints in the user query must be handled by insertion of the appropriate operations into the kernel version. Also, the kernel version uses only a subject of the operators supported by the full outer language and it also uses a few operations that are not available to the user. This is done to create a version of the program (i.e., the kernel version) that can be executed more efficiently, while allowing the user to express queries with more convenient, expressive or intuitive operations. The logic 504 detects whether a query has multiple table constants on an input connector of a relator operation and, in response, defines a set to include the table constants and converts the query to refer to the new set, rather than the table constants. This is performed by inserting an operation into the kernel program called Constants to Set. (see “Operators and Types” section) It ensures that a relator's input arguments are in a prescribed order. If one input is a scalar and the other is a column reference, then the column reference is placed first. If both inputs are column references, the column reference with a “lower address” (i.e., based on set type, set identifier, and column identifier, in that order) is placed first. If the inputs are swapped as a result of the above, the operation relator is converted accordingly, e.g., Less Than converted to Greater Than.
  • The outer language allows user queries to use regular-expression pattern matching in expressing queries. To handle this, [0058] logic 504 also converts certain constraints involving regular-expression-pattern matching and the constraints' column references to an “intensional” regular-expression set and an Inner Join that joins the data to be matched with the set of all possible generated strings as expressed in the intensional form. This creates a state machine equivalent to the regular expression to programmatically describe a string of interest, and the state machine is implemented as a table, i.e., an intensional form for the string.
  • [0059] Logic 504 also detects when the constraints used in a query refer to DPSets and in such cases injects into the kernel program a corresponding Select Column and Select Row operators having the DPSets as input. Any other operators having a DPSet as an input are converted so that a DSet is formed from the DPSet and so that the kernel operator refers to the DSet.
  • [0060] Logic 504 also detects when constraint blocks in the outer language query imply Select Rows, Inner Joins, Inner Restricts, Theta Joins, Theta Restricts, or Concatenate Columns and in such cases it injects the corresponding operations into the kernel version. (The conversion logic here avoids the use of theta joins in cases where an inner join may be used.) For example, if an operation uses columns from different tables in the same input connector, logic 504 converts the operation to have columns from the same table by combining the prior two tables using an inserted Inner Join based on the equality constraints specified in the query for the two tables.
  • [0061] Logic 504 also ensures that certain input ordering is maintained (e.g., that table constants come after variables), transforming the operator if necessary. Logic 504 also transforms nested disjunctions (i.e., ORs).
  • More specifically, regarding the above paragraphs, the outer language allows queries to be created that have one or more constraints between (1) columns within a table, (2) columns of different tables, or (3) a column and a scalar value. These constraints imply row selection if they are within a table, or some kind of join (e.g., Inner or Theta) if they are between tables. To perform the above, [0062] logic 504 compares elements in the parts list with pre-programmed patterns (not to be confused with the patterns sent to client to describe how an operator block should be drawn). The recognition of such implied operating use conventional algorithms.
  • [0063] Logic 504 also transforms function calls (i.e., Evaluate) and expands “macro” operations. The actual instantiation of a function call is not performed until the query is being executed by the internal engine (more below), at which time the function call is identified by a query name. Regarding macro expansion, one exemplary embodiment implements Aggregate-By-Group and Concatenate Strings as macros. In the Aggregate-By-Group example, the macro is transformed into corresponding kernel operators that select rows for each group of constraints on the Aggregate-By-Group operator; Cross Products the name for that group with the selection; Unions the named selections; and Aggregates-By-Key the union adding the group name as the first key column. In the Concatenate String example, the operator is expanded into a graph of Binary-Concatenate-String operators.
  • [0064] Logic 504 also transforms constraint disjunctions on Select elements into a Union of the results of Select elements for each disjunct. For example, a Select having DPSet rows with two or more groups of constraints between column references and scalars is transformed into a Select having DPSet rows for just the first group of constraints, Select DPSet rows for the other groups of constraints and a Union of the outputs of these two Select row operators. As part of this group of conversions, an Aggregate-By-Key is converted into a graph of kernel operators that perform a keyed statistical aggregation operator for each statistical operator identified in the Aggregate-By-Key, and Unions the results.
  • After [0065] Logic 504 is finished converting the normalized program, the logic 506 to normalize the kernel version of the outer language expression analyzes the “data flow” of the query to verify that each operation potentially has enough information to be executed. This is done to determine whether the query is legitimate from a data flow perspective. A new QRSet is created and organized according to the data flow dependence as suggested by the query. The new QRSet also has its identifiers and names normalized to facilitate further processing.
  • After [0066] Logic 506 is finished, the logic 508 to convert the normalized kernel version to an “inner program” performs another set of optimizations and transformations to produce an inner program, or QRSet (this is not yet the set of operation primitives). Each operator in the outer program has at least one corresponding inner program operator. However, the operation assumptions between the outer and inner operations are not necessarily identical. For example, the Inner Join operator of the inner language assumes that its inputs are sorted, whereas the same operator in the outer language makes no such assumption. Consequently, the conversion logic 508 detects operations that do not have semantically equivalent inner forms and makes the necessary changes. In the example of Inner Join, this transformation would include the analysis to determine whether the inputs were necessarily sorted (e.g., because of a prior operation) and if the inputs were not necessarily sorted, the logic would insert the necessary sort operations into the QRSet.
  • More specifically, logic [0067] 508 detects instances of Inner Join, Intersect, Relative Complement, Relative Complement By Key, Symmetric Difference, Union, and Substitute By Key, and analyzes the keys involved to determine if they are sort comparable. One embodiment, for example, uses two broad sort comparison types: numbers and strings. If the keys are not sort comparable then the conversion injects a Select Columns operation to one of the inputs so that the keys are sort comparable. Select Columns, like all other operations, can be used to perform a type conversion through appropriate specification of DSet inputs and outputs. Select Columns is preferred as an independent “injected” operation for this task as its implementation is fast. Some queries may use appropriate modification of other operations, rather than injection of an independent operation, but independent operations have an advantage of easy implementation.
  • Logic [0068] 508 then analyzes the inner program to detect operators that correspond to primitive operators that require their inputs to be sorted and upon such detections inserts the appropriate sort operation according to the corresponding keys. More specifically the logic 508 detects instances of Inner Join, Intersect, Relative Complement, Symmetric Difference, Union, Group-And-Count, Sum over Groups by Key, Max, Min, Average, and Standard Deviation, Row Mode, Median, Count Unique Rows, Substitute By Key, and Relative Complement by Key and injects into the QRSet, before the operators themselves, the appropriate sort, corresponding to the keys in these operators. The sort takes all keys in key order as sort key inputs and then uses all other domains on the operator needing sorting as the non-sorted inputs to the sort by key operator. The outputs of the sort by key operator will go to the inputs of the questioned operator (such as sum over groups) both keyed and unkeyed as appropriate.
  • Logic [0069] 508 detects and removes redundant Select Columns operations, i.e., selection operations that perform a selection already being done by a necessarily preceding section of the program.
  • After the logic [0070] 508 is done the query is now represented as a QRSet. This inner language form, as outlined above, is then evaluated by the inner evaluation system 404 which, in turn, creates another expression of the query, the oper set. The inner evaluation system 404 is directly called by the outer evaluation system 402, as described with reference to FIG. 4.
  • The [0071] inner evaluation system 404 is shown in FIG. 6 in more detail. Inner evaluation system 404 includes
  • [0072] logic 602 to transfer the inner language expression to an optimized form;
  • [0073] logic 604 which converts the optimized inner form to a set of primitive operations that may be executed by the server's internal engine (described below); and
  • [0074] logic 606 that invokes DSet Operation Execution system 406
  • [0075] Logic 602 performs yet more optimizations on the query. These optimizations may alternatively be performed in the outer evaluation system 402. Logic 602 combines sorts. More specifically, it finds pairs of sort operators that sort the same input table such that the column sequences defining the sort orderings are equal up to the length of the shorter of the two column sequences defining the key orderings. Logic 602 then replaces these sorts by deleting the second sort and substituting, for the first sort, a Sort by Key that has the longer of the two column sequences and it includes all of the data columns present in either of the two sort operators. This sort subsumption is valid because the implementation of all operation primitives preserve the “sortedness” of data.
  • [0076] Logic 602 also inserts a Select Column operation before every Sort operator and defines the Select Column operation so that it selects only as many columns from the table as needed for the sort key and output columns. This avoids copying of unneeded data within the sort implementation, described in a later section.
  • [0077] Logic 602, analogously to logic 508, detects and removes redundant Select Columns operations. Logic 602 also normalizes the identifiers and names of the inner language expression.
  • [0078] Logic 602 then performs global optimization of the query by using tables already available in the workspace, i.e., cached results. If the “intension” for a table in the workspace does all or part of the work needed by a portion of the query being evaluated, then that portion of the query is removed and the parts of the query that relied on the output of the removed portion are changed to use the pre-existing table. One embodiment determines that a workspace table may be used by looking for identical intensions between the query being optimized and prior queries. That is, the logic compares the stored intension of a table (i.e., a QRSet) with portions of the query being evaluated. Alternatively, the logic may incorporate algorithms to detect semantic equivalence of intensional forms. These algorithms may consider associativity, commutivity, transitivity, and other properties in detecting equivalence. Also the optimization may look for tables that satisfy portions of the query but which may not be as restrictive as sought. For example, a query being evaluated may have a set membership restriction asking for “all males in Wichita, Kansas.” The most restrictive intension that would satisfy this would be a table having an intension identical to or semantically equivalent to “all males in Wichita, Kansas.” However, other forms may be useful. In particular existing tables with less restrictive intensions may be used beneficially. Of these less restrictive forms, the most restrictive of them would fit best and could be used as a starting point, in which other operations are added to yield the intension sought. That is, the most restrictive of the less restrictive forms could be used as a starting point and be substituted into the query and operations may be injected to complete the intension, i.e., to further restrict the substituted form into the form sought. For example, a table may exist having “all males in Kansas.” This table may be used (provided appropriate other data exists) in the expression and further restricted. Finding all males in Wichita from this table would be less costly than finding all males in Wichita Kansas from the original data sources. The optimization logic finds such tables and uses them and inserts the appropriate subsequent restrictions.
  • Under one embodiment, [0079] logic 602 also selects an appropriate inner language operation depending on the context. As stated above, each outer language operation corresponds to at least one inner language operation. Multiple inner language operations are used because some operation implementations may be better suited to a particular context. A given “sort” for example may be the best choice for large sets but not for small ones. Consequently, logic 602 selects a context sensitive theorem and considers, among other things, the size of the sets, the size of the sets relative to the size of memory buffers, and the size of the operands, relative to one another, in selecting the appropriate inner language operation. Another embodiment considers the frequency of query interaction. A Hash Join, for example, provides good performance on a single use because among other things it does not require presorted input data. However, if the operation is to be repeated, an alternative join (with the injected overhead of sorting data) will be preferable. That is, the multiple uses will offset the cost of injecting an appropriate sort.
  • [0080] Logic 604 then copies the inner language program to form a set of operation primitives to be executed. This process includes adding control structures to each operation in the primitive operation set to facilitate the control thereof according to a data flow scheduling algorithm. This form is the oper set, i.e., a sequence of opers. Each oper is a control structure for each operation in the inner language program. All opers reference the inner language program so that updates to the inputs and outputs can be made so that the DSet operation execution system 406 can perform data flow scheduling.
  • After the oper set is formed, the [0081] logic 606 receives the oper set and invokes the DSet operation execution system 406, as the operators become “ready” to execute. In a preferred embodiment, “ready” means from a data flow scheduling perspective of which the inputs are valid. This logic 606 invokes the DSet system 406 to schedule the primitives for execution according to a data flow scheduling algorithm, in which an operation is “fired” when all of its necessary inputs are detected as being “ready”. Inputs are ready when they exist in extensional form in the workspace. Preferably, a given primitive operation is “fired” to execute as its own execution thread on the server, thus increasing the execution parallelism further. The scheduling of opers within an oper set are implemented with a separate event queue. This event queue has specific logic to handle the control structures associated with an oper and to detect when an oper is ready to be executed.
  • The DSet [0082] operation execution system 406 is shown in more detail in FIG. 7. The DSet operation execution system 406 includes set up logic 702; internal engine 704; workspace interaction logic 706; depictor system 708; RShip construction logic 710; type access system 712; and memory utilities 714.
  • The set up [0083] logic 702 ensures that the DSets involved are ready, i.e., with open files. Under one embodiment this logic will determine whether the input DSets exist in extensional form and if not will cause them to be so. Another embodiment however performs the above just before global optimization is performed by inserting the appropriate operation blocks before the blocks requesting the DSets.
  • The set up [0084] logic 702 cooperates with the outer evaluation system 402, as outlined above. In particular, the set up logic 702 detects the existence of Evaluate opers and sends a solve event to the outer evaluation system 402 so that it may evaluate the instantiations of the corresponding query. In response to such an event from the DSet operation execution system 406 the outer evaluation system 402 will instantiate and submit it to full outer and inner language evaluation, as necessary. If a QRSet exists, that will be the starting point of function instantiation.
  • The set up [0085] logic 702 is also responsible for re-expressing a given oper from an operation with input and output DSets to a primitive operation that operates in conjunction with a corresponding RShip and in which the data is in XSet form. RShips are used to control all data access and data storing in result sets. Under a preferred embodiment they are also used to convert input data (as needed) from one type to output data of another type; that is, conversion during transfer of elements. Under a preferred embodiment they are also used to convert data in relator operations, both in a comparative sense and in an arithmetic sense; for example, when relating a floating point an integer. (The types are identified by the input and output DSets.) The RShip will abstract the algorithm (for the most part) from the data types of the data involved. More specifically, the set up logic 702 invokes the depictor logic 708 to initiate the construction of an RShip for a corresponding oper. The depictor logic 708 analyzes the Depictors for input and output DSets corresponding to the oper and also analyzes the oper to determine whether it is for a relator and constructs the corresponding RShip structure to map the data conversions involved with data transfer and/or comparison aspects involved with the operation. An operation may require more than one RShip for its execution. For instance, the Add operator has an RShip that describes the domain relationship between two operands and another RShip for transferring the results. Also, an inner join, for example, involves both data transfer and comparison operations. Persons skilled in the art will appreciate from reviewing the “Operators and Types” section that the number of RShips is operation dependent as each has its own inherent number of transfer and relator sub-operations. The RShip, in addition to containing mapping information, will contain corresponding control structures involved with the conversion. For example, the RShip may contain high-level control logic for iterating over the domains involved. The RShip structure thus has the mapping information and control logic. It relies on type access logic 712 to perform the actual conversions of data.
  • For example, a select may have an input of a first type and an output of a second type. The RShip structure in such case will construct n-tuples indicating the corresponding mapping. Moreover, the operation involved may be a relator operation such as Equal To and may involve inputs of disparate types. The RShip in such case, will map the appropriate conversions needed for the comparison operation, as outlined above. Under one embodiment, loop unfolding techniques may be used to pre-load RShips and delimeters. [0086]
  • The set up [0087] logic 702 also allocates buffers as necessary for intermediate results.
  • The [0088] engine 704 includes oper algorithm logic for each of the inner language operations. The implementations follow standard expressions of the algorithms (e.g., by Knuth) but are preferably type-less, avoid conditional execution, and are maximally abstracted from the data. Preferably the algorithms use a single pass over the data. Some algorithms, as is known, require multiple passes, e.g., sort. The algorithms are implemented to tell a corresponding RShip to compare and transfer data as needed by the operation. In some instances the data will be transferred to a buffer so that it may be further manipulated by the algorithm logic. In other instances, the algorithm logic will not need to do any further manipulations (e.g., Inner Join and Select). In short, the algorithms are the higher level controls to transfer and/or compare and at times to perform some further manipulation. The actual transfer, comparison, and conversion are abstracted and performed by the RShip and thus the algorithms are made type-less where the operation permits. (For example, some string manipulation operations will need to actually manipulate the data, not just instruct an RShip to transfer and compare, and inherently convert, it.)
  • The [0089] set handling engine 704 includes logic allowing multi-threaded access to files. This logic is included because the operations are handled by multiple threads. Some operating systems do not permit transparent multiple access to a file from different places in multiple threads. To address this, logic references Xset files and includes access counters to interface file access. The multiple threads access the engine's interface logic rather than the underlying file system to see if files are open and to cause the corresponding file operations. (Mutex controls are used to maintain coherency of the interface logic.)
  • The engine memory utilities are used for accessing and allocating memory. For example, they may be used to allocate contiguous physical chunks of memory. The buffer allocation is mapped to table rows and the services may be used to access the desired row of a table, while hiding the mapping details. Other memory utilities (lower level) are preferably used to abstract type conversions, such as endian conversions for floating point data. [0090]
  • The [0091] workspace logic 708, cooperates with the workspace system, and is used both in allocation and committal of DSets. When an operation is being executed, the logic needs to allocate DSets for outputs. The workspace logic 708 will make such requests to the workspace system. When an operation has finished execution, thus defining the data for the corresponding output DSet, it is “committed” to the workspace indicating it is valid. At this point it may be used for viewing and/or global optimizations, and as inputs to operations, among others.
    • Operators and Types
  • This section described the operators and types of a preferred embodiment. Persons skilled in the art will appreciate that the set of operators and types may be modified significantly without departing from the scope of the invention. [0092]
  • Data Types [0093]
  • The data types for both the outer and also the inner programming languages include Int8; Int16; Int32; Int64; Uint8; Uint16; Uint32; Uint64; FloatS (32-bit floating point number); FloatB (64-bit floating point number); Float precision (64-bit floating point with specified precision) StrFA (common string type); StrXA (DPSet string type for “delimited” strings); Byte ( basic string type used to implement both StrFA and also StrXA); Time; Date; and DateTime. [0094]
  • Outer Operators and Constraints [0095]
  • The operators defined for the outer language are described below. The “non-kernel” operators are “macro” operators that are converted to program fragments built out of kernel operators. The non-kernel constraints are replaced by a combination of kernel operators and modifications of existing portions of the query program. Some of the non-client-visible kernel operators are used by the server to implement some of the non-kernel operators and constraints (e.g. Binary Concatenate Strings is used to implement Concatenate Strings, Evaluate is used to implement Query, Generate Strings from Regular Expression is used with Inner Equijoin to implement the Regular Expression Match constraints). [0096]
  • Constraints [0097]
  • 1. Equal To (constraint, non-kernel) [0098]
  • 1. equal_to(In1, In2) [0099]
  • Constrain any use of tables elsewhere in the program, containing the equal_to constraint operator, where those tables are referred to by In1 or In2 items, to be references to the corresponding columns of the table that is the inner equi-join of the In1 and In2 tables with join keys of In1 and In2 if In1 and In2 are from different tables, or the select-rows of the table In1 and In2 such that the corresponding columns of In1 and In2 are equal if In1 and In2 are from the same table. [0100]
  • 2. Greater Than Or Equal To (constraint, non-kernel) [0101]
  • 1. greater_than_or_equal_to(In1, In2) [0102]
  • Same definition as the greater_than constraint operator, except using the greater_than_or_equal_to relator instead of the greater_than relator. [0103]
  • 3. Greater Than (constraint, non-kernel) [0104]
  • 1. greater_than(In1, In2) [0105]
  • Constrain any use of tables elsewhere in the program, containing the greater_than constraint operator, where those tables are referred to by In1 or In2 items, to be references to the corresponding columns of the table that is the theta join of the In1 and In2 tables with theta join constraint of In1 greater_than In2 if In1 and In2 are from different tables, or the select-rows of the table of In1 and In2 such that In1 greater_than In2 if In1 and In2 are from the same table. [0106]
  • 4. Less Than (constraint, non-kernel) [0107]
  • 1. less_than(In1, In2) [0108]
  • Same definition as the greater_than constraint operator, except using the less_than relator instead of the greater_than relator. [0109]
  • 5. Less Than Or Equal To (constraint, non-kernel) [0110]
  • 1. less_than_or_equal_to(In1, In2) [0111]
  • Same definition as the greater_than constraint operator, except using the less_than_or_equal_to relator instead of the greater_than relator. [0112]
  • 6. Not Equal To (constraint, non-kernel) [0113]
  • 1. not_equal_to(In1, In2) [0114]
  • Same definition as the greater_than constraint operator, except using the not_equal_to relator instead of the greater_than relator. [0115]
  • 7. Regular Expression Match, Right (constraint, non-kernel) [0116]
  • 1. regexp_right(Data, Pattern). [0117]
  • Same definition as equal_to, but instead of “the inner equi-join of the In1 and In2 tables with join keys of In1 and In2 ” this is the inner equi-join restriction of the data table and the single column of the output GenStrings table of “generate_strings_regexp(Pattern)→GenStrings”: inner_equijoin(Data, Data, GenStrings, [ ])→RestrictedData. [0118]
  • 8. Regular Expression Match, Left (constraint, non-kernel) [0119]
  • 1. regexp_left(Pattern, Data). [0120]
  • Same definition as Right Regular Expression Match, with the order of the operands reversed. [0121]
  • Operators [0122]
  • 1. Aggregate By Group (non-kernel) [0123]
  • 1. aggbygroup(Data, StatOps, Key, GroupNames, Left, Relator, Right)→Output [0124]
  • Data and StatOps have the same number of items. The StatOp item corresponding to a Data item indicates the statistic (e.g. min, max, average) that is to be calculated for the column indicated by that Data item. Left/Relator/Right is a triple of connectors that specifies a collection of constraints. These constraints are used to select rows from Data. The collection may define a disjunction of multiple conjunctions of constraints (e.g. ((A[0125] 1<A2 and B1=B2) or (C1>=C2 and D1!=D2) or (E1=E2 and F1<=F2))). GroupNames associates a “name” with each disjunct of Left/Relator/Right (e.g. G1 with (A1<A2 and B1=B2), G2 with (C1>=C2 and D1!=D2), and G3 with (E1=E2 and F1<=F2)). Output has one column for each item in Data, plus a column for “group names”.
  • If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to all of the cells in the column corresponding Data item of the selected data for that group. If Key has one or more items, then the selected data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups. [0126]
  • 2. Aggregate By Key (non-kernel) [0127]
  • 1. aggbykey(Data, StatOps, Key)→Output [0128]
  • Data and StatOps have the same number of items. The StatOp item corresponding to a Data item indicates the statistic (e.g. min, max, average) that is to be calculated for the column indicated by that Data item. Output has one column for each item in Data. If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to the data of all of the cells in the column of the corresponding Data item. If Key has one or more items, then the data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups. [0129]
  • 3. Concatenate Strings (non-kernel) [0130]
  • 1. concatstr(Input)→Output [0131]
  • Input contains two or more items. All of the column references among these items must be to the same table. There may be any number of scalar items interspersed among column reference items. Output contains one column Output contains the same number of rows as the input table implied by the column references of Input. Each cell of Output is the concatenation of the string representations of the values of the specified columns of the corresponding input table row, in the order given in Input and with the scalars if Input interspersed as specified in Input. [0132]
  • 4. Query (program structure, non-kernel) [0133]
  • 1. query<FunctionName>(In1, . . . , InJ)→[Out1, . . . , OutK][0134]
  • FunctionName is a scalar string value. This operator has the same definition as: evaluate(FunctionNameItem, In1, . . . , InJ)→[Out1, . . . , OutK]. FunctionNameItem is a scalar item with the value of FunctionName. See the evaluate operator definition. [0135]
  • Client-visible Operators [0136]
  • 1. Absolute Value [0137]
  • 1. abs(Input)→Output. [0138]
  • Each cell of Output contains the absolute value of a corresponding cell in Input. [0139]
  • 2. Accept [0140]
  • 1. accept(In1, In2)→Output. [0141]
  • If In1 is “void” then Output is derived from In2. If In2 is “void” then Output is derived from In1. If In1 and In2 are both non-void, then Output is union(In, In2). [0142]
  • 3. Add [0143]
  • 1. add(In1, In2)→Output [0144]
  • If In1 is a scalar S, then each cell of Output is the S plus the corresponding cell of In2. If In2 is a scalar S, then each cell of Output is the S plus the corresponding cell of In1. If In1 and In2 are both tables (non-scalars), then each cell of Output is the corresponding cell of In1 plus the corresponding cell of In2. [0145]
  • 4. Average [0146]
  • 1. avg(Input, Key)→Output [0147]
  • Output has the same number of columns as Input. If Key is empty, then Output has one row where each cell of the row is the average value of the cells for the corresponding column of Input. If Key is not empty, then the rows of Input are grouped according to Key and Output has one row for each key group where each cell of a row is the average value of the corresponding group of cells for the corresponding column of Input. [0148]
  • 5. Compare [0149]
  • 1. compare(Left, Relator, Right)→Output [0150]
  • Left/Relator/Right are a connector triple that defines a conjunction of constraints. Either one of Left and Right contains a scalar value and the other one a column reference, or else both Left and Right contain column references and the columns are in the same table. Left and Right column references imply an input table. [0151]
  • Output contains one column. Output contains the same number of rows as the implied input table of Left and Right. Each cell in Output contains 1 if the conjunction of Left/Relator/Right is true for the corresponding row of the input table, otherwise it is 0. [0152]
  • 6. Concatenate Columns [0153]
  • 1. concatcol(Input)→Output [0154]
  • Input contains two or more column reference items, not necessarily from the same table. Output has the same number of columns as the items in Input. Output has the same number of rows as the smallest of the input tables associated with the items of Input. Each cell in a row of Output is copied from the cell of the corresponding Input column reference. [0155]
  • 7. Constants To Set [0156]
  • 1. constset(Input)→Output [0157]
  • Input contains one or more scalar items. Output contains one column. Output contains one row for each item in Input. Each cell of Output is the value of the corresponding item from Input. [0158]
  • 8. Convert To Lower [0159]
  • 1. cvrtlower(Input)→Output [0160]
  • Input contains one or more column references, all from the same table. Output contains one column for each item of Input. Output contains one row for each row of the input table. Each cell of Output contains the lower case representation of the string representation of the corresponding input column and row cell's value. [0161]
  • 9. Convert To Proper [0162]
  • 1. cvrtproper(Input)→Output [0163]
  • Same as cvrtlower, but “proper case” instead of lower case. [0164]
  • 10. Convert To Upper [0165]
  • 1. cvrtupper(Input)→Output [0166]
  • Same as cvrtlower, but “upper case” instead of lower case. [0167]
  • 11. Correlation [0168]
  • 1. correlation(In1, In2)→Output [0169]
  • In1 has one column reference. In2 has one column reference. If In1 and In2 are different tables then they are “coerced” to a single table (by simulating a concatenate columns of the two tables). Output contains three columns: Pearson's correlation coefficient, the confidence level, and Fisher's Z. Output contains one row with the three columns calculated from the two input columns. [0170]
  • The design of this may be changed to be a single input connector: correlation(Input)→Output Input has two column references . . . the rest is the same as above. [0171]
  • 12. Count Rows [0172]
  • 1. count(Input)→Output [0173]
  • Input contains a column reference. Output contains one column. Output contains one row. The cell of Output is the count of the number of rows in column referred to by the item of Input. [0174]
  • 13. Count Unique Rows [0175]
  • 1. countunique(Input)→Output [0176]
  • Input contains a column reference. Output contains one column. Output contains one row. The cell of Output is the count of the number of unique values in the column referred to by the item of Input. [0177]
  • 14. Cross Product [0178]
  • 1. crossprod(In1, In2)→Output [0179]
  • In1 and In2 contain one or more column references. Output contains one column for each item in In1 and In2. Let N[0180] 1 be the cardinality of the table of In1, and N2 be the cardinality of the table of In2. Output contains one N1*N2 rows. The rows of Output are all possible combinations of rows of the two input tables, with only the columns identified in In1 and In2 and ordered and typed as specified in the Output definition.
  • 15. Divide [0181]
  • 1. divide(In1, In2)→Output [0182]
  • Same semantics as Add, except the “divide” scalar operation is used instead of “plus”. [0183]
  • 16. Extract Substring [0184]
  • 1. extractstr(Data, Start, Length)→Output [0185]
  • Data, Start, and Length must all have the same number of items, all of which are column references. Start and Length must have 1 row each. Output has the same number of columns as Data. Output has the same number of rows as Data. Each cell in Output is the substring of the corresponding cell of Data, starting at the position given by the value of the corresponding cell of Start, and having the length given by the value of the corresponding cell of Length. The single row of Start and Length is re-used for each row of Data. [0186]
  • 17. Group And Count [0187]
  • 1. groupcount(Data, Key)→Output [0188]
  • Data has one or more column references. Key has zero or more column references from the same table as Data's column references. Output has the same number of columns as Data plus one “count” column. [0189]
  • Output has the same number of rows as there are groups in Data as defined by Key. Each row corresponds to one such group. Each row has the corresponding values from a row of the group for the columns of Data, plus the count of rows in Data for that group. An empty Key defines the whole table to be a single group. [0190]
  • 18. Inner Join [0191]
  • 1. inner_equijoin(InA, KeyA, KeyB, InB)→Output [0192]
  • InA has zero or more column references. KeyA has one or more column references from the same table as InA's column references. InB has zero or more column references. KeyB has one or more column references from the same table as InB's column references. There must be at least one column reference in either InA or InB. Output has the same number of columns as InA plus InB. Output has the one row for each combination of rows from InA and InB such that KeyA equals KeyB. The cells in a row of Output have the same values as the corresponding cells in InA and InB. [0193]
  • 19. Input (program structure) [0194]
  • 1. input(Ordinal, Name)→Output [0195]
  • Ordinal is a scalar integer greater than 0. Let S be the multiset of all of the ordinals of the input operators in a program: min(S)=1 and max(S)=cardinality(S). Name is a scalar string “naming” the input parameter defined by this operator. Output has one parameter item or one or more column references to table constants. The parameter item may be origin of other parameter items in the program, and these other parameter items may be used in place of a column reference or scalar item. If a parameter item is used, then the containing program is a program definition that must be instantiated before it can be evaluated. The instantiation process replaces all input-derived parameter items with either column reference or scalar items, depending on the argument items provided by the process invoking the instantiation. [0196]
  • 20. Intersection [0197]
  • 1. intersection(InA, InB)→Output [0198]
  • InA has one or more column references. InB has the same number of column references as InA. Output has the same number of columns as InA. Output has the one row for each combination of rows from InA and InB such that InA equals InB. The cells in a row of Output have the same values as the corresponding cells in InA. [0199]
  • 21. Keyed Relative Complement [0200]
  • 1. keyed_relcomp(InA, KeyA, KeyB)→Output [0201]
  • InA has one or more column references. KeyA has one or more column references to InA's table. KeyB has the same number of column references as KeyA. Output has the same number of columns as InA. Output has the one row for each of row from InA such that there does not exist a row of the table for KeyB where KeyA equals KeyB. The cells in a row of Output have the same values as the corresponding cells in InA. [0202]
  • 22. Length [0203]
  • 1. lengthstr(Input)→Output [0204]
  • Input has one or more column references. Output has the same number of columns as Input. Output has the same number of rows as Input. Each cell of Output is the length of the string representation of the corresponding Input cell's value. [0205]
  • 23. Maximum [0206]
  • 1. max(In1, In2)→Output [0207]
  • Same definition as the Add operator except the cell of Output is calculated using max instead of plus. [0208]
  • 24. Maximum Row [0209]
  • 1. maxrow(Input, Key)→Output [0210]
  • Output has the same number of columns as Input. Output has one row per group defined by Key that is the maximum row in that group of rows in Input table according to the sort order specified by the column references in Input. If Key is empty, then the entire table is considered a single group. [0211]
  • 25. Median Row [0212]
  • 1. medrow(Input, Key)→Output [0213]
  • Same definition as Maximum Row, except the row aggregate operator is “median” instead of “maximum”. [0214]
  • 26. Minimum [0215]
  • 1. min(In1, In2)→Output [0216]
  • Same definition as the Add operator except the cell of Output is calculated using min instead of plus. [0217]
  • 27. Minimum Row [0218]
  • 1. minrow(Input, Key)→Output [0219]
  • Same definition as Maximum Row, except the row aggregate operator is “minimum” instead of “maximum”. [0220]
  • 28. Mode Row [0221]
  • 1. moderow(Input, Key)→Output [0222]
  • Same definition as Maximum Row, except the row aggregate operator is “mode” instead of “maximum”. [0223]
  • 29. Modulus [0224]
  • 1. mod(In1, In2)→Output. [0225]
  • Same semantics as Add, except the “modulus” scalar operation is used instead of “plus”. [0226]
  • 30. Multiply [0227]
  • 1. multiply(In1, In2)→Output [0228]
  • Same semantics as Add, except the “multiply” scalar operation is used instead of “plus”. [0229]
  • 31. Negate [0230]
  • 1. negate(Input)→Output. [0231]
  • Same semantics as Absolute Value, except the “negate” scalar operation is used instead of “absolute value”. [0232]
  • 32. Output (program structure) [0233]
  • 1. output(Ordinal, Input)→Output [0234]
  • Ordinal is a scalar integer greater than 0. Let S be the multiset of all of the ordinals of the input operators in a program: min(S)=1 and max(S)=cardinality(S). [0235]
  • Input is one or more items. In a function instantiation, these items must all be column references. In a function definition, these items may be a mixture of column references and parameter items. Output has the same number of items as Input. The Output table has the same number of rows as the Input table. Each cell in a row of the Output table has the same value as the corresponding cell in the Input table. Thus, the only differences between the Input and Output tables are the column order, column names, column data types, and table names. [0236]
  • 33. Position [0237]
  • 1. position(Source, Search)→Output [0238]
  • Source contains one or more items. Search contains the same number of items as Source. If all of the items of Source and Search are column references, and they are from different tables, then an implicit concatenate columns is performed. Output has the same number of columns as Source. Output has the same number of rows as the Source/Search table. Each cell's value is the index of the string representation of the corresponding Search cell's value in the string representation of the corresponding Source cell's value. If the Search cell's value is not a substring of the Source cell's value, then the index is 0. If all of the items of Source are scalar, then Search must contain column references. Output has the same number of rows and columns as Search. The cells of Output are calculated as above, but the single scalar row of Source is reused for each Search row. If all of the items of Search are scalar, then Source must contain column references. Output has the same number of rows and columns as Source. The cells of Output are calculated as above, but the single scalar row of Search is reused for each Source row. [0239]
  • 34. Rates [0240]
  • 1. rates(In1, In2)→Output [0241]
  • In1 has one or more column references. In2 has the same number of column references as In1. If In1 and In2 are from different tables then the “effective input” table is the concatenate columns of In1 and In2. Output has the same number of columns as In1. Output has the same number of rows as the “effective input” table, minus 1. Each cell's value Yk of a row k is calculated from the corresponding cell's values of rows k and k+1 (A[0242] 1 and A2 from In1, and B1 and B2 from In2) of the input table: Yk=(A2−A1)/(B2−B1).
  • 35. Relative Complement [0243]
  • 1. relcomp(InA, InB)→Output [0244]
  • InA has one or more column references. InB has the same number of column references as InA. Output has the same number of columns as InA. Output has the one row for each of row from InA such that there does not exist a row of the table for InB where InA equals InB. The cells in a row of Output have the same values as the corresponding cells in InA. See also Keyed Relative Complement. [0245]
  • 36. Sample Data [0246]
  • 1. sample(Data, Size, Start, Step)→Output [0247]
  • Data has one or more column references. Size, Start, and Step each have one scalar item. Output has the same number of columns as Data. Output has rows selected from Data based on the values in Size, Start, and Step. Size can be specified as derived from the Start and Step values, as a percentage of the number of rows in Data, or as “as many as possible up to” a specified maximum value. Start can be specified as either to be picked randomly or using a given row index (a negative index counts backward from the end of Data). The Step value can be specified as either: calculated base on the Size and Start values, to be picked randomly, or else using a specified step value (a negative step value steps from the end of Data back toward the beginning). [0248]
  • 37. Select Columns [0249]
  • 1. select_columns(Input)→Output [0250]
  • Input has one or more column references. Output has the same number of columns as Input has items. Output's result table has the same number of rows as the table identified by Input's column references. Each row in Output is based on the corresponding row of the Input table. Each cell's value in Output's table is the (converted if necessary) value of the corresponding cell in Input's table. [0251]
  • 38. Select Rows [0252]
  • 1. select_rows(Data, Left, Relator, Right)→Output [0253]
  • Data is one or more column references. Left, Relator, and Right are a triple of connectors that define selection constraints. Left and Right have column references or scalar items. The column references must be to the same table as the column references of Data. Relator has only scalar items that specify comparisons (<, >, =, <=, >=, !=). Left, Relator, and Right must have the same number of items in the kernel outer language. They may have differing numbers of items in the full outer language if “open” and “close” markers are used to define “nested disjunctions”. For instance, (A=B and C<D) is represented in Left/Relator/Right form as: Left=[A,C], Relator=[=,<], and Right=[B,D]. A “nested disjunction” of (A=(B or E) and C<D) (which is interpreted as ((A=B or A=E) and C<D)) is represented in Left/Relator/Right form as: Left=[A,C], Relator=[=,<], Right=[open,B,E,close,D]. Output has the same number of columns as Data. Output has one row for every row in Data that satisfies the constraints of Left/Relator/Right. [0254]
  • 39. Sort All Ascending [0255]
  • 1. sortasc(Input)→Output [0256]
  • Input has one or more column references. Output has the same number of columns as Input has items. Output's table has the same number of rows as Input's table. Each row of Output's table is derived from the corresponding row of the sorted version of Input's table, in the same fashion as for the Select Columns operator. The sorted version of Input's table is created by sorting the table in ascending order on each of the columns of Input, with the columns have “sort precedence” in the order they appear in Input. [0257]
  • 40. Sort All Descending [0258]
  • 1. sortdsc(Input)→Output [0259]
  • Same definition as Sort All Ascending, except the sorted version of Input's table is created using a descending sort on each column instead of an ascending sort. [0260]
  • 41. Sort By Key Ascending [0261]
  • 1. sortasckey(Input, Key)→Output [0262]
  • Key has zero or more column references from the same table as Input. Same definition as Sort All Ascending, except if Key has one or more items in which case the sorted version of Input's table is created using the columns referenced by Key to define the sort order instead of the columns referenced by Input. [0263]
  • 42. Sort By Key Descending [0264]
  • 1. sortdsckey(Input, Key)→Output [0265]
  • Same definition as Sort By Key Ascending, except the sorted version of Input's table is created using a descending sort on each column instead of an ascending sort. [0266]
  • 43. Soundex Match, Right (proposed, constraint, non-kernel) [0267]
  • 1. soundex_right(Data, Pattern). [0268]
  • Same definition as Right Regular Expression Match, but using the generate_strings_soundex operator instead of the generate_strings_regexp operator. [0269]
  • 44. Soundex Match, Left (proposed, constraint, non-kernel) [0270]
  • 1. soundex_right(Data, Pattern). [0271]
  • Same definition as Right Soundex Match, but with the operands reversed. [0272]
  • 45. Standard Deviation [0273]
  • 1. stddev(Input, Key)→Output [0274]
  • Same semantics as Average, except the “standard deviation” scalar operation is used instead of “average”. [0275]
  • 46. Substitute [0276]
  • 1. substitute(InA, KeyA, KeyB, InB)→Output [0277]
  • InA has one or more column references. KeyA has one or more column references. InB has the same number of column references as InA. KeyB has the same number of column references as KeyA. Output's table has the same number of columns as InA has items. Output's table has the same number of rows as InA's table. Each row of Output's table has values from the corresponding row of the processed version of InA's table, in the same manner as is done for Select Columns. The processed version of InA's table has one row for each row of InA's table. For a given row of InA, if there is a row of InB's table such that KeyB equals KeyA, then the processed row of InA takes the values of the (first) matching InB row. If there is no matching row of InB, then the processed row of InA takes the values of the original InA. [0278]
  • 47. Subtract [0279]
  • 1. subtract(In1, In2)→Output [0280]
  • Same semantics as Add, except the “subtract” scalar operation is used instead of “plus”. [0281]
  • 48. Subtract Substring [0282]
  • 1. subtractstr(Source, Search)→Output [0283]
  • Same definition as Position, except each Output cell value is the string representation of Source minus the first occurrence of the string representation of Search. [0284]
  • 49. Sum [0285]
  • 1. sum(Input, Key)→Output [0286]
  • Same semantics as Average, except the “sum” scalar operation is used instead of “average”. [0287]
  • 50. Switch (program structure) [0288]
  • 1. switch(Control, Data)→[Out1, Out2, . . . , OutK][0289]
  • Control has one column reference. Data has one or more column references. Only Out1 and Out2 are defined in the non-kernel/client-visible use of this operator. All Outi have the same number of column definitions as Data has column references. If there is a cell value convertible to integer N in the Control column, then OutN's table is defined the same as a select_columns(Data)→OutN. If there is a cell value convertible to integer 0 in the Control column, then this is interpreted the same as an integer of K (i.e. the last Out connector). For all J such that there is no cell value convertible to it (according to the above, including the special handling of 0), Outj's table is the special value “void”. The void table has no columns or rows. [0290]
  • 51. Symmetric Difference [0291]
  • 1. symdiff(In1, In2)→Output [0292]
  • In1 and In2 must have only column references and must have the same number of them. Output has the same number of column definitions as In1 has column references. Output's table is “select_columns(SymDiff)→Output” where SymDiff is the symmetric difference table of In1's table and In2's table. The symmetric difference table of In1's table and In2's table contains one row for each row of In1 that is not in In2 where the symmetric difference table's row is a copy of the In1 table's row, and similarly one row for each row of In2 that is not in In1. See also Relative Complement. [0293]
  • 52. Theta Join [0294]
  • 1. inner thetajoin(InA, InB, Left, Relator, Right)→Output [0295]
  • InA has zero or more column references. InB has zero or more column references. Left, Relator, and Right have the same structure and interpretation as in Select Rows, with the additional restriction that Left has only column references from InA's table and Right has only column references from InB's table. There must be at least one column reference in either InA or InB. Output has the same number of columns as InA plus InB. Output has the one row for each combination of rows from InA and InB such that the constraints of the Left/Relator/Right connectors are satisfied. The cells in a row of Output have the same values as the corresponding cells in InA and InB, converted as necessary if the corresponding columns have different data types. [0296]
  • 53. Union [0297]
  • 1. union(InA, InB)→Output [0298]
  • InA has one or more column references. InB has the same number of column references as InA. Output has the same number of columns as InA. Output's table has the one row for each row from InA's table and each row from InB's table. The rows of Output's table have the same relative ordering as the rows of InA's and InB's tables, sorted by the “keys” defined by InA and InB items, respectively. [0299]
  • Non-client-visible Operators (Server-generated) [0300]
  • 1. Binary Concatenate Strings (non-client) [0301]
  • 1. concatstr(In1, In2)→Output [0302]
  • In1 and In2 have a single item each. If they are both column references, they must be from the same table. Output has one column. Output has one row for each row in the input table. Each cell is the concatenation of the string representations of the two corresponding input values. [0303]
  • 2. Evaluate (program structure, non-client) [0304]
  • 1. evaluate(FunctionName, In1, . . . , InJ)→[Out1, . . . , OutK][0305]
  • FunctionName is a scalar item. The interpretation of Ini and Outi are dependent on the definition of the function named FunctionName in the same workspace as the query containing this operator. A function definition is an outer language program with any number of input and output parameter operators. Each input parameter operator defines an input connector: if the input parameter operator's ordinal input is X, then it maps to the X'th connector of a query <FunctionName> operator. Each output parameter operator defines an output connector: if the output parameter operator's ordinal input is Y, then it maps to the Y'th output connector. The relationship between Ini and Outi connectors, whether items on one require items on the other or whether there are “standard” column references that always appear on one or more Outi connectors, all depends on the definition of FunctionName. [0306]
  • 3. Generate Strings from Regular Expression (non-client) [0307]
  • 1. generate_strings_regexp(Pattern)→Output [0308]
  • Pattern is one or more scalar items. Output's table is an intensional table of one column. Output's table has many rows as there are strings that can be “generated” from the regular expression given on the Pattern connector: a string is generated from a regular expression R if that R matches that string under the “regular expression matching” algorithm. Since there may be infinitely many strings that could match a given regular expression, Output's table cannot (in general) have an explicit “extensional” representation. For an intensional table only a few operations may be defined. One of these is always “is_member(X)”, where the intensional table “system” can always answer yes or no to such a question for a particular intensional table. This mechanism is currently used (through the RShip system) to implement inner-join restriction of a “normal” extensional table against an intensional table to produce a new extensional table. [0309]
  • Other Operators [0310]
  • 1. Aggregate By Group (non-kernel) [0311]
  • 1. aggbygroup(Data, StatOps, Key, GroupNames, Left, Relator, Right)→Output [0312]
  • Data and StatOps have the same number of items. The StatOp item corresponding to a Data item indicates the statistic (e.g. min, max, average) that is to be calculated for the column indicated by that Data item. Left/Relator/Right is a triple of connectors that specifies a collection of constraints. These constraints are used to select rows from Data. The collection may define a disjunction of multiple conjunctions of constraints (e.g. ((A[0313] 1<A2 and B1=B2) or (C1>=C2 and D1!=D2) or (E1=E2 and F1<=F2))). GroupNames associates a “name” with each disjunct of Left/Relator/Right (e.g. G1 with (A1<A2 and B1=B2), G2 with (C1>=C2 and D1!=D2), and G3 with (E1=E2 and F1<=F2)). Output has one column for each item in Data, plus a column for “group names”. If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to all of the cells in the column corresponding Data item of the selected data for that group. If Key has one or more items, then the selected data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups.
  • 2. Aggregate By Key (non-kernel) [0314]
  • 1. aggbykey(Data, StatOps, Key)→Output [0315]
  • Data and StatOps have the same number of items. The StatOp item corresponding to a Data item indicates the statistic (e.g. min, max, average) that is to be calculated for the column indicated by that Data item. Output has one column for each item in Data. If Key has no items, then Output has one row for each group name such that a non-group-name cell in a row is the result of the corresponding statistic from StatOps applied to the data of all of the cells in the column of the corresponding Data item. If Key has one or more items, then the data of the corresponding Data item is grouped according to the Key items and the corresponding statistic is calculated for each such group; there are as many rows in Output per group as there are key-groups. [0316]
  • 3. Concatenate Strings (non-kernel) [0317]
  • 1. concatstr(Input)→Output [0318]
  • Input contains two or more items. All of the column references among these items must be to the same table. There may be any number of scalar items interspersed among column reference items. Output contains one column Output contains the same number of rows as the input table implied by the column references of Input. Each cell of Output is the concatenation of the string representations of the values of the specified columns of the corresponding input table row, in the order given in Input and with the scalars if Input interspersed as specified in Input. [0319]
  • 4. Query (program structure, non-kernel) [0320]
  • 1. query<FunctionName>(n1, . . . , InJ)→[Out1, . . . , OutK][0321]
  • FunctionName is a scalar string value. This operator has the same definition as: evaluate(FunctionNameItem, In1, . . . , InJ)→[Out1, . . . , OutK]. FunctionNameItem is a scalar item with the value of FunctionName. See the evaluate operator definition. [0322]
    • Advantages
  • The system and method just described allows: [0323]
  • 1. Scalability through composition: the preferred embodiments use theorem-based compositions of sets into tables and as a result sequences of sets may be composed into tables which are much larger than can be effectively handled by small operating system file systems. [0324]
  • 2. Set-Theoretic Programming Language: a query in the preferred embodiments is a form of set language program. The query language is diagrammatic, supports parameterized queries, recursive functions, and decision arbitration components. [0325]
  • 3. Parallel Query Evaluation: the preferred embodiments schedule evaluation and solving based upon operation input readiness. [0326]
  • 4. Second Order Logic Theorem Proving: a preferred embodiments accomplish a second order logic theorem proving system. This includes theorems for translating from a high level form into a low level form, theorems that recognize and properly set up the low level executable form, theorems that optimize sequences of instruction to ensure proper preparation prior to execution of certain types of instructions and to remove redundant instructions and perform local optimizations based on set-theoretic identities, and theorems for global optimization, such as identifying cache results that may be used for a current query. [0327]
  • 5. Adaptive Physical Design: the extensional form of sets forms the physical “building blocks” of the data base. As the extensional forms are added or deleted from the database through the use of the system, the physical design of the database changes. These changes reflect optimizations for disc space and set or structure utilization. [0328]
  • 6. All types or relational types: the architecture supports virtually all types as relational types, even pictures, sounds, and tables. This means that these complex or non-traditional types can be used as keys in relational operations such as various selection operations. [0329]
  • 7. Composition and Decomposition of Types: types may be composed and decomposed to create new types or break up domains or compose domains within tables and between tables. For instance, a date may be decomposed into three integers, and likewise, three integers may be composed into a date. A name may be decomposed into three name part strings and three strings may be composed into a name. [0330]
  • 8. All aspects of the server may be utilized in queries: for instance, a user may query the system concerning the current set of active clients, number of sets which are greater than three million records, or how many sets have been created within the last three weeks. Most important, queries may be used to question the system about the form of the database as it exists, the domains and sets present, and the relation of names, types and size of sets as a match for possible operations. The results of these query functions may be used to guide query execution, to post to the user such things as consistency and normalization information about tables and functional dependency information and possible join opportunities between tables and with end tables. [0331]
  • 9. Peer-client type and server architecture: each serve may be a client to another server allowing databases to be shared among different machines and to exist on multiple machines. [0332]
  • 10. Database composition: this allows users to create databases and then share the results of analysis and optimization with other databases. The sharing occurs through inheritance. The sets that exist in a “super” database are available to all databases that are derived from the super database. [0333]
  • 11. Programming Language. Each operator and function call includes table and column definitions for each of its output sets, eliminating the need for separate from the query schema definitions processing or storage; i.e., eliminating the need to maintain and work with schema definitions. [0334]
  • It will be apparent from the following description that alternative arrangements may be easily employed and that they should be considered within the scope of the invention. For example, a multi-tiered approach may be used in which the server logic is divided for execution on multiple, hierarchically-arranged servers. For example, [0335] server 104 may be implemented on multiple computer nodes to improve performance, distribute processing, improve fault tolerance, or the like. Skilled artisans will appreciate, upon reading the disclosure, that alternative data arrangements may be substituted.

Claims (1)

What is claimed is:
1. A method for exploring relationships in data stored in a computer readable medium, comprising the steps of:
providing a set of operators that includes relational operators and non-relational operators;
receiving a query having at least one operator chosen from the set of operators; transforming the query into a set program having at least one operation structure, corresponding to the at least one operator;
type-independently, and without inherent meaning, performing an operation, corresponding to the at least one operator, and
providing a data structure that cooperates with the operation structure and handles data access and storage associated with the operation.
US10/629,125 1998-10-05 2003-07-29 Data exploration system and method Abandoned US20040073546A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/629,125 US20040073546A1 (en) 1998-10-05 2003-07-29 Data exploration system and method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/166,556 US6601058B2 (en) 1998-10-05 1998-10-05 Data exploration system and method
US10/629,125 US20040073546A1 (en) 1998-10-05 2003-07-29 Data exploration system and method

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/166,556 Continuation US6601058B2 (en) 1998-10-05 1998-10-05 Data exploration system and method

Publications (1)

Publication Number Publication Date
US20040073546A1 true US20040073546A1 (en) 2004-04-15

Family

ID=22603818

Family Applications (2)

Application Number Title Priority Date Filing Date
US09/166,556 Expired - Lifetime US6601058B2 (en) 1998-10-05 1998-10-05 Data exploration system and method
US10/629,125 Abandoned US20040073546A1 (en) 1998-10-05 2003-07-29 Data exploration system and method

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09/166,556 Expired - Lifetime US6601058B2 (en) 1998-10-05 1998-10-05 Data exploration system and method

Country Status (3)

Country Link
US (2) US6601058B2 (en)
AU (1) AU1095200A (en)
WO (1) WO2000020996A1 (en)

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060090125A1 (en) * 2004-05-24 2006-04-27 Wolfgang Becker Interface-controlled display of a matrix document in regions
US20060101394A1 (en) * 2004-05-24 2006-05-11 Wolfgang Becker Computer-implemented outputting of table rows to an output device
US20070115282A1 (en) * 2005-11-18 2007-05-24 David Turner Server-client architecture in medical imaging
US20070214134A1 (en) * 2006-03-09 2007-09-13 Microsoft Corporation Data parsing with annotated patterns
US20070282712A1 (en) * 2006-06-02 2007-12-06 Matthew Royce Ullman Source-and venue-specific inventory data processing and identification system
US20070299836A1 (en) * 2006-06-23 2007-12-27 Xue Qiao Hou Database query language transformation method, transformation apparatus and database query system
US20080154868A1 (en) * 2006-12-20 2008-06-26 International Business Machines Corporation Method and apparatus for xml query evaluation using early-outs and multiple passes
US7499917B2 (en) * 2005-01-28 2009-03-03 International Business Machines Corporation Processing cross-table non-Boolean term conditions in database queries
US20100082778A1 (en) * 2008-10-01 2010-04-01 Matt Muilenburg Systems and methods for configuring a network of affiliated websites
US20100082780A1 (en) * 2008-10-01 2010-04-01 Matt Muilenburg Systems and methods for configuring a website having a plurality of operational modes
US7958164B2 (en) 2006-02-16 2011-06-07 Microsoft Corporation Visual design of annotated regular expression
US20120265773A1 (en) * 2011-04-12 2012-10-18 Microsoft Corporation Query-based diagrammatic presentation of data
US20120323973A1 (en) * 2011-06-16 2012-12-20 Hon Hai Precision Industry Co., Ltd. System and method for converting component data
US9471630B2 (en) * 2012-03-06 2016-10-18 International Business Machines Corporation Efficient query processing on ordered views
US10326858B2 (en) 2017-05-23 2019-06-18 Cdk Global, Llc System and method for dynamically generating personalized websites
US10332068B2 (en) 2016-04-21 2019-06-25 Cdk Global, Llc Systems and methods for stocking an automobile
US10482475B2 (en) 2011-02-10 2019-11-19 Adp Dealer Services, Inc. Systems and methods for providing targeted advertising
US10853769B2 (en) 2016-04-21 2020-12-01 Cdk Global Llc Scheduling an automobile service appointment in a dealer service bay based on diagnostic trouble codes and service bay attributes
US10867285B2 (en) 2016-04-21 2020-12-15 Cdk Global, Llc Automatic automobile repair service scheduling based on diagnostic trouble codes and service center attributes
US11080734B2 (en) 2013-03-15 2021-08-03 Cdk Global, Llc Pricing system for identifying prices for vehicles offered by vehicle dealerships and other entities
US11080105B1 (en) 2020-11-18 2021-08-03 Cdk Global, Llc Systems, methods, and apparatuses for routing API calls
US11190608B2 (en) 2018-03-21 2021-11-30 Cdk Global Llc Systems and methods for an automotive commerce exchange
US11501351B2 (en) 2018-03-21 2022-11-15 Cdk Global, Llc Servers, systems, and methods for single sign-on of an automotive commerce exchange
US11514021B2 (en) 2021-01-22 2022-11-29 Cdk Global, Llc Systems, methods, and apparatuses for scanning a legacy database
US11803535B2 (en) 2021-05-24 2023-10-31 Cdk Global, Llc Systems, methods, and apparatuses for simultaneously running parallel databases

Families Citing this family (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6601058B2 (en) * 1998-10-05 2003-07-29 Michael Forster Data exploration system and method
GB9903697D0 (en) * 1999-02-19 1999-04-14 Pattern Computing Limited A computer-based method for matching patterns
US6681220B1 (en) * 1999-05-28 2004-01-20 International Business Machines Corporation Reduction and optimization of information processing systems
ES2288467T3 (en) * 1999-10-26 2008-01-16 Iontas Limited SUPERVISION OF THE USE OF COMPUTERS.
US6571233B2 (en) * 2000-12-06 2003-05-27 International Business Machines Corporation Optimization of SQL queries using filtering predicates
US6662175B1 (en) * 2001-05-08 2003-12-09 Ncr Corporation Semantic query optimization using value correlation
US7254810B2 (en) * 2002-04-18 2007-08-07 International Business Machines Corporation Apparatus and method for using database knowledge to optimize a computer program
US7711675B2 (en) * 2002-07-22 2010-05-04 Microsoft Corporation Database simulation of data types
JP2004171249A (en) * 2002-11-20 2004-06-17 Hitachi Ltd Backup execution decision method for database
WO2005081136A1 (en) * 2004-01-20 2005-09-01 Cedars-Sinai Medical Center Data repository system
US7519577B2 (en) * 2003-06-23 2009-04-14 Microsoft Corporation Query intermediate language method and system
US7272591B1 (en) * 2004-01-22 2007-09-18 Ncr Corporation Method and system for updating value correlation optimizations
US7440942B2 (en) * 2004-02-05 2008-10-21 The State Of Oregon Acting By And Through The State Board Of Higher Education On Behalf Of The University Of Oregon Database manipulations using group theory
US20050187905A1 (en) * 2004-02-05 2005-08-25 Dixon Heidi E. Database manipulations using group theory
US7446769B2 (en) * 2004-02-10 2008-11-04 International Business Machines Corporation Tightly-coupled synchronized selection, filtering, and sorting between log tables and log charts
US7587685B2 (en) * 2004-02-17 2009-09-08 Wallace James H Data exploration system
US8055683B2 (en) * 2004-10-14 2011-11-08 International Business Machines Corporation Management of relationships between database tables
JP4855710B2 (en) * 2005-04-28 2012-01-18 株式会社東芝 Software plug-in method and application program
US20070239691A1 (en) * 2006-04-06 2007-10-11 Carlos Ordonez Optimization techniques for linear recursive queries in sql
US7877370B2 (en) * 2006-05-15 2011-01-25 Algebraix Data Corporation Systems and methods for data storage and retrieval using algebraic relations composed from query language statements
US7720806B2 (en) * 2006-05-15 2010-05-18 Algebraix Data Corporation Systems and methods for data manipulation using multiple storage formats
US7613734B2 (en) * 2006-05-15 2009-11-03 Xsprada Corporation Systems and methods for providing data sets using a store of albegraic relations
US7797319B2 (en) * 2006-05-15 2010-09-14 Algebraix Data Corporation Systems and methods for data model mapping
US7769754B2 (en) * 2006-05-15 2010-08-03 Algebraix Data Corporation Systems and methods for data storage and retrieval using algebraic optimization
US7865503B2 (en) * 2006-05-15 2011-01-04 Algebraix Data Corporation Systems and methods for data storage and retrieval using virtual data sets
US7865515B2 (en) * 2006-08-28 2011-01-04 Microsoft Corporation Server side bucketization of parameterized queries
US7774189B2 (en) * 2006-12-01 2010-08-10 International Business Machines Corporation System and method for simulating data flow using dataflow computing system
US8145474B1 (en) 2006-12-22 2012-03-27 Avaya Inc. Computer mediated natural language based communication augmented by arbitrary and flexibly assigned personality classification systems
US7860863B2 (en) * 2007-09-05 2010-12-28 International Business Machines Corporation Optimization model for processing hierarchical data in stream systems
US7941460B2 (en) * 2007-09-05 2011-05-10 International Business Machines Corporation Compilation model for processing hierarchical data in stream systems
US8140547B2 (en) * 2007-10-02 2012-03-20 International Business Machines Corporation Systems, methods and computer products for a monitoring context generator
US8315990B2 (en) * 2007-11-08 2012-11-20 Microsoft Corporation Consistency sensitive streaming operators
US8161380B2 (en) * 2008-06-26 2012-04-17 International Business Machines Corporation Pipeline optimization based on polymorphic schema knowledge
US20100088325A1 (en) * 2008-10-07 2010-04-08 Microsoft Corporation Streaming Queries
US10152504B2 (en) 2009-03-11 2018-12-11 Actian Netherlands B.V. Column-store database architecture utilizing positional delta tree update system and methods
US8266172B2 (en) * 2009-04-03 2012-09-11 Microsoft Corporation Data parallel query analysis
US8413169B2 (en) * 2009-10-21 2013-04-02 Microsoft Corporation Time-based event processing using punctuation events
US9158816B2 (en) 2009-10-21 2015-10-13 Microsoft Technology Licensing, Llc Event processing with XML query based on reusable XML query template
CN102314514B (en) * 2011-09-20 2013-01-09 北京航空航天大学 Scoping method of table data structuration
US9922090B1 (en) 2012-03-27 2018-03-20 Actian Netherlands, B.V. System and method for automatic vertical decomposition of a table for improving input/output and memory utilization in a database
US8583687B1 (en) 2012-05-15 2013-11-12 Algebraix Data Corporation Systems and methods for indirect algebraic partitioning
US8990232B2 (en) * 2012-05-15 2015-03-24 Telefonaktiebolaget L M Ericsson (Publ) Apparatus and method for parallel regular expression matching
US11507574B1 (en) * 2013-03-13 2022-11-22 Actian Netherlands B.V. Adaptive selection of a processing method based on observed performance for improved and robust system efficiency
US9922032B2 (en) * 2013-12-02 2018-03-20 Qbase, LLC Featured co-occurrence knowledge base from a corpus of documents
US10928970B2 (en) * 2014-07-18 2021-02-23 Apple Inc. User-interface for developing applications that apply machine learning
US20160117371A1 (en) * 2014-10-23 2016-04-28 Microsoft Corporation Job authoring with data preview
GB2535710A (en) * 2015-02-24 2016-08-31 Siemens Ag Computer device and method for detecting correlations within data
US10114864B1 (en) * 2015-05-19 2018-10-30 Amazon Technologies, Inc. List element query support and processing
WO2017031082A1 (en) * 2015-08-14 2017-02-23 California Institute Of Technology Algebraic query language (aql) database management system
US9396248B1 (en) 2016-01-04 2016-07-19 International Business Machines Corporation Modified data query function instantiations
US11580444B2 (en) 2019-04-16 2023-02-14 Apple Inc. Data visualization machine learning model performance

Citations (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4989132A (en) * 1988-10-24 1991-01-29 Eastman Kodak Company Object-oriented, logic, and database programming tool with garbage collection
US5019961A (en) * 1989-04-05 1991-05-28 Cadware, Inc. Computer apparatus and method for logical modelling
US5121494A (en) * 1989-10-05 1992-06-09 Ibm Corporation Joining two database relations on a common field in a parallel relational database field
US5193185A (en) * 1989-05-15 1993-03-09 David Lanter Method and means for lineage tracing of a spatial information processing and database system
US5226111A (en) * 1987-01-06 1993-07-06 Hewlett-Packard Company Organization of theory based systems
US5259066A (en) * 1990-04-16 1993-11-02 Schmidt Richard Q Associative program control
US5276870A (en) * 1987-12-11 1994-01-04 Hewlett-Packard Company View composition in a data base management system
US5301317A (en) * 1992-04-27 1994-04-05 International Business Machines Corporation System for adapting query optimization effort to expected execution time
US5325525A (en) * 1991-04-04 1994-06-28 Hewlett-Packard Company Method of automatically controlling the allocation of resources of a parallel processor computer system by calculating a minimum execution time of a task and scheduling subtasks against resources to execute the task in the minimum time
US5418942A (en) * 1989-07-06 1995-05-23 Krawchuk; Kenneth V. System and method for storing and managing information
US5418943A (en) * 1991-10-23 1995-05-23 At&T Corp. Information system with knowledge base and data base
US5469568A (en) * 1993-01-07 1995-11-21 International Business Machines Corporation Method for choosing largest selectivities among eligible predicates of join equivalence classes for query optimization
US5481700A (en) * 1991-09-27 1996-01-02 The Mitre Corporation Apparatus for design of a multilevel secure database management system based on a multilevel logic programming system
US5495606A (en) * 1993-11-04 1996-02-27 International Business Machines Corporation System for parallel processing of complex read-only database queries using master and slave central processor complexes
US5515531A (en) * 1992-05-29 1996-05-07 Hitachi, Ltd. Parallel database processing system and retrieval method using secondary key
US5537590A (en) * 1993-08-05 1996-07-16 Amado; Armando Apparatus for applying analysis rules to data sets in a relational database to generate a database of diagnostic records linked to the data sets
US5546571A (en) * 1988-12-19 1996-08-13 Hewlett-Packard Company Method of recursively deriving and storing data in, and retrieving recursively-derived data from, a computer database system
US5548770A (en) * 1993-02-25 1996-08-20 Data Parallel Systems, Inc. Method and apparatus for improving retrieval of data from a database
US5548755A (en) * 1995-02-17 1996-08-20 International Business Machines Corporation System for optimizing correlated SQL queries in a relational database using magic decorrelation
US5548749A (en) * 1993-10-29 1996-08-20 Wall Data Incorporated Semantic orbject modeling system for creating relational database schemas
US5550971A (en) * 1993-06-30 1996-08-27 U S West Technologies, Inc. Method and system for generating a user interface adaptable to various database management systems
US5555404A (en) * 1992-03-17 1996-09-10 Telenor As Continuously available database server having multiple groups of nodes with minimum intersecting sets of database fragment replicas
US5560006A (en) * 1991-05-15 1996-09-24 Automated Technology Associates, Inc. Entity-relation database
US5574900A (en) * 1994-02-25 1996-11-12 International Business Machines Corporation System and method for optimizing parallel processing of database queries
US5590319A (en) * 1993-12-15 1996-12-31 Information Builders, Inc. Query processor for parallel processing in homogenous and heterogenous databases
US5701454A (en) * 1995-01-17 1997-12-23 International Business Machines Corporation Simplification of SQL queries using generalized inference propagation and generalized transitive closure
US6601058B2 (en) * 1998-10-05 2003-07-29 Michael Forster Data exploration system and method
US20030187864A1 (en) * 2002-04-02 2003-10-02 Mcgoveran David O. Accessing and updating views and relations in a relational database

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5596682A (en) 1989-07-18 1997-01-21 Kabushiki Kaisha Toshiba Method and apparatus of automated theorem proving for information processing
US5606693A (en) 1991-10-02 1997-02-25 International Business Machines Corporation Distributed database management over a network
US5307445A (en) 1991-12-02 1994-04-26 International Business Machines Corporation Query optimization by type lattices in object-oriented logic programs and deductive databases
JP3183736B2 (en) 1992-12-28 2001-07-09 富士通株式会社 Dynamic change method of database logical data structure
US5687369A (en) 1993-09-02 1997-11-11 International Business Machines Corporation Selecting buckets for redistributing data between nodes in a parallel database in the incremental mode
US5630124A (en) 1993-12-06 1997-05-13 International Business Machines Corporation System and method for assuring atomicity of distributed update requests in a parallel database
US5655116A (en) 1994-02-28 1997-08-05 Lucent Technologies Inc. Apparatus and methods for retrieving information
US5664171A (en) 1994-04-14 1997-09-02 International Business Machines Corporation System and method for query optimization using quantile values of a large unordered data set
US5664172A (en) 1994-07-19 1997-09-02 Oracle Corporation Range-based query optimizer
US5666524A (en) 1994-08-31 1997-09-09 Price Waterhouse Llp Parallel processing system for traversing a transactional database
US5588150A (en) 1994-09-29 1996-12-24 International Business Machines Corporation Push down optimization in a distributed, multi-database system
US5615341A (en) 1995-05-08 1997-03-25 International Business Machines Corporation System and method for mining generalized association rules in databases
US5655080A (en) 1995-08-14 1997-08-05 International Business Machines Corporation Distributed hash group-by cooperative processing
US5953716A (en) * 1996-05-30 1999-09-14 Massachusetts Inst Technology Querying heterogeneous data sources distributed over a network using context interchange
US6006224A (en) * 1997-02-14 1999-12-21 Organicnet, Inc. Crucible query system
US5963932A (en) * 1997-04-29 1999-10-05 Oracle Corporation Method and apparatus for transforming queries
US6327587B1 (en) * 1998-10-05 2001-12-04 Digital Archaeology, Inc. Caching optimization with disk and/or memory cache management

Patent Citations (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5226111A (en) * 1987-01-06 1993-07-06 Hewlett-Packard Company Organization of theory based systems
US5276870A (en) * 1987-12-11 1994-01-04 Hewlett-Packard Company View composition in a data base management system
US4989132A (en) * 1988-10-24 1991-01-29 Eastman Kodak Company Object-oriented, logic, and database programming tool with garbage collection
US5546571A (en) * 1988-12-19 1996-08-13 Hewlett-Packard Company Method of recursively deriving and storing data in, and retrieving recursively-derived data from, a computer database system
US5019961A (en) * 1989-04-05 1991-05-28 Cadware, Inc. Computer apparatus and method for logical modelling
US5193185A (en) * 1989-05-15 1993-03-09 David Lanter Method and means for lineage tracing of a spatial information processing and database system
US5418942A (en) * 1989-07-06 1995-05-23 Krawchuk; Kenneth V. System and method for storing and managing information
US5121494A (en) * 1989-10-05 1992-06-09 Ibm Corporation Joining two database relations on a common field in a parallel relational database field
US5259066A (en) * 1990-04-16 1993-11-02 Schmidt Richard Q Associative program control
US5325525A (en) * 1991-04-04 1994-06-28 Hewlett-Packard Company Method of automatically controlling the allocation of resources of a parallel processor computer system by calculating a minimum execution time of a task and scheduling subtasks against resources to execute the task in the minimum time
US5560006A (en) * 1991-05-15 1996-09-24 Automated Technology Associates, Inc. Entity-relation database
US5481700A (en) * 1991-09-27 1996-01-02 The Mitre Corporation Apparatus for design of a multilevel secure database management system based on a multilevel logic programming system
US5418943A (en) * 1991-10-23 1995-05-23 At&T Corp. Information system with knowledge base and data base
US5555404A (en) * 1992-03-17 1996-09-10 Telenor As Continuously available database server having multiple groups of nodes with minimum intersecting sets of database fragment replicas
US5301317A (en) * 1992-04-27 1994-04-05 International Business Machines Corporation System for adapting query optimization effort to expected execution time
US5515531A (en) * 1992-05-29 1996-05-07 Hitachi, Ltd. Parallel database processing system and retrieval method using secondary key
US5469568A (en) * 1993-01-07 1995-11-21 International Business Machines Corporation Method for choosing largest selectivities among eligible predicates of join equivalence classes for query optimization
US5542073A (en) * 1993-01-07 1996-07-30 International Business Machines Corporation Computer program product for choosing largest selectivities among eligible predicates of join equivalence classes for query optimization
US5548770A (en) * 1993-02-25 1996-08-20 Data Parallel Systems, Inc. Method and apparatus for improving retrieval of data from a database
US5550971A (en) * 1993-06-30 1996-08-27 U S West Technologies, Inc. Method and system for generating a user interface adaptable to various database management systems
US5537590A (en) * 1993-08-05 1996-07-16 Amado; Armando Apparatus for applying analysis rules to data sets in a relational database to generate a database of diagnostic records linked to the data sets
US5548749A (en) * 1993-10-29 1996-08-20 Wall Data Incorporated Semantic orbject modeling system for creating relational database schemas
US5495606A (en) * 1993-11-04 1996-02-27 International Business Machines Corporation System for parallel processing of complex read-only database queries using master and slave central processor complexes
US5590319A (en) * 1993-12-15 1996-12-31 Information Builders, Inc. Query processor for parallel processing in homogenous and heterogenous databases
US5574900A (en) * 1994-02-25 1996-11-12 International Business Machines Corporation System and method for optimizing parallel processing of database queries
US5701454A (en) * 1995-01-17 1997-12-23 International Business Machines Corporation Simplification of SQL queries using generalized inference propagation and generalized transitive closure
US5548755A (en) * 1995-02-17 1996-08-20 International Business Machines Corporation System for optimizing correlated SQL queries in a relational database using magic decorrelation
US6601058B2 (en) * 1998-10-05 2003-07-29 Michael Forster Data exploration system and method
US20030187864A1 (en) * 2002-04-02 2003-10-02 Mcgoveran David O. Accessing and updating views and relations in a relational database

Cited By (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060101394A1 (en) * 2004-05-24 2006-05-11 Wolfgang Becker Computer-implemented outputting of table rows to an output device
US20060090125A1 (en) * 2004-05-24 2006-04-27 Wolfgang Becker Interface-controlled display of a matrix document in regions
US7477234B2 (en) 2004-05-24 2009-01-13 Sap Ag Interface-controlled display of a matrix document in regions
US7499917B2 (en) * 2005-01-28 2009-03-03 International Business Machines Corporation Processing cross-table non-Boolean term conditions in database queries
US20070115282A1 (en) * 2005-11-18 2007-05-24 David Turner Server-client architecture in medical imaging
US7890573B2 (en) * 2005-11-18 2011-02-15 Toshiba Medical Visualization Systems Europe, Limited Server-client architecture in medical imaging
US7958164B2 (en) 2006-02-16 2011-06-07 Microsoft Corporation Visual design of annotated regular expression
US20070214134A1 (en) * 2006-03-09 2007-09-13 Microsoft Corporation Data parsing with annotated patterns
US7860881B2 (en) * 2006-03-09 2010-12-28 Microsoft Corporation Data parsing with annotated patterns
US20070282713A1 (en) * 2006-06-02 2007-12-06 Matthew Royce Ullman Source- and venue-specific inventory data processing and identification system
US8538894B2 (en) 2006-06-02 2013-09-17 Adp Dealer Services, Inc. Source- and venue-specific inventory data processing and identification system
US20070282712A1 (en) * 2006-06-02 2007-12-06 Matthew Royce Ullman Source-and venue-specific inventory data processing and identification system
US8275717B2 (en) 2006-06-02 2012-09-25 Adp Dealer Services, Inc. Source- and venue-specific inventory data processing and identification system
US20070299836A1 (en) * 2006-06-23 2007-12-27 Xue Qiao Hou Database query language transformation method, transformation apparatus and database query system
US20090094216A1 (en) * 2006-06-23 2009-04-09 International Business Machines Corporation Database query language transformation method, transformation apparatus and database query system
US7668818B2 (en) * 2006-06-23 2010-02-23 International Business Machines Corporation Database query language transformation method, transformation apparatus and database query system
US9223827B2 (en) 2006-06-23 2015-12-29 International Business Machines Corporation Database query language transformation method, transformation apparatus and database query system
US7716210B2 (en) * 2006-12-20 2010-05-11 International Business Machines Corporation Method and apparatus for XML query evaluation using early-outs and multiple passes
US20080154868A1 (en) * 2006-12-20 2008-06-26 International Business Machines Corporation Method and apparatus for xml query evaluation using early-outs and multiple passes
US20100082780A1 (en) * 2008-10-01 2010-04-01 Matt Muilenburg Systems and methods for configuring a website having a plurality of operational modes
US8051159B2 (en) 2008-10-01 2011-11-01 The Cobalt Group, Inc. Systems and methods for configuring a network of affiliated websites
US20100082778A1 (en) * 2008-10-01 2010-04-01 Matt Muilenburg Systems and methods for configuring a network of affiliated websites
US8438310B2 (en) 2008-10-01 2013-05-07 Adp Dealer Services, Inc. Systems and methods for configuring a website having a plurality of operational modes
US10482475B2 (en) 2011-02-10 2019-11-19 Adp Dealer Services, Inc. Systems and methods for providing targeted advertising
US8924385B2 (en) * 2011-04-12 2014-12-30 Microsoft Corporation Query-based diagrammatic presentation of data
US20120265773A1 (en) * 2011-04-12 2012-10-18 Microsoft Corporation Query-based diagrammatic presentation of data
US20120323973A1 (en) * 2011-06-16 2012-12-20 Hon Hai Precision Industry Co., Ltd. System and method for converting component data
US8825714B2 (en) * 2011-06-16 2014-09-02 Hon Hai Precision Industry Co., Ltd. System and method for converting component data
US9471630B2 (en) * 2012-03-06 2016-10-18 International Business Machines Corporation Efficient query processing on ordered views
US11080734B2 (en) 2013-03-15 2021-08-03 Cdk Global, Llc Pricing system for identifying prices for vehicles offered by vehicle dealerships and other entities
US10332068B2 (en) 2016-04-21 2019-06-25 Cdk Global, Llc Systems and methods for stocking an automobile
US10853769B2 (en) 2016-04-21 2020-12-01 Cdk Global Llc Scheduling an automobile service appointment in a dealer service bay based on diagnostic trouble codes and service bay attributes
US10867285B2 (en) 2016-04-21 2020-12-15 Cdk Global, Llc Automatic automobile repair service scheduling based on diagnostic trouble codes and service center attributes
US10326858B2 (en) 2017-05-23 2019-06-18 Cdk Global, Llc System and method for dynamically generating personalized websites
US11190608B2 (en) 2018-03-21 2021-11-30 Cdk Global Llc Systems and methods for an automotive commerce exchange
US11501351B2 (en) 2018-03-21 2022-11-15 Cdk Global, Llc Servers, systems, and methods for single sign-on of an automotive commerce exchange
US11616856B2 (en) 2018-03-21 2023-03-28 Cdk Global, Llc Systems and methods for an automotive commerce exchange
US11080105B1 (en) 2020-11-18 2021-08-03 Cdk Global, Llc Systems, methods, and apparatuses for routing API calls
US11514021B2 (en) 2021-01-22 2022-11-29 Cdk Global, Llc Systems, methods, and apparatuses for scanning a legacy database
US11803535B2 (en) 2021-05-24 2023-10-31 Cdk Global, Llc Systems, methods, and apparatuses for simultaneously running parallel databases

Also Published As

Publication number Publication date
WO2000020996A1 (en) 2000-04-13
AU1095200A (en) 2000-04-26
US20020083049A1 (en) 2002-06-27
US6601058B2 (en) 2003-07-29

Similar Documents

Publication Publication Date Title
US6327587B1 (en) Caching optimization with disk and/or memory cache management
US6601058B2 (en) Data exploration system and method
US7739223B2 (en) Mapping architecture for arbitrary data models
US4769772A (en) Automated query optimization method using both global and parallel local optimizations for materialization access planning for distributed databases
US5448727A (en) Domain based partitioning and reclustering of relations in object-oriented relational database management systems
US7383273B2 (en) System and method for managing object to relational one-to-many mapping
US4930071A (en) Method for integrating a knowledge-based system with an arbitrary database system
US7409385B2 (en) Method, system and program for executing a query having a UNION operator
USRE42664E1 (en) Method and apparatus for implementing parallel operations in a database management system
US7457797B2 (en) Method and apparatus for associating logical conditions with the re-use of a database query execution strategy
US9286393B2 (en) Performing a function on rows of data determined from transitive relationships between columns
US6449606B1 (en) Using a materialized view to process a related query containing an antijoin
US20070038658A1 (en) Communication optimization for parallel execution of user-defined table functions
US20060155719A1 (en) Complex event discovery in event databases
EP1637993A2 (en) Impact analysis in an object model
US20120072412A1 (en) Evaluating execution plan changes after a wakeup threshold time
US7958160B2 (en) Executing filter subqueries using a parallel single cursor model
US7213014B2 (en) Apparatus and method for using a predefined database operation as a data source for a different database operation
US6618720B1 (en) Common spool files for maintaining join indexes
US5953715A (en) Utilizing pseudotables as a method and mechanism providing database monitor information
Zhu et al. Developing a dynamic materialized view index for efficiently discovering usable views for progressive queries
Shu Using constraint satisfaction for view update
EP0378367A2 (en) Functional database system
Finance et al. Query processing in IRO-DB
JP3538322B2 (en) Database management system and query processing method

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION