WO1998002825A2 - Computer implemented machine learning method and system - Google Patents

Abstract

One or more machine code entities (50) such as programs or functions are created which represent solutions to a problem and are directly executable by a computer system (10). The programs (50) are created and altered by a program (32) in a higher level language such as 'C' which is not directly executable, but requires translation into executable machine code through compilation, interpretation, translation, etc. The entities (50) are initially created as an integer array (32b) that can be altered by the program (32) as data, and are executed by the program (32) by recasting a pointer to the array as a function type. The entities (50) are evaluated by executing them with training data as inputs, and calculating fitnesses based on a predetermined criterion. The entities (50) are then altered based on their fitnesses using a genetic machine learning algorithm by recasting the pointer to the array as a data (e.g. integer) type. This process is iteratively repeated until an end criterion is reached. The entities (50) evolve in such a manner as to improve their fitness, and one entity is ultimately produced which represents an optimal solution to the problem. Each entity (50) includes a plurality of directly executable machine code instructions (52a, 52b, 52c), a header (50a), a footer (50c), and a return instruction (50d). The instructions (50) can include branch instructions which enable subroutines, leaf functions, external function calls, recursion, and loops. The system (10) can be implemented on an integrated circuit chip (90), with the entities (50) stored in high speed memory (96) in a central processing unit (92). The system (10) can be used to control an autonomous agent such as a robot.

Description

COMPUTER IMPLEMENTED MACHINE LEARNING METHOD AND SYSTEM

Technical Field

The present invention generally relates to the art of computerized computation systems for performing repeated computations on data that is not known until a computer user is running the system ("run-time", and "run-time data") and more specifically to a system of creating, initializing, storing, altering and executing both tlie run-time data and the computer code necessary to execute the repeated computations (the "related code") in native machine code operating on a register machine. The present invention relates also to tlie art of computerized learning systems (which are usually characterized by tlie need to perform repeated computations on run-time data) and more specifically to a register machine learning method in which the information and/or tlie computer program(s) that constitute solutions to a problem are created, initialized, stored, altered and executed in native machine code by using a higher level programming language to produce an optimized solution tlirough tlie direct application of learning algorithms to tlie information stored in the native machine code.

Background Art

Machine learning systems have been proposed in the art for tl e solution of problems such as classification, prediction of time-series data, symbolic regression, optimal control, etc. Examples of various machine learning systems are neural networks, fuzzy networks, genetic algorithms (including genetic programming and classifier systems), Evolutionary Strategies, Evolutionary Programming, AD ATE program induction, cellular automata. Box Jenkins optimization, ARMA optimization and many others. Rather than applying a direct computational approach, these systems create one or more proposed solutions in the form of data and computer program entities, and iteratively alter die data and/or entities for the purpose of finding an optimal solution to a problem. One such approach is described in, for example, U.S. Patent No. 4,935,877, entitled "NON-LINEAR GENETIC ALGORITHMS FOR SOLVING PROBLEMS", issued June 19, 1990 to John Koza.

The set of practically solvable problems is highly related to tl e efficiency of tlie algorithm and implementation. It is therefore important to minimize the overhead involved in executing any machine learning system. Machine learning systems create learned solutions to problems. These solutions have two elements:

(1) Elements that hold the information that is learned during the execution of the machine learning system (tlie "Learned Elements"); and (2) Elements that are necessary to convert the Learned Elements into meaningful action or output by the computer (the "Conversion Elements"). Existing machine learning approaches other than tlie present invention can be classified into two categories in terms of how they store and execute both the Learned Elements and tlie Conversion Elements. Those two categories are compiler based systems and interpreted systems.

An interpreted system is written in a high level language such as LISP and both the Learned and

Conversion Elements are held in data-like structures such as LISP lists, which are converted ("interpreted") into native machine code at run-time by an interpreter. The interpreter, itself, also contains Conversion Elements for such interpreted systems. So for example, U.S. Patent No. 4,935,877 uses, as its Learning Elements various high level LISP expressions, customized for the problem at hand, to represent, symbolically, a "computer program " That is, a high level "program" structure symbolized by the LISP List, is itself the subject of learning in that system. In this system, the Learned Elements are represented as a hierarchical tree structure. This solution gives good flexibility and tlie ability to customize the language depending on tl e constraints of the problem at hand. The principal disadvantage of this interpreting approach to machine learning is that the Learned

Elements and many of tlie Conversion Elements are stored in high level, symbolic data-like structures. Computers can operate only by executing native machine code. Thus, interpreting machine learning systems learn by modifying high level symbolic representations (tlie Learned Elements) that are, ultimately, converted into machine code by tlie interpreter at run-time. Tlie need to convert (interpret) the Learned Elements and some of tlie Conversion Elements into native machine code at run-time before any useful action or output may be had from a computer is very time consuming and involves a large amount of overhead in machine resources such as CPU time, RAM memory, and disk space. In effect, all of tlie Learned Elements and tlie Conversion Elements in tlie LISP List are treated as run-time data that must be accessed and converted to machine code before any useful action or output may be had. Simply put, interpreted systems are slow and use a lot of a computer system's resources.

Other machine learning systems are written in high level compiled computer languages such a C, C++, Pascal and so forth. This is the "compiler based" approach to machine learning. Large amounts of the Conversion Elements in such systems are compiled before run-time into native machine code in such a compiler based approach. Because those compiled Conversion Elements are already in native machine code at run-time, there is no need to interpret tliese Conversion Elements at run-time. Using a compiler based approach instead of an interpreted approach usually results in a substantial speed-up of the execution of the machine learning system, often increasing tlie speed of learning by a factor of ten times. Examples of such compiler based systems are GPC by Walter Tackett (genetic programming), Lil-GP (genetic programming), and EvoC (evolutionary strategies).

The compiler based approach to machine learning, while faster than tlie interpreted approach, must still access run-time data that is stored in data structures held in RAM memory or in some otlicr form of memory such as hard disk. The reason that run-tiine data structures must be accessed in compiler based machine learning systems (or any machine learning system other than the present invention) is that tlie process of learning involves initializing and then altering the Learned Elements at run-time.

For example, the weights in a neural network are Learned Elements for neural network applications. Compiler based neural network systems hold those weights in data structures such as arrays or linked lists. Similarly, compiler based genetic programming systems store symbolic representations of program structures (the Learned Elements in genetic programming) in RAM data structures such as arrays, linked lists or parse trees. In all compiler based machine learning systems, the already compiled Conversion Elements must repeatedly access the Learned Elements (weights or symbolic representations of program structures) from the data structures in RAM memory in order to execute and evaluate the Learned Elements and to modify the Learned Elements according to the learning algorithm that is being used to modify tlie Learned Elements during learning. Such repeated access is necessary before any meaningful output or action may be had from a computer based on the Learned Elements. Such repeated accesses to RAM data structures is time consuming and uses extensive amounts of RAM to hold the Learned Elements. More generally, computing systems that perform repeated calculations on run-time data may also be categorized as compiler based systems or interpreted systems. They store access, alter and execute run-time data in a manner similar to tlie storage, access, alteration and execution of the Learned Elements in tlie systems described above and are subject to the same limitations of slow execution and system overhead as the systems described above. By tlie plirasc, "repeated calculations (or computations) on (or of) run-time data," this application means tlie execution of one or more instructions that must access one or more elements of run-time data (from data storage such as RAM or hard disk) more than once on tlie same values of tlie run-time data.

DISCLOSURE OF INVENTION The present invention utilizes tlie lowest level binary machine code as tlie "entities" or individuals or solutions. Every individual is a piece of machine code that is called and manipulated by tlie genetic operators.

There is no intermediate language or interpreting part of the program. The machine code program segments are invoked with a standard C function call. Legal and valid C-functions are put together, at run time, directly in memory by tlie genetic algorithm. Tlie present system can therefore be considered as a "compiling" machine learning implementation.

The present system generates binary code directly from an example set, and there are no interpreting steps. The invention uses the real machine instead of a virtual machine, and any loss in flexibility will be well compensated for by increased efficiency.

More specifically, one or more machine code entities such as functions are created which represent (1 ) solutions to a problem or (2) code that will perform repeated calculations on "run-time data" that is encapsulated into the machine code. These entities are directly executable by a computer. The programs are created and altered by a program in a higher level language such as "C" which is not directly executable, but requires translation into executable machine code through compilation, interpretation, translation, etc.

The entities are initially created as an integer array that can be altered by the program as data, and are executed by tl e program by recasting a pointer to tlie array as a function type. The entities are evaluated by executing them with training data (as defined elsewhere) as inputs, and calculating "fitnesses" based on a predetermined criterion or by recovering the output as the result of one of the repeated calculations on run-time data.

After one or more "executions," tlie entities are then altered by recasting the pointer to the array as a data (e.g. integer) type. Or the original data pointer to tlie array may have been typecast earlier to a function pointer in a way that did not permanently change the type of the pointer. In that case, the original data pointer is used in its original form. This process is iteratively repeated until an end criterion is reached.

In tlie case of machine learning, the entities change in such a manner as to improve their fitness, and one entity is ultimately produced which represents an optimal solution to tlie problem. In the case of repeated calculations on run-time data, the entity permits very rapid calculations to be performed generating usable output from tlie run-time data. Each entity includes a plurality of directly executable machine code instructions, a header, a footer, and a return instruction. The alteration process is controlled such that only valid instructions are produced. The headers, footers and return instructions are protected from alteration.

The system can be implemented on an integrated circuit chip, with tlie entities stored in high speed memory in a central processing unit.

The present invention overcomes the drawbacks of tlie prior art by eliminating all compiling, interpreting or other steps that are required to convert a high level programming language instruction such as a LISP S-expression into machine code prior to execution.

Experiments have demonstrated that tlie present approach can speed up the execution of a machine learning or a repeated calculation system by 1 ,000 times or more as compared to systems which provide potential solutions in tlie foπn of high level "program" expressions in the inteφreted approach. The speedup is in excess of CΛ) times over tlie compiler based approach. This makes possible tlie practical solutions to problems which could not heretofore be solved due to excessive computation time. For example, a solution to a difficult problem can be produced by the present system in hours, whereas a comparable solution might take years using conventional techniques.

These and other features and advantages of tlie present invention will be apparent to those skilled in the art from tlie following detailed description, taken together with the accompanying drawings, in which like reference numerals refer to like parts.

BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a block diagram illustrating a machine learning system according to tlie present invention as implemented on a general puφose computer;

FIG. 2 is a block diagram illustrating a computer program which implements the functionality of tlie present invention as stored in a random access memory of tlie computer of FIG. 1 ;

FIG. 3 is a block diagram illustrating banks of registers in the computer used in tl e preferred implementation of the invention;

FIG. 4 is a flowchart illustrating a method of machine learning according to the invention; FIG. 5 is a diagram illustrating types of machine code instructions which are used by the preferred implementation of the invention;

FIG. 6 is a detailed flowchart illustrating a specific embodiment of the preferred implementation of tlie invention;

FIG. 7 is a diagram illustrating an array of functions that are used by the invention; FIG. 8 is a diagram illustrating an alternative function that can be used by the invention; FIG. 9 is a diagram illustrating the present machine learning system as implemented on a specially designed integrated circuit chip; FIGs. 10a and 10b are diagrams illustrating a genetic uniform crossover operation;

FIGs. 1 la and 1 lb are diagrams illustrating a genetic 2-point "swap" crossover operation; FIGs. 12a and 12b are diagrams illustrating a genetic operator crossover operation; FIGs. 13a and 13b are diagrams illustrating a genetic operand crossover operation; FIG. 14 is a diagram illustrating a genetic operator mutation operation; FIG. 15 is a diagram illustrating a genetic operand mutation operation; FIGs. 16a to 16d in combination constitute a flowchart illustrating a generic implementation of the invention to machine learning;

FIGs. 17a to 17d in combination constitute a flowchart illustrating a generic implementation of the invention to repetitive computations based on run time data;

FIGs. 18a to 18h in combination constitute a flowchart illustrating an machine learning implementation of tlie system which induces machine code functions where the Learned Elements are machine code instruction structure and machine code instruction contents. This is one implementation of what is referred to loosely as Compiling Genetic Programming Systems;

FIGs. 19 to 21 are diagrams illustrating tlie use of functions and registers in accordance with tlie invention; FIGs. 22a to 22k are flowcharts illustrating a detailed implementation of tlie invention;

FIG. 23 is a diagram illustrating non-memory control of an autonomous agent such as a robot according to tlie present invention;

FIG. 24 is a flowchart illustrating tlie non-memory control process of FIG. 23; FIGs. 25 and 26 are diagrams illustrating control of an autonomous agent using a learning unit and a memory; and

FIGs. 27 and 28 are flowcharts illustrating the control process of FIGs. 25 and 26.

MODE(S1 FOR CARRYING OUT THE INVENTION

The present method must be implemented using a computer due to tlie immense number of complex computations that must be made. The computer can be a general purpose computer, with the functionality of tlie inetliod being implemented by software. Alternatively, as will be described below, part or all of the functionality of the system can be implemented on a specially designed integrated circuit chip.

Introduction

The present invention utilizes the lowest level native machine code with no immediate step of compilation or inteφretation to store, access, initialize, create, alter and execute run-time data and related code where repeated computations (embodied in the related code) must be performed using that run-time data. In tlie context of machine learning, the run-time data (the Learned Elements) and the computations that are to be performed on tl e run time data (tlie Conversion Elements) are created, initialized, stored, altered and executed directly in native machine code with no intermediate step of compilation or inteφretation. Thus, in the context of machine learning, tlie present invention stores both Learned Elements and the Conversion Elements in native machine code. All or many of the Learned Elements and all of the Conversion elements are created, initialized. stored, altered and executed directly in tlie native machine code with no intermediate step of compilation or interpretation.

The present invention is not limited to any particular machine learning paradigm such as, and by way of example only, genetic programming, genetic algorithms, simulated annealing or neural networks. Rather, it is a inetliod of creating, initializing, storing, altering and executing all or part of the Learned Elements and the Conversion Elements in any machine learning paradigm directly in native machine code.

For example, when tlie present invention is applied to evolving native machine code structure and contents, it creates, initializes, stores, alters and executes the programs structures that are to be evolved (tlie Learned Elements) directly in native machine code. The approach of the present invention is completely unlike t e compiler based and inteφreting approaches to evolving "computer programs," which only create, store, alter and (with the aid of a compiler or an inteφreter) execute high level symbolic representations of high level "program structures" in a high level programming language that, ultimately, represent and are converted into actual machine code by an inteφreter or that are executed by compiled code making repeated accesses to tlie run-time Learned Elements (tlie representations of high level "program structures") in RAM. As a further example, when tlie present invention is applied to neural networks, it creates, initializes, stores, alters and executes tlie weights of the network (the Learned Elements) and tlie code that executes a network characterized by those weights (tlie Conversion Elements) directly in native machine code. This is completely unlike the compiler based and inteφreting approaches to neural networks, which create, initialize, store, alter and (with the aid of a compiler or an inteφreter) execute high level symbolic representations of weights and network structures that, ultimately, represent and are converted by an inteφreter into tl e actual machine code that executes tlie neural network or that are executed by compiled code making repeated accesses to the run-time Learned Elements in RAM.

The present invention utilizes functions created in tlie lowest level machine code as tlie "learning entity" for machine learning or as the "computation entity" for repeated computations based on run-time data. Every such learning entity or computation entity (also referred to as an "entity" or an "individual" or a "solution") in tl e present invention is a complete, executablenativc machine code function that can be executed from a higher level language such as C or assembler by a simple standard function call ("Native Functions"). All of tlie run-time data and related code, the Learned Elements and the Conversion Elements are assembled by the present invention at πin time into Native Functions, which constitute tlie learning entity or the computation entity, and the learning entity or computation entity are then stored, altered and executed directly as Native Functions. When the run-time data or tl e Learned Elements change, the learning entities or tlie computation entities must also be changed because the previous run-time data, including the previous Learned Elements, are already included in tlie Native Function learning entity. The present invention is, in part, a method of making such changes directly in the learning and computation entities. In a sense, the present system acts as an on-the-fly compiler of both real time data and related code (for computation entities), and of Learned Elements, and Conversion Elements (for learning entities). The present system can therefore be loosely considered as a "compiling" machine learning implementation. In the specific case of evolving machine code structures and contents as the Learned Elements, the present methodology can be loosely described as a "Compiling Genetic Programming System (CGPS)".

The present system generates low level machine code directly from an example set, and there are no inteφreting or compiling steps. The invention uses the real machine instead of a virtual machine, and any loss in flexibility will be well compensated for by increased efficiency. More specifically, one or more individuals (or "entities" or "solutions") are created which represent solutions to a problem and are directly executable by a computer as Native Functions. Tlie programs are created, initialized, stored, altered and execuled by a program in a higher level language such as "C" which is not directly executable, but requires translation into executable machine code through compilation, inteφretation, translation, etc. The entities are initially created as an integer array that can be created, initialized, stored and altered by the higher level program as data. Altliough tlie present implementation of the system uses arrays of integers, t e system could be implemented as arrays of any other data type and sucli implementations are within the scope and spirit of tlie present invention.

To understand how the system chooses what integers are included in this array, one must understand what a low level machine instruction looks like. The low level instructions on all existing CPU's are groups of binary bits. On tlie Sun system for which most of tlie development of tl e present system has taken place, all instructions are 32 bits long. In low level machine code, each bit has a particular meaning to tlie CPU. In tlie present system, tlie integers chosen for the integer arrays that constitute the entities are integers that, when viewed in binary format, correspond to valid low level machine code instructions for the CPU used in the computer system to which tlie present invention is being applied. Altliough the present system is implemented on Sun and Intel systems, it could be implemented on any CPU and many other chips. Such alternative implementations are within the scope and spirit of tlie present invention.

In tlie present implementation of the invention in C. tlie entities are executed by typecasting a pointer to tlie array of integers that constitute tlie entity to be a pointer to a function. The function pointed to by the entity is then executed by calling the function using the function pointer like a regular C function. The important point here is that tlie entities are viewed, alternatively, as data and as functions at different times in tlie system. At the time tlie entities are created, initialized, stored and altered, tlie entities are viewed by the system as data-an array of integers. At tlie time the entities are executed, they are viewed as Native Functions. So if the present system was implemented in assembler, tlie entities would be created as integers (as data) and they would be executed as Native Functions. No typecast would be required in assembler but such an implementation is within tlie scope and spirit of tlie present invention.

When tlie entities are executed, they use training data (defined below) as inputs. The results of tlie execution are converted into some measure of how well the problem is solved by tlie entity, said measure being determined by the machine learning paradigm or algorithm being used. Said measure will be referred to herein as "fitness" regardless of tlie machine learning paradigm or algorithm being used. Many machine learning algorithms take the measure of fitness and feed it back into the algorithm. For example, evolutionary based learning algorithms use fitness as tlie basis for selecting entities in the population for genetic operations. On tlie other hand simulated annealing uses fitness, inter alia, to determine whether to accept or reject an entity as being tlie basis for further learning. After entities are initialized, most machine learning algorithms, go through repeated iterations of alteration of entities and execution of entities until some termination criterion is reached.

In tlie particular implementation of tlie present invention, the entities are initially created and initialized as an integer array that can be altered by the program as data and are executed by tlie program by recasting a pointer to the array as a pointer to a function type The entities are evaluated by executing them with training dala as inputs, and calculating fitnesses based on a predeteπnined criterion applied to the output of an entity after execution.

The entities are then altered based on their fitnesses using a machine learning algorithm by recasting the pointer to tlie array as a data (e.g. integer) type or by using tlie original data pointer. This process is iteratively repeated until an end criterion is reached.

The entities evolve in such a manner as to improve their fitness, and one entity is ultimately produced which represents tlie best solution to tlie problem.

Each entity includes a plurality of directly executable machine code instructions, a header, a footer, and a return instruction. The alteration process is controlled such that only valid instructions are produced. The headers, footers and return instructions are protected from alteration.

The system can be implemented on an integrated circuit chip, with tlie entities stored in high speed memory or in special on chip registers in a central processing unit and with the creation initialization, storage, alteration and execution operators stored in chip based microcode or on ROM. The present invention overcomes tlie drawbacks of the prior art by eliminating all compiling, inteφreting or other steps that are required to convert a high level programming language instruction such as a LISP S-expression into machine code prior to execution. It overcomes the need of compiler based systems to access, repeatedly, tlie Learned Elements as run-time data, which greatly slows down the system. It also represents tlie problem to tlie computer in tlie native language of the computer, as opposed to high level languages that were designed for human, not for machine, understanding. It acts directly on tlie native units of the computer, tlie CPU registers, rather than through the medium of interpreters or compilers. It is believed that by using an intermediate process such as inteφretation or compilation of high level codes, previous systems may actually interfere with the ability of computers to evolve good solutions to problems.

Experiments have demonstrated that the present approach can speed up the execution of a machine learning system by 1,000 times or more as compared to systems which provide potential solutions in the form of high symbolic level program expressions and by 60 times or more as compared to compiler based systems. This speed advantage makes possible practical solutions to problems which could not heretofore be solved due to excessive computation time. For example, a solution to a difficult problem can be produced by the present system in hours, whereas a comparable solution might take years using conventional techniques. These and other features and advantages of tlie present invention will be apparent to those skilled in tlie art from tlie following detailed description, taken together with the accompanying drawings, in which like reference numerals refer to like parts. Detailed Implementation

FIG. 1 illustrates a computer system 10 for implementing a machine learning method according to the present invention. The system 10 includes a Random Access Memory (RAM) 12 in which tlie software program for implementing tlie functionality of the invention is stored, and a processor 14 for executing the program code. The system 10 furtlicr includes a visual display unit or monitor 16 for providing a visual display of relevant information, a read-only memory (ROM) 18 for storing firmware, and an input-output (I/O) unit 10 for co iection to a printer, modem, etc.

The system 10 further includes a mass data storage 20 which can be any combination of suitable elements such as a fixed (hard) magnetic disk drive, a removable (floppy) disk drive, an optical (CD-ROM) drive, etc. Especially large programs whicli implement the invention may be stored in the storage 20, and blocks of the programs loaded into tlie RAM 12 for execution as required.

User access to tlie system 10 is provided using an input device 22 such as alphanumeric keyboard 1 14 and/or a pointing device such as a mouse (not shown). The elements of the system 10 are interconnected by a bus system 24. The system 10 is preferably based on die SUN SPARC architecture due to its register structure. Tlie invention is also preferably implemented using tlie "C" programming language due to its freedom in the use of data types and tlie ability to cast between different types, especially pointers to arrays and pointers to functions and a standard SUN operating system compiler, altliough tlie invention is not limited to any particular configuration. The invention has been practiced using die above configuration in the SPARC environment whicli generally a stable architecture for low level manipulation and program patching. The particular platform used was tlie SUN SPARCSTATION 1 +.

The RAM 12 is illustrated in FIG. 2 as storing an operating system 30 such as UNIX or one of its variants, and a program 32 written in tlie "C" language which provides the functionality of the invention The program 32 includes high level machine language instructions 32a, and one or more machine code array 32b.

The high level instructions 32a are not directly executableby the processor 14, but must be converted into directly executable machine code by a compiler. The array 32b, however, is provided in native (lowest level) machine code including binary instnictions that are directly recognized and executed by the processor 14.

The performance of tlie present invention is greatly enhanced by implementation thereof using a processor having banks of registers. This architecture is provided by tlie SPARC system as illustrated in FIG. 3.

The components of tlie SPARC processor 14 which are most relevant to understanding the present invention include an Arithmetic Logic Unit (ALU) 40 for performing actual numeric and logical computations, a program counter 42 for storing an address of a next instruction to be executed, and a control logic 44 which controls and coordinates the operations of the processor 14 including memory access operations. The processor 14 further includes banks of registers which are collectively designated as 46. The processor 14 also includes a number of additional registers which are not necessary for understanding tl e principles of the present invention, and will not be described. Although tlie invention is presently implemented using integer arithmetic, it may be applied to Floating Point arithmetic easily and sucli an application is within the scope and spirit of the invention.

More specifically, tlie registers 46 include eight register banks (only one is shown in tl e drawing), each of which has 8 output registers O₀ to O,, 8 input registers 1₀ to I₇, 8 local registers L₀ to L₇. There are, also, 8 global registers G₀ to G₇ accessible from any bank. The implementation of the invention using this architecture will be described below.

FIG. 4 is a flowchart illustrating the general machine learning method of the present invention which is implemented by tlie high level instructions 32a in tlie C programming language. Figures 22a to 22k are more detailed flowcharts illustrating the present invention The first step is to define a problem to be solved, a fitness criterion for evaluating tlie quality of individual solutions, and an end criterion. The end criterion can be a predetermined number of iterations for performing the process (e.g. 1 ,000), achievemenlof apredeteπnined fitness level by one or more solutions, a change in total or individual fitness which is smaller than a predcteπnined value, or any other suitable criterion.

Next, the input, output, calculation, and stale registers must be identified. An input register is a register that is initialized before each calculation with data. In the present implementation of tlie invention, tlie input registers may be any one of the registers referred to as 10 through 15 in tlie Sun architecture. See Figure 3, number 46. In tlie present implementation of tlie invention, it is, therefore possible to have up to six input registers. However, it would be easy to extend die system to have many more input registers and such an extension is within tl e scope and spirit of the present invention. In any event, the designer must pick how many of tlie eight input registers to use.

There are many ways to initialize these registers but the preferred implementation is to include the input variables as parameters when the individual is called as a Native Function. For example, if native function array is an array of integers that constitute valid machine code instructions that amount to a Native Function and that amount to an entity in this invention, then the following type definition and function call execute tlie native function array individual and pass two input variables to it.

Typedef unsigned int(* function-ptr) ();

{the type definition} a=((function_ptr) native function_array)(2,3);

{calls the function and passes two inputs (2 and 3), which are placed in 10 and 11 }

The items in curly brackets are descriptive and not part of the code. In tlie present implementation of this invention, the two input variables are put into 10 and 11. Accordingly, if the designer decides to have two input registers, they must be 10 and 11 in the present implementation. Other methods of initializing registers such as putting instructions in the header of the entity (see Figure 5, number 50) that retrieve data from RAM provide more flexibility as to what registers are input registers and more input registers and they are within the scope and spirit of tlie present invention. This process of picking and initializing input registers is referred to in Figures 16b, c, and d; 17b. c. and d; 18b, d, and f; and 22b, e. and g.

An output register is read after the individual has finished executing on a given set of inputs. In other words, an output register is the output of tlie individual on a given set of inputs. Tlie designer must designate one 5 or more registers to be tlie output register(s) and tlie number depends on the problem to be solved. In the present implementation, the value in 10 at die end of tlie execution of tlie individual is automatically set as tlie single output register and as tlie output of tlie individual. The above code automatically puts tl e value in 10 into tlie variable "a. " It is possible to designate multiple output registers and to preserve tlie values therein when an individual has been executed. One such method would be to include instructions in the footer of tlie individual

10 (see Figure 5, number 50) that move tlie value(s) in tlie designated output registers in tlie individual that has just executed to storage registers or to locations in RAM memory. Such implementations are within the scope and spirit of tlie present invention.

This process of picking and recovering tlie values in output registers is referred to in Figures 16b. c, and d; 17b, c, and d: 18b, d, and f: and 22b, e, and g.

1 A calculation register is a register that is neither an input nor an output register but that is initialized to a given value every time tlie individual is executed. Zero or one is the preferred initialization value. A calculation register may be initialized in die headers of the individuals. (See Figure 5, number 50). In any event tlie designer must decide how many, if any, calculation registers he or she wishes to use and then designate specific registers to be calculation registers. So, for example, if there were two input registers, 10 and 11 and if

20 10 were tlie output register, then 12 tlirough 15, and L0 through L7, among others would be available slots for a calculation register. So if tlie designer desires one calculation register, he could pick 12 as the calculation register. This process of picking and initializing calculation registers is referred to in Figures 16b, c, and d; 17b, c, and d; 18b, d, and f; and 22b, e and g.

A state register is a register that is neither an input nor an output register but that is not initialized

25 every lime tlie individual is executed. That is, values from some previous execution of tlie individual are retained in a state register. The precise retention policy is a matter of decision for tlie designer. At tlie beginning of the execution of the individual, the state register has to be initialized to the value it held at tlie end of a previous execution of tlie individual. If that value was saved in a storage register or in RAM, then the header of tl e individual (See Figure 5, number 50) must have an instruction moving the value from tlie storage register or RAM

30 to the state register or tlie previous state could be passed to the individual as a parameter when tlie individual is called as a Native Function. In this case, the footer of the individual may have an instruction saving tlie value of the stale register at the end of execution to tl e storage register or to RAM. There are many available ways to implement state registers and to save and initialize their values and all of them arc within tlie scope and spirit of this invention. In any event die designer must decide how many, if any, state registers he or she wishes to use and

35 then designate specific registers to be state registers. So, for example, if there were two input registers, 10 and 11. and one calculation register, 12, and if 10 were the output register, dien 13 through 15, and L0 through L7, among others would be available for a state register. So if the designer desires one state register, he or she could pick 13 as the state register.

This process of picking and initializing state registers is referred to in Figures 16b, c, and d; 17b. c, and d; 18b, d, and f; and 22b, e and g.

Once all registers have been designated, die designer has established the "Register Set." In die example above, tlie Register Set would be as follows

input 10, I I output 10 calculation 12 state 13

Once die Register Set has been picked, it is essential that all individuals be initialized widi instructions that contain only references to valid members of die register set. See Figures 16c, 17c. 18d, and 22e. It is also essential diat all changes to tlie individuals, including those made widi genetic operators, modify die individual in such a way as to make sure that all register references in the modified instructions are to valid members of the Register Set. Next, a solution or population of solutions is defined as an array of machine code entities. These entities are preferably "C" language functions which each include at least one directly executable (machine language) instruction. An instruction is directly executableby being constituted by die lowest level (native) binary numbers that are directly recognized by die control logic 44 and ALU 40 as valid instructions. These instructions do not require any compilation, inteφretation, etc. to be executable. The invention will be described further using die exemplary case in which die entities are "C" language functions. However, it will be understood that the invention is not so limited, and that die entities can be constituted by any data structures including machine code instructions that can alternatively be manipulated as data and executed by a higher level program.

After die function or array of functions has been defined, they are initialized by inserting valid machine code instructions in die appropriate locations dierein. As illustrated in FIG. 5, a function 50 includes a header 50a, an instruction body 50b. a footer 50c, and a return instruction 50d.

The header 50a deals widi administration which is required when entering a function. This normally involves manipulation of the registers 46, including passing arguments for the function to and from the various registers 46, saving tl e values in die registers that exist when die function is called and saving die address of the calling instruction so that, when die function is fmished, program execution can begin from where it left off. The header can also be used to perform many other useful functions at the option of die designer including initializing registers and recovering die values of saved state registers. There can also be processing to ensure consistency of processor registers.

The header 50a is often constant and can be added at d e beginning of the initialization of die individual machine code functions in die population. Mutation, crossover and any ouier operator d at alters die entity must be prevented from changing this header field when they are applied to an individual function or tlie field must be repaired after the application of the operator.

The footer 50c is similar to the header 50a, but does the operations in die opposite order and "cleans up" after the function call by, among other things, restoring the registers to their state before tlie function call. recovering tlie output of the function, saving state registers, and many other useful functions at the option of the designer. The footer field must also be protected from change by the genetic or other alteration operators or tlie field must be repaired after tl e application of the operator.

The return instruction 50d forces the system to leave the function and return program control to the calling procedure via tlie program counter 42. If variable length programs arc desired, tlie return instruction can be allowed to move within a range defined by a minimum and maximum program size. The return instruction must also be protected from change by tlie genetic or other alteration operators or tlie field must be repaired after the application of the operator.

The function body 50b includes at least one directly executable machine code instruction, for example instructions 52a, 52b and 52c. In tlie SPARC architecture, instructions have a fixed length of 32 bits. Instructions can be of two types: a first type which is designated as 54 and consists of a single 32 bit operator; and a second type which is designated as 56 and includes a 19 bit operator and a 13 bit operand. The operand represents data in tlie foπn of a numerical variable.

During tl e initialization step, tlie instruction bodies 50b of the functions 50 are filled with valid machine code instructions in a suitable manner. In tlie preferred implementation, the instructions are initialized from a Default Instruction Template Set, which contains partiallv "blank" instructions that contain information on what types of instruction is represented by a particular template. For example, is it an instruction that adds the values in two registers together or an instruction that adds a constant to a register value? What is left blank in tlie template is tlie registers) to which tlie instruction applies and the constant values to be included in tlie instruction. The register references and tlie constant values may be added by methods such as those described in Figure 18d. In Figure 18d, instruction templates are randomly chosen and then tlie blanks are filled out so that tlie integer array constituting tlie individual contains only valid machine code instructions, taking care to limit tlie register references therein members of tlie Register Set. See Figures 17c, 18b and 18d. The instructions can be selected randomly, or using any suitable selection criterion.

In the most basic form of the invention, to limit tlie search space and to avoid complex control mechanisms and thus achieve maximum perfoπnancc, tlie set of machine code instructions is limited in a number of ways. Only two registers and only those machine code instructions of two addressing mode types are used.

All of them operate internally in the processor. However, more lengthy and complex implementations of the

Register Set are described above and more complex implementations of the instruction set are within tlie spirit and scope of the invention. The first addressing mode takes one argument from memory immediately afterwards and performs an operation on this argument and a processor register and leaves the result in a register. The other takes tl e operand from the second register, performs tlie operation and leaves the result in the first register. With tliese kinds of instructions it is possible to reach the maximum processor throughput Tiic instructions arc also constrained to those which take an integer as argument and return an integer Tins however docs not unt the problem space lo mathematical problems or |ust to integer aπthmetic No control instructions such .is lump instructions arc allow ed which means that no loops can be formed These limitations reduce the complcMtN and thus execution time of the individual programs

The basic instructions arc listed in tlic following TABLE

ADD (mini) ADD (reg2) Add register one to an immediate operand or add register one to register two

SUB (imin). SUB (reg2) Subtract register one from an immediate operand or subtract register one from register two

MUL (imin) MUL (reg2) Multipi . register one b\ an immediate operand or niultiph register one b\ register two

DIV (mini) DIV (reg2) Divide register one b\ an immediate operand or divide register one b\ register two

OR (mini) OR (reg2) Logical OR of register one and an immediate operand or logical OR of register one and register 2

AND (imin). AND (reg2) Logical AND of register one and an immediate operand or logical AND of register one and register two

XOR (imin). XOR (reg2) Logical EXCLUSIVE OR of register one and an immediate operand or logical EXCLUSIVE OR of register one and register two

SETHI Set the high bits of a register, used when an operand bigger than 13 bits hits needs to be loaded into a register

SRL Logical shift right of register one to an immediate operand or logical shift right of register one a number of steps defined bv register two

SLL Logical shift left of register one to an immediate operand or logical shift left of register one and register two

XNOR Logical EXCLUSIVE NOR of register one to an immediate operand or logical EXCLUSIVE NOR of register one and register two These functions can and are used to implement other common processor functions. Loading of operands into registers is perfoπned by die OR instruction, clearing a register is performed by tlic AND instruction, and the negation instruction can be performed by tlie XNOR instruction etc.

An exemplary array 60 of functions F₀ to F₆ is illustrated in FIG 7. Each function includes a header 50a including instructions which are collectively designated as H. a body 50b including instructions B, a footer

50c including one or more instructions F, and a return instruction 50d which is indicated as R. Although tlie figure shows a length of three instructions for tlie header and one for the footer, those particular numbers are for illustrative purposes only.

The functions of tlie array 60 arc illustrated as having a fixed length. However, tlie invention is not so limited, and tlie functions can have variable lengths.

FIG. 8 illustrates an alternative embodiment of tlie invention in which only a single solution in the foπn of a function 62 is provided. The function 62 is configured as a continuous array that contains valid machine code instructions, and is typically larger than the functions of tlie array 60. Such a single array would be a typical way to implement a system not involving machine learning but involving repeated computations on run time data. Certain machine learning approaches would use such a single structure also.

After Ute array is initialized, it is recast as a function type array, and die registers 46 are initialized wiϋi training data. The training data is normally part of a "training set", each element of which consists of one or more inputs and a desired output. The training set represents the problem diat is to be solved. However, ϋie invention is not so limited, and ϋie training data can alternatively include testing, validation, prediction, or any odier data suitable for machine learning or for repetitive calculations on run-time data.

For ϋ e machine learning applications, the purpose of the learning process is to train ϋic functions using ϋie training data by causing ϋie functions to evolve until one of them produces outputs in ϋie output register(s) in response to ϋie training inputs in ϋie input registers) that closely approximate ϋ e desired outputs. The inputs are passed to ϋie functions by initializing ϋie registers wiϋi ϋie training data as described elsewhere. The functions of ϋie array (or single function as illustrated in FIG. 8) are then executed widi Uie training data as inputs. This execution causes ϋie functions to produce outputs which are calculatedin accordance wiϋi ϋie instructions in ϋie functions.

The output register or registers are then read, stored and compared with actual or desired outputs from Uie training data to calculate fitnesses of the functions. A function has a high fitness if its output closely approximates ϋ e desired output, and vice-versa. If ϋie end criterion is met, the program ends.

If ϋie end criterion is not met, die array is recast or used as a data array (e.g. integer), and the instruction bodies 50b are altered. During this process, selected individual instructions are altered, replaced, added or deleted such diat ϋie altered function includes at least one valid machine code instruction. In other words, regardless of Uie manner in which a particular function is altered, the program ensures that the altered function will not include any invalid instructions.

The function(s) can be altered using any machine learning algoriϋim. One preferred mediodology is evolution of machine code structures and machine code contents using crossover and mutation operators as will be descπbed in detail below However, Uie invention is not limited to any particular learning paradigm, nor is it limited to machine learning applications The invention can be used, for example, to alter run-time data encoded in Uie machine code for Uie purpose of performing repetitive calculations on Uie run-time data

As descπbed above, it is important Uiat Uie header, footer and return instruction will not be altered ^•5 This can be accomplished by protecting Uiese elements from alteration using masking or oUier programming tecliniques Alternatively, Uiese elements can be allowed to be altered, and their initial states subsequently restored by repair or replacement As another alternative, Uie instructions can be separated from Uie function altered and then relumed to Uie function There are oUier ways to protect Uie header, footer and return instructions from being altered all of which are wiUun Uie scope and spiπt of this invention 0 After performing the alteration. Uie program loops back to Uie step of recasting Uie arrav as a function and executing Uie function The program lteratively repeats Uiese steps until Uie end cπtenon is met

The output of Uie system when used for machine learning, is Uie function that has Uie highest fitness and Uiereby produces outputs Uiat are closest to the desired outputs in response to Uie training inputs The Junctions evolve as Ute result of Uie alterations wiUi one individual finally emerging as being improved to Uie "S highest fitness The policy for selecting a "best ιndιvιdual(s)" is a matter of discretion by Uie designer This system applies to any such policy

As descπbed above wiUi reference to FIG 3, Uie registers 46 include banks of input registers and output registers Tliese registers are used to pass data from a calling function to a called function, and are used to initialize Uie functions with training data as further desenbed above 0 More specifically functions use their input registers to store vanables When a function calls another function, Uie Spare Architecture transfers Uie contents of its input registers to its output registers The called function transfers the contents of Uie calling function's output registers into its own input registers, and operates on Uie data using its input registers The opposite operation takes place when control is returned from the called function to the calling function ^*5 Thus, ϋie C program initializes a function wiUi training data by stoπng ϋie training data in its output registers and ϋien calling the function This mcϋiod of initialization is preferred in ϋie SPARC implementation, but ϋie invention is not so limited Oϋier mechanisms for initializing functions wiϋi training data may be preferable wiϋi oϋier computer architectures or for more complex application on ϋie SPARC architecture

An important feature of the present invention is to create and alter machine code functions as data, 0 and execute ϋie functions, from a high level language such as C The following illustrates how this manipulation can be accomplished using the SPARC architecture

As a simplified example, define a simple function Uiat computes ϋie sum of two input parameters and returns ϋie result from this addition sum(a,b) = a+b 5 Tins function can be translated by a compiler in ϋie SPARC architecture to ϋie following five assembly language instructions sav e add \%ι ,\%ιl ,\%ι restore ret nop

These five assembly language instructions are stored sequentially in memory as five 32-bit numbers having Uie following decimal equivalents 2178940928 2953183257 2179465216 2177359880 16777216

The "save" instruction corresponds to Uie number 2178940928, and is a special machine code instruction in Uie SPARC architecture that saves ϋie contents of Uie calling function's registers and transfers Uie parameters to ϋic called function

The "add" instruction adds ϋie contents of input register "lO" to the contents of input register "i l ", and stores ϋie result in input register "ι0" Tins insuTiction is coded as Uie integer 2953183257 Additions between oϋicr registers are represented by oϋier integers

Tlie "restore" instruction restores ϋie registers from Uie calling function and transfers ϋie result lrom the addition (present in register lO) for access from Uie calling function

The return instruction "ret" jumps back to the memory location from where the function was called

The "nop" instruction is a no operation lnsunction that does noUnng but "entertain" the processor while lumping back This instruction could be left out if Uie order of ϋie "restore" and "ret" instructions was reversed as will be descπbed below Implementation of ϋie present invention involves calling a machine code function as exemplified above from ϋie C or oϋicr high level language Adding Uie numbers 2 and 3 by calling Uie function sum(2.3) can be represented in assembly language as follows

mov 2,\%ol mov 3,\%oO call sum nop These instructions are represented by Uiese four integers: 2450530306 2484084739 2147483646 16777216

The first instruction stores Uie input parameter "2" in ϋie output register %o 1. The second instruction stores ϋie input parameter "3" in ϋic output register %o().

The call instruction jumps to Uie location of ϋie sum function and stores ϋie address of itself in a register (output register %o7 by convention). This causes ϋie program counter 42 to contain Uie address or location of ϋie first instruction of Uic sum function. In oUier words, it contains the address of a memory location which stores Uie integer 2178940928 (save), and execution continues from here until Uie return instruction is encountered. After ϋie return from the function, ϋie output register %o0 will contain the result from Uie summation which can be used in further calculations.

When a save instruction is executed, it copies ϋie contents of all output registers to an unused set of input registers. In this way ϋie sets of input and output registers are used as a small internal stack for transfer of parameters and saving of register contents. The calling function stores the parameters of Uie called function in Uie output registers, Uie save instniction copies them to the input registers of Uie called function, and Uie restore instruction later copies all input registers of the called function into ϋie output registers of ϋie calling function. If ϋie number of parameters is larger than ϋie number of registers, ϋie memory in the stack has to be used, as in most oϋier processors.

In ϋiis manner, the input data (training data in the machine learning system and run-time data for computational entities) is passed to ϋic array of functions (solutions) by storing ϋie data in ϋie output registers of ϋie calling function, and ϋien calling Uie array.

The "nop" does noϋiing but keep ϋie processor occupied while jumping to the function. In a more effcctivebut less clear version, ϋie "mov 3,%o0" instruction is placed after the "call", which makes the processor execute this instruction "while" jumping to Uic function.

Tlie following illusϋates how to call Uic sum function defined above from Uie C-language. typedef unsigned int(* function_ptr) () unsigned int sumarray[]={ 2178940928, 2953183257, 2179465216, 2177359880, 16777216}; a=((function_pu-) sumarray)(2,3);

These Uiree lines of C-code will compute ϋie sum of 2 and 3 and put the result in a variable "a" by calling a function defined by ϋie integers in ϋie "sumarray".

The first line of code defines ϋie function pointer type in C, because Uiis type is not predefined in ϋie C-language. The second line of code declares an integer array containing integers for the instructions in ϋie sum function as defined above.

The last line of code converts Uie address of Uie sumarray from a pointer to an integer array to a pointer to a function, and ϋien calls Uiis function wiϋi ϋie arguments 2 and 3. Tlie result from Uie function call is placed in variable "a".

This example illustrates Uie basic principles of manipulating binary programs. The sumarray can be allered like any oUier array as long as its contents are valid functions that can be called (executed) to evaluate Uie results of ϋie algoriϋim.

The present meUiod as implemented in Uie C programming language utilizes four instructions for initializing, altering, and executing machine code functions.

1. A first insunction Uiat points to and designates machine code stored in a memory as data.

2. A second instruction Uiat points to and designates machine code stored in Uic memory as at least one directly executable function.

3. A third instruction Uiat alters machine code pointed to by ϋie first instruction.

4. A fourth instruction Uiat executes machine code pointed to by Uie second instruction. Examples of ϋie four instructions are presented below.

1. unsigned int * Uieintegerpointer; {declaration of integer pointer variable}

Uieintegerpointer = (unsigned int *) malloc(Max_Individual_Size * Instruction Size) {Tlie instruction creating a pointer to an integer array}

2. Alternative 1

typedef unsigned int(*function_ptr) ();

{definition of Uie function_ptr type as a pointer to a function} function_ptr Uiefunctionpointer

{declares Uiefunctionpointer as a variable of type function_ptr) thefunctionpointer=(function_ptr) theintegerpointer; {Instruction Uiat typecasts Uieintegerpointer as a pointer to a function}

Alternative 2

typedef unsigned int(*function_ptr) ();

{definition of Uie function_ptr type as a pointer to a function} Predicted_Output = ((functionjjtr) theintegerpointer)( Input Data Here);

{the code "((function_ptr) Uieintegerpointer)" is an expression within an instruction ϋiat performs a typecast. }

3. theintegerpointer[2] = Uieintegerpointer! 2] ] 1234;

{XOR's ϋie value in ϋie second insunction wiϋi Uie integer 1234} or theintegerpointer[3] = 16777216;

{places an nop instruction in Uie third insunction slot, overwriting the previous instruction} 4 var a = thefunctιonpoιnter(Input Data Here), or

Predιcted_Output= ((functιon_ptr) tlιeιntege oιnter)(Input Data Here), In the CGPS embodiment of tlie invention which utilizes genetic crossover and mutation operations, 5 the tvpes of operations that can be perfoπned depend on the type of instruction, more specifically if tlie instruction includes only an operator, or if it includes an operator and an operand Examples of genetic crossover and mutation operations are illustrated in FIGs 10 to 15

FIGs 10a and 10b illustrates a uniform crossover operation in which like numbers of adiacent complete instructions are exchanged or swapped between two functions FIG 10a illustrates two functions 70 and 10 72 Unifoπn crossover is performed b\ exchanging, for example, two instructions indicated as "4" and "2" in tlie lunction 70 for two instructions indicated as "5" and "9" in the function 72

The results are illustrated in FIG 10b An altered function 70' includes all of its original instructions except lor tlie "4" and tlie "2" which are replaced by tlie "5" and tlie "9" from the function 72 The function 72 includes all of its oπg al instructions except for the "5" and the "9" which are replaced by the "4" and tlie "2" 15 from Uic function 70

FIGs 1 la and 1 lb illustrate "2-poιnt crossover" in which blocks of different numbers of complete instructions are exchanged between two functions In this case, two points are selected in each function, and all of tlie instructions between tlie two points in one function are exchanged for all of the instructions between the points m tlie other function 0 In the example of FIG 1 la, instructions indicated as "7" and "8" in a function 74 are exchanged for instructions "4", "7", "6", and "1" in a function 76 to produce functions 74' and 76' as illustrated in FIG l ib FIGs 12a and 12b illustrate how components of functions can be crossed over In FIG 12a, two instructions 78 and 80 have operators 78a and 80a and operands 78b and 80b respectively In this case, the operator 78a of tlie function 78 which is indicated as "OP1 " is exchanged for the operator 80a of the function 25 which is indicated as OP2 The result of the crossover is illustrated in FIG 12b

FIGs 13a and 13b illustrate an example of how uniform crossover can be performed between all or parts of operands In FIG 13a, a function 82 has an operator 82a and an operand 82b, whereas a function 84 has an operator 84a and an operand 84b The πghtmost two bits of the operand 82b are exchanged for tlie nghtmost two bits of tlie operand 84b to produce functions 82' and 84' with operands 82' and 84' as illustrated m FIG 13b 30 FIG 14 illustrates how the operator of a function 86 can be mutated In this case the function 86 initially has an operator which is indicated as OP1, and is altered or replaced so thai a mutated function 86' has an operator OP2 It is necessary that both operators OP1 and OP2 be valid machine code instructions in tlie set of instructions used by tlie system

FIG 15 illustrates how all or part of an operand can be mutated A function 88 has an operator 88a 3 and an operand 88b In this example tlie second least significant bit of Uie operand 88 is changed from "1 " to "0" or "flipped" to produce an altered function 88' having an altered operand 88b'

As set forth above, tlie present invention can be applied to any applicable problem by using any suitable machine learning algorithm. The principles described in detail above can be applied to implement a particular application and computing environment.

In addition to implementing the present machine learning system on a general purpose computer as described above, il is possible to implement part or all of tlie system as a specially designed integrated circuit chip 90 such as an Application Specific Integrated Circuit (ASIC) which is symbolically illustrated in FIG. 9.

The chip 90 comprises a Central Processing Unit (CPU) 92 including a processor 94 and a RAM 96. The processor 94 includes normal CPU microcode plus microcode implementing storage, initialization, creation, and alteration operators. The RAM 96 is preferably a high speed cache memory which tlie processor 94 can access at a much higher speed than tlie processor 94 can access off-chip memory. The RAM 96 can include a number of registers, or can have a conventional address based architecture.

Preferably, tlie population of functions (solutions) is stored in the RAM 96 for rapid manipulation and execution by the processor 94. Other alternatives include additionally storing tlie high level program in the RAM 96. As yet another alternative, Uie chip 90 can include a ROM 98 for storing, for example, a kernel of the high level program. The on chip memory, alternatively could be registers dedicated to storing individuals or high speed, on chip RAM that is not a cache. In addition, the CPU could alternatively execute machine learning operators such as crossover and mutation or any otlicr operators that initialize, create, evaluate or alter individuals in microcode or in high speed ROM.

A preferred implementation of the invention evolves machine code structures and machine code contents as a way of learning tlie solution to a problem. A detailed flowchart of this system is presented in FIG. 6 as an example for reference purposes.

The program utilizes a small tournament in combination with genetic crossover and mutation operations, and includes the following basic steps.

1. Randomly pick four individuals from the population. 2. Evaluate them in pairs, two at a time according to their fitness.

3. Let the two winners breed.

4. Replace tlie losers with the children of tlie two winners.

5. Repeat step 1-4 until the success predicate is true or the maximum number of tries is reached. The flowchart is believed to be self-explanatory except for probabilistically choosing one operation or another. Tliese choices are analogous to flipping a coin, and can be implemented using, for example, a random number generator which generates random numbers between 0 and 1. If the number is 0.5 or less, a first probibilistic branch Pa is taken. If tlie number is higher than 0.5, a second probibilistic branch Pb is taken. Choices having more than two options can be implemented by dividing the range between 0 and 1 into a number of subranges equal to the number of choices. FIGs. 16a to 16d are further provided for reference, and in combination constitute a detailed flowchart of the generic machine learning system of the present invention. The entries in tlie flowchart are believed to be self-explanatory, and will not be described in detail. FIG. 16a outlines tlie general operation of tlie system. FIG. 16b sets forth the details of die block entitled "SYSTEM DEFINITION" in FIG. 16a. FIG. 16c sets forth Uie details of Uie block entitled "INITIALIZATION" in FIG. 16a, whereas FIG. 16d sets forth the details of the block entitled "LEARN FOR ONE ITERATION in FIG. 16a. Figures 17a through 17d are further provided for reference, and in combination constitute a detailed flowchart of U e application of Uic invention to any computation problem Uiat involves repeated access to run-time data. The entries in the flowchart use die terminology of this application and are believed to be self explanatory in the context of Uiis application, and will not be described in detail.

Figures 18a Uirough 18g are further provided for reference, and in combination constitute a detailed flowchart of Uie application of Uie invention as a system Uiat induces die solution to problems by learning Uie structure and content of machine code entities. The entries in the flowchart use the terminology of this application and are believed to be self explanatory in U e context of Uiis application, and will not be described in detail. The mutation and crossover operators referred to in said Figure 18g arc Uie operators described elsewhere in Uiis application. FIG. 16a outlines U e general operation of the system when it is applied to a generic machine learning problem.

FIG. 16b sets forth Uie details of Uie block entitled "SYSTEM DEFINITION" in FIG. 16a. The steps in Uiis figure show, inter alia, what steps to take to analyze any machine learning problem to permit Uie designer to encode entities that contain boUi Uie Learned Elements and Uie Conversion Elements into a machine code entity Uiat can be created, initialized, stored, modified and executed by the means described in Uiis application. Numeric or oUier values may be encoded in Uie machine code as constants.

FIG. 16c sets forth Uie details of Ute block entitled "INITIALIZATION" in FIG. 16a. This Figure sets forth, inter alia, a set of steps Unit will result in Uie creation of one or more learning entity or entities that will be Uie learning entity or entities in the machine learning system Uiat is being implemented using die present invention. Such entity or entities will be created, stored, initialized, modified and executed by the methods set forth herein but when and how such steps take place will be set according to Uie requirements of Uie particular machine learning algorithm being implemented, as shown in FIG. 16d. It is important to note Uiat Uie entity created according to Uie procedure outlined in Figure 16c ordinarily includes not only Uie Conversion Elements but also contains the first set of Learning Elements for evaluation. Should said first set not be available when die initialization occurs, then Uie entity should be initialized with dummy values in U e places in the instructions where the real Learning Elements will reside. Then, when Uie first set of real Learning Elements are known, it can be placed into Uiose places using Uie procedure under "Start Modification" in FIG. 16d.

FIG. 16d sets forth Uie details of the block entiUed "LEARN FOR ONE ITERATION" in FIG. 16a. This figure shows, inter alia, how to modify and how to evaluate an entity when Uie particular machine learning algorithm being implemented calls for either of those steps. The particular implementation of an application of Uie invention to a particular machine learning problem will vary substantially from problem to problem and among various machine learning systems. So Uiis Figure is general in terms of when to evaluate Uie entity (referred to in FIGs. 16a-16d as a "solution") and when to modify Uie enUty. Because of Uie breadth of various approaches to machine learning, this Figure indicates Uiat steps other than the ones shown specifically are appropriate and Uiat systems including such other steps are wiUiin Uie spirit and scope of the present invention.

FIG. 17a outlines the general operation of the invention when Uie invention is to be used to perform 5 repeated computations on run-time data. This application of die invention could be handled in oUier and alternative manners both in Uie general matters set forth in FIG. 17a and in 17b Uirough 17d and those oUier and alternative manners are wiUiin the scope and spirit of Uiis invention.

FIG. 17b sets forth Uie details of the block entitled "SYSTEM DEFINITION" in FIG. 17a. This

Figure shows, inter alia, how to define a repeated computation on run-time data problem so Uiat it may be

10 implemented using Uic invention. Numeric or other values may be encoded in Uie machine code as constants.

FIG. 17c sets forth the details of Uie block entitled "INITIALIZATION" in FIG. 17a. This Figure sets forth, inter alia, a set of steps Uiat will result in Uie creation of a computational entity Uiat will perform repeated computations on run-time data based on Uie system definition perfoπned in FIG. 17b. It is important lo note Uiat Uie computation entity created ordinarily includes not only Uie related code but also contains Uic first

15 set of run-time data on which the system will perform repeated calculations. Should said first set not be available when Uie initialization occurs, then Uie computational entity should be initialized wiUi dummy values in Uie places in the instructions where Uie real run-time data will reside. Then, when Uie real run-time data is known, it can be placed into Uiose places using Uie procedure under "Start Modification" in FIG. 17d.

FIG. 17d sets forth Uie details of the block entitled "EVALUATE OR MODIFY FUNCTION" in FIG. 20 17a. The particular implementation of an application of the invention to repeated computations on run-time data will vary substantially from problem to problem. So this Figure is general in terms of when to execute Uie computational entity (referred to in FIGs. 17a-17d as a "solution") and when to modify the computational entity. The computational entity should be modified each time Uiat the designer wishes to perform a particular computation (related code) on run-time data of the type that is encoded in Uie computational entity but Uiat has 25 not yet been placed in Uie computational entity. The computational entity should be executed every time Uie designer wants to perform a calculation on run-time data that has been encoded into Uie machine code of Uie computational entity.

FIG. 18a outlines Uie general operation of the invention when it is applied to learning machine code structures and machine code contents Uiat will operate a register machine for Uie purpose of learning a solution

30 to a problem. This application of Uie invention could be handled in other and alternative manners boUi in Uie general matters set forth in FIG. 18a and in 18b through 18h and those other and alternative manners are within

Uie scope and spirit of Uiis invention.

FIG. 18b sets forth the details of Uie block entitled "SETUP" in FIG. 18a. This Figure describes various steps necessary to setup an application of the invention to a CGPS run on a particular problem. Numeric 5 or other values may be encoded in Uie machine code as constants.

FIG. 18c sets forth the details of Uie block entitled "INITIALIZATION" in FIG. 18a. This Figure describes, inter alia, a method for initializing a population of entities for the purpose of conducting a CGPS run. FIG. 18d sets forth Uie details of the block entitled "CREATE INSTRUCTION" in FIG. 18c. This Figure describes, inter alia, one meUiod of creating a single machine code instruction to be included in Uie body of an entity.

FIG. 18e sets forth Uie details of the block entitled "MAIN CGPS LOOP" in FIG.18a. This Figure sets forth, inter alia, one approach to implementing Uie basic CGPS learning algorithm.

FIG. 18f sets forth Uie details of the block entitled "CALCULATE INDIV|N] FITNESS**" in FIG.18c. The fitness function described therein is simple. More complex fitness functions may easily be implemented and are wiUiin Uie spirit and scope of Uiis invention.

FIG. 18g sets forth Uie details of Uie block entitled "PERFORM GENETIC OPERATIONS . . ." in FIG. 18c. The mutation and crossover operators referred to in Figure 18g may be all or any of Uie operators described elsewhere in Uiis application.

FIG. 18h sets forth Uie details of Uie block entitled "DECOMPILE CHOSEN SOLUTION" in FIG. 18a. This Figure describes a meUiod of converting an individual into a standard C language function Uiat may Uien be linked and compiled into any C application. Other and more complex meUiods may be easily implemented and are widiin die spirit and scope of Uiis invention.

TURING COMPLETE MACHINE LEARNING SYSTEM A basic machine learning system and method according to U e present invention is described above. The system can be made Turing complete though the addition of branching instructions as will be described in detail below. This capability increases d e flexibility and power of the system by enabling subroutines, leaf functions, external function calls, recursion, loops, and other types of conditional operations. Introduction

The modern computer has grown in instruction capabilities and machine word size, which means Uiat Uiere are more tasks that can be carried out elegantly by a straightforward machine language subroutines. For example, die precision of U e ariUimetic is enough for direct usage, and die memory organization is often flat which makes pointer operations less complex, etc.

Conventional application programs often fail to use the computer to its full potential because Uie algoriUims are not implemented in Uie most efficient way. This fact is even more relevant in Uie use of algorithms for machine learning, artificial intelligence, artificial life, etc., repeated computations on run-time data which rely on complex manipulations and are often of the meta type where programs manipulate program structures. The present machine learning system is a unique approach to implementation of algorithms of this kind. The approach is unknown, in spite of die fact Uiat it is possible to use on the oldest and smallest computer. It can present a number of advantages like a speed up of a 1,000 times, low memory consumption, compact representation, etc.

The execution speed enhancement is important both for real-life applications, and for simulations and experiments in science. A large efficiency improvement of this magnitude can make real life applications feasible, and it could also mean Uie difference of whether an experiment or simulation will take three days or one year. This can make a given algorithm useful in a whole new domain of problems.

If an algoriUim uses a representation in a program Uiat later can be evaluated by some kind of intcφretation as a program, even in a limited sense, Uiis approach can be used. It could be Uie rules in Cellular Automata, Uie creatures in artificial life, decision trees, rules, simulations of adaptive behavior, or as demonstrated below evolving Turing complete algoriUims widi an evolutionary algoriUim. As described above, die present approach is referred to as a compiling approach because there are no inteφreting parts and die individuals arc, in effect, directly compiled.

The present method of binary machine code manipulation should not be confused with translating a machine learning algorithm into assembly code, the way a compiler operates. Instead, die present invention is capable of meta manipulating machine code in an efficient way.

The Present Compiling Approach

It is suφrising diat tiiere are no existing programming languages with specialized features and tools for Uiis programming paradigm.

The present invention provides advantages of d e prior art including die following. Higher execution speed

Compact kernel Compact representation Uncomplicated memory management Low complexity algoriUims The efficiency improvement in binary meta manipulations comes from deletion of d e inteφreting steps. Instead of using a structure defined by die programmer to be run dirough an inteφreter, a match between the data structures of die application and die binary machine code structures is sought. The algoridim then manipulates d e binaries and instead of inteφreting, d e system executes the binary directly. The efficiency benefit comes mostly from deletion of Uie inteφreting steps, but also from the simplicity of manipulating Uie linear integer array that constitutes Uie binary machine code.

The lack of an inteφreting or compiling step makes the kernel of d e algorithm compact, because no definition of the inteφretation is needed. The machine code binary format is often compact in itself, which is important when working with applications Uiat use a large set of structures, for example evolutionary algoritiims. cellular automata, etc. The inherent linear structure of a binary code program forces die design of a system Uiat uses uncomplicated memory management The decreased use of pointer and garbage collection imposes a straightforward handling of memory that is an advantage in for example real-time applications.

Some parts of the machine learning algoridim might be simplified by die restriction of the instructions to integer arithmetic. Handling of a sequence of integers is a task Uiat a processor handles efficiently and compactly, which could simplify die suncture of the algoriUim.

The approach of binary manipulating machine learning algorithms builds on die idea behind a von Neumann machine, and it is dius applicable on most of today computers from die largest super computers to die small invisible systems in cars, cameras and washing machines.

Von Neumann Machines

A Von Neumann machine is a machine where die program of die computer resides in die same storage as die data used by Uiat program. This machine is named after die famous Hungarian/ American maύiematician

Jon von Neumann, and almost all computers are today of the von Neumann type. The fact diat die program could be considered as just anotiier kind of data makes it possible to build programs diat manipulate programs, and programs diat manipulate tiiemselves.

The memory in a machine of diis type could be viewed as an indexed array of integers, and a program is thus also an array of integer numbers. Different machines use different maximal sizes of integers. A 32-bit processor is currently die most commonly commercially available type. This means dial die memory of diis machine could be viewed as an array of integers widi a maximum size of 2⁵²-l, which is equal to 4294967295, and program in such a machine is notiiing more man an array of numbers between zero -and 4294967295.

A program diat manipulates anotiier program's binary instructions is just a program diat manipulates an array of integers. The idea of regarding program and data as somediing different is however deeply rooted in our way of tiiinking. It is so deeply rooted diat most designers of higher language programmed have made it impossible for the programmer to access and manipulate binary programs. It is also surprising dial no languages are designed for diis kind of task. There are a no higher level languages that directly support diis paradigm with die appropriate tools and sunctures. The C language is desirable for practicing die present invention because it makes it possible to manipulate die memory where die program is stored.

The Processor

The processor is a "black box" which does die "intelligent" work in a computer. The principles of different available processors are surprisingly similar. The processor consists of several parts as illustrated in FIG. 3, including die control logic 44 which access die memory 12, the ALU 40, and the registers 46.

The control logic 44 uses a register or some arithmetic combination of registers to get an index number or address. The content of die memory array element with this index number is tiien placed in one of die registers of die processor.

A register is a place inside die processor where an integer can be stored. Normally a register can store an integer with die same size as die so-called word size of die processor. A 32-bit processor have registers that can store integers between 0 and 4294967295.

The most important register is the program counter (PC) which stores the address of die next insunction to be executed by die processor. The processor looks at the contents of die memory array at the position of die program counter and interprets diis integer as an instruction that might be an addition of two registers or placing a value from memory into a register. An addition of a number to die program counter itself causes transfer of control to anodier part of the memory, in odier words a jump to anodier part of die program. After doing an instruction die program counter is incremented by one and anodier instruction is read from memory and executed.

The ALU in die processor perform arithmetic and logic instructions between registers. All processors can do addition, subϋ^*action, logical "and", logical "or", etc. More advanced processors do multiplication and division of integers, and some have floating point units widi corresponding registers.

Every behavior we can see in modern computers is based on tliese simple principles. Computers doing graphics, animations, controlling a washing machine or watching die ignition system in a car all does diese memory manipulations and register operations. The principles above do not anywhere prevent a program from manipulating the contents of memory locations d at later will be placed in die program counter and run by die processor as it interprets it as instructions. This is die basis of die binary manipulating machine learning code in accordance widi die present invention.

Machine Code and Assembly Code

Machine language is die integers diat constitute die program diat die processor is executing. These numbers could be expressed widi, for example, different radix such as decimal, octal, hexadecimal or binary. By binary machine code we mean die actual numbers stored (in binary format) in die computer.

When programming or discussing machine language, it is often impractical to use numbers for instructions. To remember diat die addition insuTiction for example is represented by die integer 2416058368 in die SUN SPARC architecture is not natural to d e human mind. If we represent it in hexadecimal radix (E219922D), it will be more compact and it is more easy to deal widi dian its binary equivalent, but it is still not natural to remember. For d is reason, assembly language has been developed. Assembly language uses mnemonics to represent machine code instruction. For example, addition is represented by die diree letters "ADD".

The grammar for assembly language is very simple, and die U^*anslation or mapping from assembly language to machine code is simple and sϋ^*aightforward. Assembly language is not. however, machine language, and cannot be executed by die processor directly without die U^*anslation step.

RISC and CISC

The present invention has been implemented using both processors of the CISC type including die

Intel 80486, and of die RISC type including the SUN-SPARC architecture. CISC stands for Complex Instruction Set Computer, and RISC stands for Reduced Instruction Set Computer. As indicated by d e acronym, d e RISC processor has fewer and less complex instructions dian the CISC processor. This means diat die RISC processor can be implemented differently in hardware and diat it therefore will be faster.

There are advantages of bodi paradigms when using diem for binary manipulating machine learning. The CISC has die advantages of many more types of instructions and addressing modes, and a subset can re readily found which makes a particular implementation straightforward. The RISC processor on die odier hand has die advantage diat die structure or "grammar" of an instruction is very simple, for example die number of bits is constant. This sometimes makes the manipulation easier. It is easier to check for a valid insunction. A RISC is also often faster dian a CISC.

Although die present invention can be practiced using CISC and RISC systems, it is somewhat easier to administrate on an RISC based architecture.

Anodier important factor is Uic operating system and hardware support of the development environment. A system like UNIX which offers separateprocesses and hardware implemented memory protection is definitely to recommend during development, because it will save time when bugs appear during die development phase. These systems do not have to be restarted after a serious bug like an illegal insunction. It should, however, be noted diat once the system is debugged it will run as safely on any platform as any odier conventional program.

Stnicture of Machine Code Function

A procedure and a function can be regarded as very similar concepts in machine code or even in C A procedure is a function diat does not return a value. When implemendng a machine learning system with a binary manipulating technique, functions are die basic structures. For example, the individuals in compiling genetic algoridims are implemented as machine code functions. A function call has to perform three different sub tasks:

Program control transfer and saving die start address

Saving processor registers Transfer parameters

The most important instruction for functions and procedures is die call instniction. It is present in all kinds of processors, and works as a jump insunction diat saves the memory location or address of where it was jumping from. This will allow a return insunction to return back to diis memory location when execution of the function is complete. Most call instructions save die return address in a special memory segment called die stack, but some save it internally in a register in die processor. The SUN SPARC architecture saves die return address in a register, if diere is enough room for it.

A call insunction is not sufficient to make a complete function call. The contents of die registers must be saved somewhere before ie actual insunctions of the function are executed. This assures diat die called function will not interfere with the processing in the calling function, and gives die called function the liberty to manipulate these registers itself. The most common place to store the contents of die calling functions registers, is the stack. Some architectures like die SPARC architecture stores it inside the processor by die special save instruction.

When the execution of die function is done, die registers of die calling function have to be restored to allow diis function to continue processing in ie context it was working in before die function call.

The last task a processor has to accomplish in order to perform a complete funcUon call is to transfer die parameters to the function It has to transfer die input parameters when die call is performed, and transfer back die return values after the execution of die function is complete Again, this is most commonly done by stonng these values in die stack but it can also be done by using special registers inside the processor

These four important steps calling die function, saving and restoring registers, transfer of parameters and return of outputs can be done in different ways It can even be done in different ways on the same processor Normally it is die |ob of die compiler to produce machine code instructions diat perform these diree steps in a suitable w ^

In order to make it possible to link object code from different compilers and to increase portability, a number of standards recommendations and conventions have emerged There is for example die calling convention whereby it is up to die calling function to save die registers or whereby it is up to die called function to save die registers In die SPARC architecture, it is die called function diat saves die callers registers In a similar manner there are conventions for how parameters should be U-ansferred

When working widi machine learning at die binary level, it is sometimes allowable to be more free and for example not always save all registers if it is known that die called function will not use diem This not only provides a more efficient implementation but also presents the opportunity to use special features of die architecture diat are hard to express in a high level language

As descπbed above, die structure of a function on the machine code level can be considered as an array of integers divided having four parts

Header Body

Footer

Return

Tlie header of a function does one or more of die uiree steps mentioned above In die SPARC architecture the header of die function saves die registers of die calling function and sometimes also transfers parameters Which of die diree steps die header does is different from processor to processor and from compiler to compiler

Although most function structures need some kind of header to perform a function call, some special kinds of functions do not need a header, such as die leaf procedure in die SPARC architecture

Tlie header is fairly constant and normally does not have to be manipulated by die machine learning part of die program It can be defined at an early stage, for example in an initialization phase of die system

The body of die function does the actual work that the function is supposed to carry out When the body of die function is entered, all of its parameters are accessible to it, and die registers from die calling function are saved The body can dien use any of die arithmetic instructions or call another function to compute the desired function The footer contains the "cleanup" instructions as desenbed elsewhere

A return ιnsu^*uctιon must always follows die footer This insunction finds out where die call to dus function was made from, and then -umps back to dus location The address to jump back to is eidier stored on die stack or in special registers in tlic processor.

SPARC Architecture

A system diat directly manipulates machine code is not easily made portable, so it is recommended to choose one platform as a base for experimentation. The preferred system includes tlie SPARC architecture and SUN workstations. The reason for this is that is one of the most widely used architectures in the research community, with a stable UNIX operating system. It is also a relatively fast RISC architecture.

SPARC is an open architecture with several different implementations and manufacturers. The SPARC International, Inc. is a non-profit consortium for evolution and standardization of this architecture. A well known user of SPARC is SUN Microsystems Inc., whicli uses tlie architecture in all of its modern workstations.

SPARC Renisters

As described above with reference to FIG. 3. tlie most important of the different kinds of registers ;ιs applicable to tlie present invention are tlie "windowed registers". It is between the windowed registers that almost all arithmetic and logic operators take place. There are a dozen other registers that are more important to tlie system software than to a client application. The program counter is also an important register. The windowed registers are divided into four classes:

Output registers O₀ to 0₇ Input registers I₀ to I₇ Local registers L₀ to L₇

There are also eight global registers G₀ to G₇. There are eight registers in each of tliese classes. When a save instruction is executed, it copies the contents of tlie output registers into a new set or bank of corresponding input registers. Register O₀ is copied into I„. O, is copied into I,, etc. The input register used is a new input register owned by tlie called function.

The values in the old input registers are kept or saved. It will be noted that tlie contents are not really copied. In reality tliese registers are the same, and only a pointer to tlie current set of register in incremented. Tlie processor has an internal storage for a few banks or sets of register like this (seven in the SPARC architecture), and if this limit is exceeded, the hardware and system software will save the contents in memory.

For the user, this banked register mechanism can be thought of as a small internal stack.

It is important to note that a save instruction copies the contents of tlie calling function's output registers into the called function's input registers, while the restore instruction copies the contents of the returning function's input registers into the calling function's output registers.

When a function wants to call another function and pass some variables, it thus places the parameters in its output registers and makes the call. After the function call has been completed, the returning values can be found again in the output registers.

The local registers arc local to the function working for the moment, and are used for temporary storage. A fresh set of local registers is provided when a new function is called, but there is no special transfer of values into these registers.

The global registers are always the same. They keep their meaning and content across function calls, and can dius be used to store global values. There are no alternative sets of u is registers like there arc widi die input, output, and local registers.

Register Implementation

Some of die registers have a reserved usage. In some cases the reserved meaning is due to hardware, but in most cases d e origin of diese constraints are software conventions, as specified by die SUN Application Binary Interface (ABI).

In accordance widi Uie present invention, almost all of die actual algoridim is written in a high level language like C. This prevents interference with die work done by the compiler. If this goal is abandoned, and die complete system were written in assembly language, even higher flexibility could be achieved. However, all portability would be lost, and it is dicrefore preferred to provide die manipulating program in a high level language such as C.

The global registers are preferably not used by die program, because diese registers are likely to be used by the code generated by die compiler. Global storage for function structures can be provided in memory.

Global register zero has a special meaning. It is not a register where values can be stored. An attempt to read global register zero, always returns tlie value zero, and an attempt to write to global register zero docs not change anything. This register is used when a zero constant is needed or when tlie result of an operation should be discarded. Global register one is by convention assumed to be destroyed across function calls, so diis register can be used by die functions diat are manipulated by the machine learning algorithm.

Registers l₆, I₇, 0₆, and O, are by convention used to store stack and frame pointers as well as tlie return address in a function call, so diese should not be used by the program.

Local registers zero and one have special uses during an interrupt but this will not affect die functions. Thus, die registers whicli are available for use in die SPARC architecture are global register G|, input registers 1₀ to I₅, output registers O₀ to O_s, and local registers L₀ to L₇.

SPARC Instπictions

SPARC is a RISC architecture with a word length of 32 bits. AH of the instructions have diis size Basically diere are three different formats of instructions, defining die meaning of the 32 bits. The processor distinguishes between the foπnats by looking at die last two bits, bit 30 and bit 31. The diree formats are: CALL instruction Branches, etc.

Arithmetic and logic insϋrυctions

In the CALL instruction. bit30 is one and bit31 is zero. The rest of die bits are inteφreted as a constant that is added to die program counter (PC). The return address is stored in output register 1₇. Branches are mostly used for conditional jumps. They look at die last performed arithmetic operation, and if it fulfills a certain criteria, for example if the result is zero, then the program counter is incremented or decremented by die value of the last 22 bits in d e instructions. In a branch, bodi bit 30 and bit 1 arc zero.

The last group of instructions are die aridimetic and logic instructions between registers. These groups of instructions have bit 31 as one and bit 30 as zero. Tliese are die preferred instructions for practicing die present invention. These instructions perform, for example, multiplication, division, subu-action. addition, AND. OR, NOT. XOR and different SHIFT instructions.

The arithmetic instructions can also be used as jump, call, and return instructions, if the result of die operation is put into die program counter register. In this way it is possible to jump to die address pointed to by die contents of a register. When die current value of die program counter is saved in anodier register (out7). diese instructions can be used as call instructions.

Call is equivalent to jmpl register, %o7, where jmpl means "jump long"; jump to die content of "register" and place the current program counter in register O,.

Return (ret) from a normal procedure is equivalent to jmpl. Return from a leaf procedure (ret) is equivalent with jmpl.

The constant eight causes die control to jump back past the original call and past diis call's delay slot

The return instruction puts die value of 0₇ with a constant added thereto into die program counter, and causes tlie execution of die program to return.

There are also instructions for loading and saving information from die memory. Control transfer instructions (CTI) like jump, call, and return, are somewhat special in die SPARC architecture. The instruction immediately after the jump instruction is executed during die transfer of control, or it could be said to be executed before die jump.

A SPARC assembly code listing can appear misleading because of the change in execution order, because a NOP instruction is placed after die CTI. The insϋ^*uction after the CTI is say to be in die delay slot of die CTI. The reason for this somewhat awkward mechanism is diat if die delay slot can be filled widi a useful instruction, it will make the execution of die overall program more effective, because die processor can do more instructions in the same cycles. It is die hardware construction of die processor the makes diis arrangement necessary.

The CTI and the delay slot are important in implementing the CGPS. Special rules require diat a CTI diat has a CTI in its delay slot. This is called a Delayed Control Transfer Couple.

The execution of a DCTI couple is awkward. The first (and only die first) instruction of die function diat die first CTI is pointing to is executed, then directly die execution jumps to die function diat die second CTI is pointing al. The insunction after die second CTI is not regarded as being in a delay slot, and it is executed when control is returned. This phenomena is important for die construction of subroutines and external functions. Leaf Procedures

The above described properties of die call and save instructions make it possible to use two particular kinds of procedures.

The first kind of procedure is a full procedure diat uses die save and restore instructions, and die procedure is consequently allowed to use and manipulate input, output and local registers. The save and restore functions, however, consume processor cycles. If diere is room inside die processor, die time consumption is moderate, but if storage in memory is needed it will take many processor cycles.

T e solution to diis problem is to use leaf procedures. They are called leaf procedures because dicy cannot call another procedure, and therefore leaves in the procedure structure of a program. A leaf procedure does not perform a save operation, and works widi die same set of registers as die aiding procedure. To avoid interference widi die content of die calling procedure, it only manipulates die output registers, which are assumed to be destroyed by die compiler across function calls. One of die elegant consequences of diis technique is diat die calling procedure does not have to know what kind of procedure it is calling. This means diat linking of procedures works normally. The difference between a leaf procedure and a full procedure is that it only manipulates die output registers, does not use save or restore, and has a special return insunction that looks for the return address in output register 0₇ seven instead of input register I₇.

These details originate mostly from software conventions implemented in compilers. When die underlying principles are known, it is possible to have many more combinations of behavior of procedures. In some of examples in *CPGS, hybrids of die normal procedures and leaf procedures are used, because it is known ow die registers are used in die calling procedures.

Aldiough the floating point features of die computer architecture are not specifically addressed herein, it is possible to use binary manipulating algorithms for floating point processors and units. This makes die present invention applicable to new domains of problems.

Evolutionary Algorithms

Examples of genetic algoridi s were present above widi reference to FIGs. 10 to 15. A genetic algoridim. for example, is an algoridim based on die principle of natural selection. A set of potential solutions, a population, is measured against a fitness criteria, and dirough iterations is refined with mutation and recombination (crossover) operators. In the original genetic algorithm, the individuals in d e populations consist of fixed lengtli binary strings. The recombination operators used are the uniform and 2-point crossover operators.

In genetic programming die goal of die system is to evolve algorithms or programs in a given language. Most genetic programming systems use a u^*ee structure for the representation of programs, as in die above referenced patent to Koza. The most commonly used tree representation form is the S-expressions used in die LISP-language. The tree representation guarantees diat die evolved programs will be syntactically correct after the genetic operators are applied. In the original model, die only genetic operator apart from selection is the subtree exchanging crossover. This crossover operator swaps two subu^*ees in die individuals diat undergo crossover.

A middle way between diese two representation forms is messy genetic algoriUims, which has a freer form of representation where for example die loci of gene is not tied to its inteφretation. Instead genes arc tagged widi a name to enable identification.

Evolution of Machine Code Structures and Content

A basic machine learning system was described above widi referenceto FIGs. 1 to 18. Our discussion of t e prior system provides many of die basic details of the present invention and is incoφorated here by reference. The algoridim evolves fixed lengtli binary strings, and uses crossover and mutation operators. The crossover is prevented from damaging certain bit fields widiin die 32 bits of the instructions. This protection procedure guarantees diat only valid machine code instructions are generated dirough crossover. The mutation operator has a similar protection feature.

The system described below is a more complete machine learning and induction system, capable of evolving Turing complete algoriduns and machine code functions. The system provides additional advantages including die following. Use of several machine registers

Dynamic allocation of memory, (no recompilation needed) Variable lengtli of programs. Multiple input parameters to functions Unlimited memory dirough indexed memory Automatic evolution of subfunctions

If-dien-else structures Jumps

Use of loop structures including for, while, repeat Recursion, direct and dirough subfunctions Protected functions e.g. division

String functions, and list functions Linking any C-function for use in the function set

These goals are met by a program written mostly in C, which uses unrestricted crossover at instruction boundaries. Unrestricted crossover means diat die crossover acting on die strings of instructions should be able to work blindly widiout checking what kind of instructions are moved and where.

There are many advantages of this if die goal is met. Tlie implementation is very efficient, because die algoridim will only consist of a loop moving a sequence of integers, something a computer is very good at. The implementation will be simple and easily extendable because iere will be a minimum of interaction and interference between parts of die program. Equally important is die fact diat die program will be more easily ported to different platforms because the architecture specific parts can be restricted to a minor part of the program, and die crossover mechanism does not have to be effected. It is easy to find examples of instructions and combinations of instructions where these properties do not hold. The normal call instructions constrains an offset that is added to the program counter. If diis instruction is moved to anodier memory position, die call will point to anodier memory location where there might not be anytiiing like a proper function. The SPARC architecture does not have calls widi absolute addresses, whicli would still work after crossover. Instead, a call to an address specified by a register is used. The value in die register will be the same even if the instruction is moved by crossover.

The same problem exists with a normal branch instruction. It is also defined by a constant added to die program counter, and it cannot eidier be moved widiout changing diis constant. But a certain class of instructions are possible to move unrestrictedly, widiout any unwanted effects. Machine language is quite flexible, and diis restriction of instruction use is possible widiout abandoning die goals of die system.

Crossover

The crossover operators (uniform and 2-point) were described in detail above. See Figure 22 j.

Mutation

The mutation operator picks an insunction at random, and checks whedier it has a constant part or if it is only an operation between registers. If it has a constant part, a bit in diis constant is mutated and also potentially die source and destination registers of die operation. If die instruction does not have a constraint part, die instruction's type, source and destination registers are mutated. See Figure 22i.

Calls and jumps are not mutated odier dian diat diey may be swapped for odier instructions.

Casting and Execution of an Array In a computer, all data is represented as integer numbers. To make programming a more easy task diere are more data types and structures in a high level program, for example: strings, pointers, characters bitmaps, etc

But all of diese structures are, as stated above, translated into integers Sometimes die programmer wants to translate an object of a certain type into an object of anodier type. The term for this is casting, one type is cast into anodier type. This is often only an operation in the high level language, at the machine code level diey might be represented by die same integer.

Speed Increase

The efficiency of die present invention can be 1,000 times faster than a system coded in an inteφreting language, and diere are reasons to believethat similar performance enhancements are possible for other kinds of meta manipulating programs. If, however, diese speed enliancements still would not be enough diere are a few ways to further improve performance.

The main system described here has a number of advanced features like subroutines, recursion, etc. If extremely fast performance is needed and if a system can do without diese features and if die goal is to only evolve aridinietic and logic expressions of a number of "in" variables, dien the efficiency can be increased even furdier.

More specifically, if it is acceptableto decompile die system between runs, for example if die problem parameters are given beforehand, dien execution speed can be increased furdier. Finally, diere is die possibility of coding and optimizing die entire system in assembly language which has die fastest execution.

System Parameters

The following is a list and a brief description of die parameters Uiat are used to control die present machine learning system.

Population size Number of fitness cases to evaluate. Training will halt when die system has tried diese many fitness cases.

Mutation probability

Crossover probability

Parsimony pressure start value Number of incremental steps for parsimony pressure

Size of each incremental parsimony pressure step

Maximum program size

Initial mean program size

Random number generator seed Success direshold for fitness

Maximum number of iterations, in recursion and loops

Maximum size of initial terminal set numbers

Number of bits to be mutated in terminal numbers

Number of input registers to use Number of output registers to use

Number of external functions

Maximum number of automatic subfunctions

Flags determining which instructions to use: ADD, SUB, MUL, SLL, SRL, XOR, AND, OR

These parameters are read from an initialization file before each training session. All effects of diese parameters are dynamically allocated, and diere is no recompilation of die system necessary. This implies that a flcxibl e system can be implemented on a small computer without an onboard compiler. These parameters would, of course, be different for odier machine learning systems or for repeat calculation systems on run-time data.

Machine Code Instruction Implementation

The following SPARC machine code instructions are used in die CGPS implementation. ADD, Addition SUB, Subtraction

MUL. Integer multiplication

SLL, Shift left

SRL, Shift nght 5 XOR, Exclusive or

AND, Logical And

OR Logical Or

Call L_{u 7}

The andinietic and logic instructions, all instructions except die last call instructions, come in foui 0 different classes These andinietic instructions can have die property of affecting a following lf-dien branch or not affect it They can also have die property of being an operation between three registers, or an operation between t\\ o registers and a constant The combinations of these two classes makes four different variants of tlie andinietic instructions

An anthmetic instruction could for example add output register O, widi output register 0₂ and store ^*ϊ t e result in output register O,, or it could add a constant to output register 0_: and store the result in output register O,

The 32-bit instruction format has room for a constant of ±4196 In this manner, a single instruction is substantially equivalent to many elements in an ordinary machine learning system one element for die operator, two elements for die two operands and one element for die destination of die result This approach is thus quite 0 memory effective using only four bytes of memory to store four nodes

There are eight different call instructions, one for each of die local registers The call insunction lumps to an address given by die content of the local registers in die function The local registers are thus reserved for diis usage in our implementation, and diey have to be initialized in the header of every function to make sure diat diey point to a function m use 5 These instructions arc the basis for external functions and subroutines How many of diese instructions are used in die initialization is determined by how many subroutines and external functions diat are used

The maximum number of external functions and subroutines are in tins implementation limited to eight If more external functions are needed, dien anotiier initialization technique can be used where die call instructions jumps to die address given by a constant added to a local register The instruction format allows for storage of such constants widnn die instruction

The division insunction can be used directly as a basic instruction Protection from division by zero can be provided by catching die interrupt generated by hardware It is, however, more effective to make protected division as an external function, with a few instructions checking for division by zero Initialization

Initialization is die first task Uie system performs when presented widi a giving training situation. See FIGs. 22b, 22c. Initialization can be divided into four steps.

Allocation of memory for Uic population, etc. Initialization of header

Initialization of footer

Initialization of Uic function body

Initialization of Memory FIGs. 22b, 22c.

The memory used for the individuals in die arrays is a linear array of integers. The array is divided into blocks determined by die system parameter maximum length. A fixed maximum lengdi is dius reserved for every individual. If diere are subroutines, dien diis memory is allocated for every subroutine according to die maximal number of subroutines. The program and its subroutines Uicn can vary in lengUi widiin Uiese boundaries_.

The advantages widi Uiis paradigm include very simple memory management, widiout garbage collection. The approach widi linear memory is efficient and natural for die use of binary code. There is also a constant - easily calculated - memory usage once die system is initialized. This can be a requirement in real- lime systems, and in systems widi limited memory resources.

An example population of one million individuals widi no subroutines and 40 nodes maximum per individual in diis way occupies 40MB of RAM memory, which is manageable on an ordinary workstation.

Initialization of Header FIGs. 22b, 22c. The initialization of die header consists of two parts; one Uiat is fixed, and one Uiat depends on die number of subroutines and external functions.

The fixed part of die header is a NOP instruction and a save instruction.

The NOP instruction is needed for die rare case uiat diis function is called from anodier function as a subroutine by a control transfer couple as described above. A control transfer couple can arise after an unrestricted crossover where two jumps or call instructions are placed after each other.

Only the first instruction of Uie first call is executed. If Uiis first instruction is a save instruction, which is normal, Uien this save would be executed alone and die control will go to die address of die second call which probably will also be a save insunction.

The first of diese two save instructions will be unbalanced, not corresponding to a restore, and die registers will be corrupt. If instead a NOP is placed in the header of every instruction, Uie NOP insunction can be executed safely without affecting die state of die machine.

This point is probably die most special case in the implementation, but it enables die use of unrestricted crossover. When an external function is to be used in die system, it should also have a NOP instruction or anoUier harmless instruction as its first instruction. The second part of die header initializes die local registers. The local registers are used to store j ump addresses of subroutines and external functions This part of die header contains load msunctions that load die appropnate addresses into die local registers The current addresses are put here die header dunng initialization and Uien yvhen an individual function is executed, it first executes die header and Uius puts die right values in die local registers, whicli guarantees diat die later call and lumps will be performed to die desired addresses This part of die header vanes in sizes by die number of subroutines and external functions

Initialization of Footer FIGs 22b, 22c

The initialization of Uic footer is simple The footer consists of two fixed instructions, one NOP instruction followed by a restore instruction The NOP instruction is used in die last instruction m the body as a control transfer instruction like a jump or a call In Uiis case, Uie NOP instruction goes into Uie delay slot of Uie jump instruction and is executed dunng dus procedure call II Uie NOP instruction was not present Uie restore instruction or die return instruction diat follows die footer would go into Uie delay slot which would corrupt die registers We have previously discussed headers and dial discussion applies here also

Initialization of Return Instruction

A return instruction must follow the footer

Initialization of Function Body FIGs 22b, 22c, 22d

The function body is initialized by for each memory cell by randomly selecting an insunction from the set of instructions Uiat Uie user has put into Uie function set, including call instructions using local registers If tlie picked instruction is an andinietic instruction, input and output registers are chosen for operands and destination according to die parameters supplied by die user We have previously discussed headers, bodies, footers and return instructions, and Uiat discussion applies here also

WiUi a ceπain probability, an instruction is given eiϋier a constant and register as operands or two registers as operands If one of die operands is a constant, this constant is randomly generated to a maximum size defined by its parameter and put into die insunction The instruction has room for constants widiin Uie range of ±4196, wiUnn Uie 32 bits of the instruction

Subroutines

Subroutines arc modularisations within an individual Uiat spontaneously change dunng evolution A subroutine as die same structure as a function call, and consists of parameters transferred in die out vanables, and a call function An individual m dus system is a linear array of numbers This array is divided into a number of pieces of uniform size The number of pieces corresponds to die maximum number of die subroutine parameter Every such memory section is a subroutine A subroutine is organized die same as a main function with a header, a looter and a function body In Uie header of a subroutine, die local registers are initialized to contain die addresses of die odier subroutines that can be called from diis subroutine. When die recursion option of die system is switched off, die local registers are only loaded widi die addresses of subroutines higher up in die hierarchy. So if die maximum number of subroutines is set to four, the local registers L₀ to L, in die main function are initialized with die addresses of subroutines 0 to 3, while die local registers L„ and L, in die first subroutine are initialized with die addresses of subroutines 3 and 4.

Remaining unused local registers are initialized widi die value of a dummy function at executes a return when called. WiUi Uie scheme it is possible to allow unrestricted crossover between individuals and between subfunctions, because the local registers will always be initialized in die header of each subfunclion to a correct address. The "call local register" instructions can thus be freely copied in die population.

Recursion

Recursion can be implemented by initializing the local registers not only to die values of subroutines higher up in Uie hierarchy, but also to Uic current subroutine itself. Recursion can also be implemented in a small external function, a leaf function. The difference between die two approaches is small, but Uie main advantage of die later niediod is Uiat die header can be kept die same regardless of if recursion is used or not, which makes die implementation less complex.

Regardless of which approach is used for recursion, there will always be Uie problem of infinite chains of instruction calls. The halting problem makes it impossible to know in advance which recursive functions will slop and which will not. The solution to diis is to have a global variable in a memory location diat is incremented every time a function is called. If a certain limit is reached, Uie execution of die individual is abandoned. The code for this checking is placed in the header of each function if Uie first recursion method is used If die second niediod is used, diis code is die first part of die recursion leaf procedure.

Leaf Procedures as Program Primitives. Loops are implemented in a way similar to recursion. A leaf procedure is used which performs a test of a variable. Depending on die outcome of die test, a loop branch is either performed or not.

The test can be whedier the last performed arithmetic insunction produced zero as a result. This is accomplished by checking d e so called zero flag in die processor Out loop structures can be used simultaneously by checking other integer conditions from Uie last insunction. This branch is made with a return instruction, which is a "longjump" instruction jumping to die address of a register wiUi a constant added diereto. This constant can be positive or negative. The normal return insunction jumps back to die address given by the content of register O, or I₇ incremented by eight. These eight bytes cause die return to skip Uie original call instruction and its delay slot.

When diis function is used in a loop, die constant is made negative so Uiat return jumps back to an address before die call to die "loop leaf procedure". The fact Uiat Uie different call addresses so far are in Uie execution of Uie individual, all are stored and accessible to Uie leaf procedure which makes it possible to construct a large number of control structures. Widi Uiis meUiod, it is possible to define efficient external leaf procedures that implement ordinary jumps, all kinds of loops, indexed memory, if-dien-else structures, and protected functions. A limit of eight external functions is implied by die number of local registers. This is enough for many applications, but if odier domains demand more external functions and subroutines, it is possible to define their addresses as constants added to local register. This technique provides the possibility of using an almost unlimited number of primitives, external function and subroutines.

External Functions

It is possible to incorporate any C-function into die function set. Any C-module can be compiled and linked into the system. There are a number of steps Uiat must be taken in order to make diis linking successful.

First die module could be compiled to assembler code by die "-S" flag of Uie "cc" compiler. A NOP operation is dien added before the other instructions in the function. The name of die function is added in a call array in the main C kernel code, and potentially a string name is added to an array for disassembling. After rccompilalion Uie system is ready to use Uie new external function. This approach can be extended and made more automatic widi dynamic linking, etc.

C-Langιιat;e Output

A disassembler can be provided whicli translates die generated binary machine code into C-language modules. This disassembler feature is provided wiUi Uie goal Uiat Uie output from Uie system should be able to be used directly as a C-module. and it should be possible to compile and link it to anodier c-program. WiUi this feature Uie system can be used as a utility to a conventional development environment. The disassembler could also be requested to produce assembler code.

Portability Methods

The main disadvantage widi Uie machine code manipulating technique is Uiat it is machine specific. There are, however, a number of mediods to make die system more portable. The first is to allow for unrestricted crossover. This will make the machine independent part of die code quite small, only defining a number corresponding to instructions, etc. In this way it is possible to write a system diat runs on multiple platforms without changes in die code.

An more radical approach is to let Uie system look for itself in its binary code how it is translated lo machine language. Many template functions exist in die C-code widi translations to machine code Uiat define how die processor is working. This niediod is only possible for compiling systems Uiat do simple functions widi aridimetic, without jumps, subroutines, etc.

Anodier approach is to lean more on die hardware interrupts for portability. For example, every time an illegal insunction is encountered by the processor, a hardware interrupt is generated. Using die interrupt features in Unix it is possible to be less restrictive when manipulating binaries knowing that die system will catch some of Uie erroneous structures.

This increases portability, because faults can sometimes be ignored diat arise due to incompatibilities of different architectures. In an extreme example, a machine learning system can be implemented by using mutation and crossover at die bit level where all of the many illegal situation are caught by die processor memory management hardware or file system protection mechanisms.

The ideal portability situation would be to have a special language, present on different platforms, for Uiis kind of run-time binary manipulation.

Using Tree Representation

In Uie preferred implementation of Uie present invention, a binary suing approach is used for representation of individuals. Binary strings are die common representation form of genetic algorithms while tree representations are used in conventional genetic programming.

There may be applications where tree representation and crossover at a tree level is interesting to study and consequently it would be interesting to have die individual in a tree form. This can be accomplished in several ways. One way is lo use two different NOP instruction as parenthesis in Uie code. The crossover operator dien scans dirough Uie code, matches diese parendiesis, and performs tree crossover accordingly. This could potentially make Uic crossover operator less efficient for large individual sizes.

Another way is to break up Uic individual into segments in die memory, where part of die tree is presented as jump statements to subtrees. This would still make it possible lo execute Uie individual directly. The crossover operator here must also scan dirough Uie individual, but it only has to move smaller blocks of code diat could be allocated dynamically.

The Uiird way is to let every individual have an exϋ^*a array associated with it, Uiat carries information about where subtrees start and stop. This array will not be involved in die execution of die individual, and it will only be used by die crossover operator to locate subtrees.

Applications

The present invention can be applied to any problem in which a computer algorithm manipulates a structure Uiat later should be inteφreted as instructions. Examples of these kind of such applications include die following.

Genetic algoriUims and genetic programming Cellular automata

Artificial life systems like Tierra

Rule induction systems

Decision trees

LISP or PROLOG interpreters The invention is especially suited for applications widiin areas that require: High execution speed Real time learning Large memory structures Low end architectures, e.g. consumer electronics Well defined memory behavior

Example

FIGs. 1 to 21 illustrate a function structure and associated registers in accordance widi die Turing complete machine learning system of die present invention.

FIG. 1 illustrates an array of functions F₀ to F₆, each of which consists of a main function MAIN and two subroutines SUB l and SUB2. Each function has a ma.ximum, but variable lengUi. Portions of die functions occupied by die header are indicated as H. die instruction body as B. die footer as F, and the return instruction as R.

It will be noted Uiat Uie main functions and die subroutines SUBl and SUB2 also have variable, but maximum lcngdis Any memory not occupied by an operative portion of a function is unused. FIG. 20 illustrates a function FUNCTION which consists of a main function MAIN, and two subroutines SUB l and SUB2. Furdier illustrated are die input registers, output registers and local registers of BANKO which is used by die main function, a leaf function, and a dummy function. The latter functions are stored in die memory 12.

The starting addresses of die functions MAIN, SUB 1 and SUB2 are designated as adO, ad 1 , and ad2. whereas die starting addresses of die functions LEAF and DUMMY are designated as ad3 and ad4 respectively. The instructions which can be placed in die functions MAIN, SUB l AND SUB2 and which are subject to alteration for die purpose of machine learning arc limited to diose which branch to the addresses adO to ad4.

The function LEAF performs Uie operation of protected division, where a variable "c" stored in Uie input register I- is to be divided by a variable "b" stored in die input register I, . More specifically, an insunction "TEST I„" tests wheϋier or not Uie variable "b" is zero. If so. Uie result of Uie division would be an infinitely large number, which constitutes an error for the system.

If die test instruction detects diat the value of I, is zero, the next instruction is skipped and die next instruction, which is, a RETURN instruction, is executed, returning control the calling function.

It Uie test insunction detects diat die value of I, is not zero, die division operation c b is performed, die result stored in die input register 1₃ (as a variable d), and control returned to die calling function.

The function DUMMY consists of a return instruction that merely returns control to Uie calling function.

Tlie headers of the functions MAIN, SUBl AND SUB2 each include a SAVE instruction, and diree instructions at initialize the local registers with the addresses of functions that can be called by the particular function. The SAVE instruction causes the contents of die output registers of a calling function to be copied into die input registers of Uie called function as described above. In die function MAIN, diese initialization instructions cause die addresses adl, ad2 and ad3 to be stored in die local registers L₀, L,. and L, respectively.

In Uie normal hierarchial arrangement of die invention, functions are only allowed to call functions which are lower on die hierarchy, and leaf functions. Thus, die function SUBl is allowed to call die function 5 SUB2, but not die function MAIN. This is accomplished by storing ad2 for d e function SUB2 in L„, ad3 for die function LEAF in L,, and ad4 for die function DUMMY in L,.

The function SUB2 is only allowed to call leaf functions. Therefore, the address ad3 for die function LEAF is stored, in L₀, and die address ad4 for die function DUMMY is stored in L, and L₂.

FIG. 20 furdier illusu^*ates how an aridimetic instruction is performed by executing a function, and 10 how a variable is passed from a calling function to a called function.

More specifically, execution of die instruction I- = I, *2 by die function MAIN causes die variable "d" in die register 1, to be multiplied by 2, and the result stored in die register 1„ as indicated by an arrow. The contents of the register I„ arc indicated as a=b*2.

To pass the value of "a" (in I„) to a called function, an insunction O„=I₀. This causes die contents 15 of die input register l_n to be copied into die output register O₀. The next insdnction, CALL L₀, causes control to be transferred to die function having die starling address adl, which is die function SUB 1. The call insuτιction causes die contents of die output registers of die function MAIN to be copied into die input registers of die function SUB l , and diereby pass die value of die variable "a" which was stored in O₀ of tlie function MAIN to die input register I₀ of die function SUB 1. 20 Upon return from die call to SUBl, die contents of die input registers of die function SUBl are copied to die output registers of die function MAIN. The next instruction in die function MAIN is I₀=O_u. This causes die contents of O₀ to be copied to 1,„ and diereby pass die variable diat was generated by the function SUB 1 and stored in I₀ diereof to die function MAIN.

FIG. 21 is similar to FIG. 20, but illustrates an arrangement including one subroutine SUB 1 and one 25 leaf function, widi die subroutine SUB 1 being allowed to perform recursion. In diis example, die function SUB 1 can call die function MAIN, itself, and die function LEAF.

This is accomplished by storing ado (die address of the function MAIN) in L₀, storing adl (die address of itself) in L,, and ad2 (die address of the function LEAF) in L₂.

A detailed flowchart of the Turing complete machine learning system is illustrated as a flowchart in 30 FIGs. 22a to 22k.

FIG. 22a is main diagram of die system.

FIG. 22b illustrates die details of a block SETUP in FIG. 22a.

FIG. 22c illustrates the details of a block INITIALIZATION in FIG. 22a.

FIG. 22d illustrates the details of a block INITIALIZE LOCAL REGISTERS IN HEADER in FIG.

35 22c.

FIG. 22e illustrates die details of a block CREATE INSTRUCTION in FIGs. 22c and 22i; FIG. 22f illustrates die details of a block MAIN CGPS LOOP in FIG. 22a. FIG. 22g illustrates die details of a block CALCULATE INDIVfN] FITNESS in FIG. 22f. The steps indicated by a left bracket are perfoπned by a C-instruction "Predicted_Output[i]=((function_ptr) lndiv|n]0(input[l)...lnput|k])".

FIG. 22h illustrates die details of a block PERFORM GENETIC OPERATIONS in FIG. 22f. FIG. 22i illustrates die details of a blocks MUTATE INDIV[1] and MUTATE INDIV[3] in FIG. 22h.

FIG. 22j illustrates the details of a block CROSSOVER INDIVS [1] and [3] in FIG. 22h.

FIG. 22k illustrates die details of a block DECOMPILE CHOSEN SOLUTION in FIG. 22a.

LEARNING AND CONTROL METHOD AND SYSTEM EXEMPLIFIED BY A METHOD AND SYSTEM THAT LEARNS HOW TO CONTROL AN AUTONOMOUS AGENT SUCH AS A ROBOT

The present invention comprises a system and/or method to perform any learning task. In its exemplary embodiments presented herein, die present invention comprises a general method for performing real time or online learning tasks such as control of an autonomous agent or a robot. In die exemplary embodiments presented below, d e autonomous agent is a propelled object, more specifically a robot. However, it will be understood diat die invention is not so limited, and die present invention can be applied to any applicable learning task, any applicable real time or online learning task, or to control any applicable autonomous objector process.

Additional disclosure relating to diis aspect of the invention is presented in a paper entitled "REAL

TIME EVOLUTION OF BEHAVIOR AND A WORLD MODEL FOR A MINIATURE ROBOT USING

GENETIC PROGRAMMING", by Peter Nordin et al. Department of Computer Science, University of Dortmund, pp. 1-32, November 1995, which is incorporated by reference herein in its entirety.

Two preferred embodiments of diis aspect of die present invention are described herein. The First is die non-memory mediod or system. Tlie second is die memory mediod or system.

Tlie two preferred implementations of this aspect of die present invention both utilize a fitting algoridim diat is able to derive a function or program that takes one set of data as input and predicts die value of anodier set of data. The particular fitting algorithm used in die preferred embodiments of die present invention is a symbolic regression algoridim, preferably die Compiling Genetic Programming System (CGPS) as described in detail above running a symbolic regression algoridim. Symbolic regression is discussed in more detail below.

The non-memory embodiment of the present invention includes die following elements:

1. A population of solution candidates (programs), where population size can vary between 30 and 50,000 individuals. Tlie population is normally initiated to a random content. The solution candidates are referred to elsewhere in diis application as "individuals," "entities," or "soluϋons."

2. A fitness measure defining die desired task of die evolved programs.

3. A recombination or crossover operator allowing for exchange of solution segments.

4. A mutation operator for changing code more locally man the crossover operator. 5. A set of basic constants, instructions, procedures or functions used as die atomic parts of die evolved programs. Genetic Programming and Machine Code

The Genetic Programming (GP) system used in die preferred embodiment of die present invention to perfoπn symbolic regression is a vaπant of GP that uses a linear genome, and stores die individuals of die population as binary machine code in memory This results in a speed-up of several orders of magnitude The mediod is also memory efficient requiπng only 32KB for the GP kernel The individuals can be stored in an economic way and memory consumption is stable dunng evolution widiout any need for garbage collection etc

All of diese properties make die system ideally suited for real-time control in low-end processor architectures such as one-chip embedded control

The present CGPS system uses vanable lengtli stπngs of 32 bit instructions for a register machine bach node in die genome is an instruction for a register machine The register machine performs aπdimetic operations on a small set of registers Each instruction might also include a small integer constant of maximal

13 bits The 32 bits in die instruction dius represent simple aπthmetic operations such as "a=b+c" or "c=b*5"

The actual format of the 32 bits corresponds to die machine code format of a SUN-4, which enables the genetic operators to manipulate binary code directly The set-up is motivated by fast execution, low memory requirements and a linear genome whicli makes reasoning about information content less complex This compact system is a prerequisite for a microcontroller version

The machine code manipulating GP system uses two-point stπng crossover A node is die atomic crossover unit in die GP structure Crossover can occur on eidier or bodi sides of a node but not widiin a node Because our particular impleinentation of GP works widi 32 bit machine code instructions, a node is a 32 bit instruction Mutation flips bits inside die 32-bit node The mutation operator ensures at only instructions in die function set with valid ranges of registers and constants are die result of a mutation All genetic operators ensure syntactic closure dunng evolution

The instructions are all low-level machine code instructions An exemplary function set consists of the aπdimetic operations ADD SUB and MUL die shift operations SLL and SLR, and finallv die logic operations

AND, OR and XOR All diese instructions operate on 32-bit registers

The following illustrates how an individual program may look if pπntcd as a 'C program a=s3 + 4, d=s2 » si b=sl - d, d=s2 + 2, c=d » 1 , b=d - d, a=c - 3 , c=s4 « a, d=a * 4,

Δ=d ~ 5, e=d + s4 c=b & c d=d + d. c=d ! 8, d=d * 10, b=e » 6 d=b & 0 c-c » b, motor2=a ] c, c=d I 7 ιnotorl=c * 9, c=e & e,

(^»,»^•■ "- >"=siuft "|"=Or "&"=And "~"=Exor

Each individual is composed of simple instructions (program lines) between vanables and input and output parameters The input is in die form of sensor values, and is represented as register vanables (s,) The resulting output or action in die foπn of propulsion control parameters or motor speeds is also given as registers

(motorl and motor2) The same basic evolutionary algonthm is at die heart of bodi a first system (the non- memory system) which leaπis directly from actual sensor input, and a second system (die memory system) which learns indirectly from stored expeπences In die preferred embodiment of die memory system, die population of programs is initialized widi random content at die outset A simple tournament is used for die competitive selection of individuals to produce die πght distπbution of offspπng The GP-system with its simple steady state tournament selection algonthm has die following execution cycle

1 Select four arbitrary programs from die population 2 For each of die programs calculate fitness

3 Make two copies (offspπng) of the two individuals with highest fitness and let die copies be subject to crossover and mutation

4 Replace die two individuals widi worst fitness with die two new offspring

5 Repeat step 1 to 4

Symbolic Regression

Bo of die preferred embodiments of die present invention, the memory system and the non-memory system, use a fitting algondim For both embodiments, die preferred fitting algoridim is a symbolic regression algoπdim Symbolic regression is die procedure of inducing a symbolic equation, function or program which fits given numencal data Genetic programming is ideal for symbolic regression, and most GP applications could be reformulated as a vaπant of symbolic regression A GP system performing symbolic regression takes a number ol numencal input/output relations, called fitness cases, and produces a function or machine language computer program that is consistent widi diese fitness cases. Consider for example die following fitness cases: f(2) = 6 f(4) = 20 f(5) = 30 f(7) = 56

These input/output pairs or fitness cases are consistent widi die function below: f(x) = x*x+x

This very simple example would, in our register machine language, look as: a = x*x; y_out = a+x; or: a=x+l ; y_out = x*a;

Here the input and die expected output both consist of a single number, but in many cases symbolic regression is performed widi vectors specifying die input output relation of die sought function. In die examples below die input vector has more dian 10 components, and die output vector has in some cases two outputs. The fiUiess used to guide die system during evolution is often some kind of error summation of die expected values versus die actual values produced by an individual program.

Kliepera Robot Experiments Experiments for both die memory and non-memory embodiments of die present invention were perfoπned with a standard autonomous miniature robot, die Swiss mobile robot platform Kliepera, which is illustrated in FIG. 23 and designated by die reference numeral 100. It is equipped widi eight infrared proximity sensors. The mobile robot has a circular shape, a diameter of 6 cm and a height of 5 cm. It possesses two motors and an on-board power supply. The motors can be independently controlled by a PID controller. The eight infrared sensors are distributed around die robot in a circular pattern. They emit infrared light, receive die reflected light and measure distances in a short range: 2-5 cm. The robot is also equipped widi a Motorola 68331 micro-controller which can be connected to a workstation via a serial cable.

It is possible to control die robot in two ways. Tlie controlling algorithm can be run on a workstation, widi data and commands communicated through die serial line. Alternatively, the controlling algoridim can be cross-compiled on die workstation and down-loaded to die robot which dien runs die complete system in a standalone fashion. At present, two versions of the system have been developed; one where die controlling GP-system is run on die workstation, and anodier where die system is downloaded and run autonomously on the microcontroller of die robot.

The micro-controller has 250KB of RAM and a large ROM containing a small operating system. The operating system has simple multi-tasking capabilities and manages die communication widi die host computer. The robot has several extension ports where peripherals such as grippers and TV cameras can be attached. Training Environment

The training environment used for die expeπmental obstacle avoiding task used in expeπments on bodi die non-memor\ and memory embodiments of die present invention was about 70 cm λ 90 cm It has an irregular boarder widi different angles and four deceptive dead-ends in each comer In die large open area in die middle, movable obstacles can be placed The fπction between wheels and surface was low, enabling die robot to slip idi its wheels dunng a collision with an obstacle There is an increase in fnction widi die walls making it hard for die circular robot to turn while in contact widi a wall

Objectives

The goal of die GP systems in die expeπments was to evolve obstacle avoiding behavior in a sensc- dunk-act context Both die memorv and non-memory systems operate m real-time and aim at obstacle avoiding behavior denved from real noisy scnsonal data The sensonal data for both systems come from eight proxιmιt- sensors mounted on die circular platform

Symbolic regression is as mentioned above, die procedure of inducing a symbolic equation fitting en numencal data Genetic programming is ideal for symbolic regression, and most GP applications could be reformulated as vanants of symbolic regression In die non-memory embodiment of die present invention which involves obstacle avoidance, we approximate a function at takes die sensor values as an input vector and returns an action in die fonn of a vector of two motor speeds f(s_l,s₂,s„s_J,s₅,s₆,s₇,s_li) = {m-.πv,} (equ 1)

This vector models die simple stimulus-response behavior of the robot The first system (die non- memory embodiment of die present invention) evolves the function directly dirough interaction widi die environment The second approach (die memory embodiment of die present invention) evolves a simulation or woi Id model which defines a relationship between inputs (sensor values), outputs (motor speeds) and corresponding predicted fitness values as follows f(s, s₂,S_ι,s₄,Sj,s₆,s₇,s₈,ιιi| m₂) = predicted fitness (equ 2) The second embodiment of die invention is memory-based in diat a sensory-motor (input-output) state is "associated" widi a fitness diat might be termed "feeling"

Fiuiess Calculation

The prefeπed fitness function for the present embodiments of die present invention is an einpincallv deπved fitness function defining die obstacle avoiding task has a pain and a pleasure part The negative contnbulion to fiuiess, called pain, is simply the sum of all proximity sensor values The closer die robot's sensors are to an object, die more pam it expenences In order to keep d e robot from standing still or gyrating it has a positive contribution to fitness, called pleasure, as well It receives pleasure from going straight and fast Both motor speed values minus die absolute value of dieir difference is dius added to die fiuiess Let s, be tlie values of die proximity sensors ranging from 0 - 1023, where a higher value means closer proximity to an object. Let , and m₂ be die left and right motor speeds resulting from an execution of an individual. The values of m, and m₂ are in the range of zero to 15. The empirical fitness value can dien be expressed more formally as: f = ∑s₁+!8-m_l!+|8-m₂i+im₁-ιn₂i (equ. 3)

The Non-Memory Method

The first mediod of the invention evolves die controlling function (equ. 1) directly, and fiuiess is calculated from die cuπent event. In die preferred embodiment of die non-memory method, the fitting algorithm used is symbolic regression using CGPS and die evolved programs arc true functions, no side-effects arc allowed. The learning algoridim had a small population size, typically less dian 50 individuals. The individuals use die eight values from die sensors as inputs and produce two output values which are transmitted to die robot as motor speeds. Each individual program did diis manipulation independent of die odiers, and dius stood for an individual behavior of die robot when it was invoked to control die motors. Table 1 gives a summary of die problem and its parameters. The modules of die learning system and die execution cycle of die GP system are illustrated in

FIGs. 23 and 24 respectively. In bodi mediods, each individual is tested against a different real-time fiuiess case. This could result in "unfair" comparison where individuals have to maneuver in situations with very different possible outcomes. However, experiments show diat over time this probabilistic sampling will even out the random effects in learning, and a set of good solutions survive.

Table

Objective Obstacle avoiding behavior in real-time

Teπninal set Integers in the range 0-8192

Function set ADD, SUB, MUL, SHL, SHR, XOR, OR, AND

Raw and standardized fitness Pleasure subtracted from pain value

desired value

Wrapper None

P;ιrameters

Maximum population size 50

Crossover Prob 90%

Mutation Prob 5%

Selection Tournament Selection

Termination critena None

Maximum number of generations None

Maximum number of nodes 256 (1024)

TABLE 1

Results Using Non-Memory Mediod

Using die non-memory method, die robot shows exploratory behavior from the first moment This is a result of die diversity in behavior residing in the first generation of programs which has been generated randomly Naturally, die behavior is erratic at the outset of a run Dunng die first minutes, die robot keeps colliding widi different objects, but as time goes on die collisions become more and more infrequent The first intelligent behavior usually emerging is some kind of backing up after a collision Then die robot gradually learns to steer away in an increasingly more sophisticated manner After about 40-60 minutes, or 120-180 generation equivalents, die robot has learned to avoid obstacles in die rectangular environment almost completely. It has learned to associate die values from die sensors widi dieir respective location on die robot and to send correct motor commands. In diis way die robot is able, for instance, to back out of a corner or turn away from an obstacle at its side. Tendencies toward adoption of a special padi in order to avoid as many obstacles as possible can also be observed. The number of collisions per minute diminishes as die robot learns and die population becomes dominated by good control strategies.

Despite only processing results for less dian 0.1% of the CPU time, die mediod competes well with odier paradigms. It is for instance 100 times faster than a related evolutionary approach evolving Neural Network controllers on the same robot platform.

The moving robot gives the impression of displaying a very complex behavior. Its behavior resembles diat of a bug or an ant exploring an environment, with small and irregular moves around die objects.

Memory Based Mediod The memory -based embodiment of die present invention consists of two separate processes or units. A computing or planning unit 101 communicates widi inputs (sensors and motors) as well as storing events in a memory buffer. A computer model unit 103 is continuously learning and inducing or evolving a model of die world consistent widi t e entries in a memory buffer.

The foπner process is called die planning process because it is involved in deciding what action to perform given a certain model of die world. The latter process is called the learning process because it consists of trying to derive a model (in die form of a function) from memory data.

Tlie present invention, in its embodiment as a memory based COΠUΌI system as illustrated in FIG. 25 includes six major components.

1. The robot 100 widi sensors and actuators. 2. A memory buffer 102 which stores event vectors representing events in die past.

3. An evolution unit in die form of a GP system 104 which evolves a model of die world diat fits die information of the event vectors.

4. A fitness calculation module 106 which calculates an empirical fitness.

5. A currently best induced individual computer model 108. 6. A search module 110 at determines die best action given die currently best world model.

The Planning Process

The main execution cycle of die planning process is illustrated in FIG. 27, and has several similarities with die execution cycle of die simple genetic control architecture. The planning unit 101 has actual communication widi die robot 100 and decides what action should be performed next. It accesses die best model 108 of die world supplied by die learning unit 103. The planning unit 101 has iree main objectives. It communicates widi die robot 100, finds a feasible action, and stores die resulting event in the memory buffer 102.

As illustrated in FIG. 27. die process starts widi reading all eight infrared proximity sensors. These values are used to instantiate die corresponding variables in die currently best world model 108. The next objective is to find a favorable action given the current sensor values. In die obstacle avoiding task used here die possible actions are 16 different motor speeds for each of die two motors. Each motor has 8 forward speeds. 7 backward speeds, and a zero speed. Combining all alternatives of the two motors, there are 256 different actions altogedier to choose from. This comparatively small figure means mat we can easily afford to search dirough all possible actions while die world model 108 provides a predicted fitness for each of them. The induced model in die form of a computer program 1 8 from die learning unit 103 can dius be seen as a simulation of die environment consistent widi past experiences, where die robot 100 can simulate different actions. The action which gives die best fitness is remembered and sent as motor speeds to die robot 100. If an autonomous agent (e.g. robot 100) had so many possible atomic actions mat an exhaustive search would be infeasible, we could use a heuristic search mediod to find an action that gives good predicted fiuiess. We would, for instance, have anodier genetic programming system evolving a stepwise plan which optimizes fiuiess according to die currently best world model.

In order to get feed-back from die environment, the planning unit 101 has to sleep and await die result of die chosen action. The planning unit 101 sleeps 300 ms while die robot 100 performs the movement defined by die motors speeds. This time is an approximate minimum in order to get usable feed-back from changes in the sensor values in the present example. Thus, die main operation of the planning unit 103 is die sleeping period waiting for feedback from the environment and it, dierefore, consumes less than 0.1% of the total CPU time of the system. After die sleeping period the sensor values are read again. These new values are used to compute a new empirical fitness value using (equ. 3). This fiuiess value is stored, togedier id die earlier sensor values and die motor speeds as an event vector. The event vector consists of 11 numbers; die eight sensor values, die two motor speeds, and die resulting calculated (empirical) fitness. This vector represents what the agent experienced, what it did, and what tlie results were of its action. The memory buffer stores 50 of diese events and shifts out old memories according to a predetermined schema.

It is men die responsibility of the learning unit 103 to evolve a program that simulates die environment as good and as consistent as possible widi respect to die events in die memory buffer 102. As will be seen below, Uiis can be done by a straight-forward application of symbolic regression dirough genetic programming. The Learning Process

The objective of die learning unit 103 is to find a function or a program which will calculate die predicted fitness of an action (set of motor speed outputs), given the initial conditions in form of the sensor values: f(S|.s-,,s,,s₄,s,,s₆,s₇,s₈,ni|,m₂) = predicted fitness (equ. 4)

In most cases an additional state variable is used as part of die program. This is simply a memory cell allowing die possibility to use side-effects in evolved solutions. This feature significantly improves performance in die complex environment used diroughout these experiments: f(s-,s-,.s_;,,s.,,s.-,s₆,s₇,s_1<.m.,nι₂statc,) = (predicted fitncss.state_t+. } (equ. 5) Each event vector stores an instance of die values; die inputs (sensor values), outputs (motor speeds) and die resulting empirical fitness.

FIG. 26 illustrates the interactions between die GP system 104 and die memory buffer 102 in die learning process.

Giviniz the System a Childhood A simple approach to managing die memory buffer 102 when all 50 places have been filled is to shift out die oldest memories as die new entries come in. However, it was deteπnined that die system dien forgot important early experiences. Early mistakes, made before a good strategy was found, are crucial to remember in order to not evolve world models diat permit the same mistakes to be made again. Therefore, die robot is given a childhood - an initial period whose memories were not so easily forgotten. The childhood also reduces die likelihood diat die system displays a strategy sometimes seen even in humans - it would only perform actions which confirmed its current (limited) world model.

Anodier important factor for successfully inducing an efficient world model is to have a stimulating childliood. It is important to have a wide set of experiences to draw conclusions from. Noise is dierefore added to the behavior in die childliood to avoid stereotvpic behavior very early in the first seconds of die system's execution. As long as experiences are too few to allow for a meaningful model of die world, diis noise is needed to assure enough diversity in early experiences. The childliood of die system is defined as die time before die memory buffer is filled which takes about 20 seconds.

Results

The memory based system quickly leams the obstacle avoiding task in most individual experiments. It normally takes only a few minutes before die robot displays a successful obstacle avoiding behavior. The obvious reason for die speed up using memory can be identified in die flowchart of the algoriUims.

In die second (memory) mediod, diere is no "sleeping" period which means mat die genetic programming system can run at die full speed possible for die CPU. This results in a speed up of more than 2000 times in the GP system. On die odier hand, diere is now a more complex task to learn. Instead of evolving an ad-hoc strategy for steenng the robot, the system now has to evolve a complete model of relationships between die eight input variables, die two action variables and the fiuiess

The population size is preferably increased from 50 individuals to 10.000 to ensure robust learning in die memory -based system as illustrated in TABLE 2 The system still has to wait for die robot 5 100 to collect enough memory events to draw some meaningful conclusions Yet die speed-up widi memory exceeds a factor of 40 which makes it possible for die system to learn a successful behavior in less 1 5 minutes on average This means diat die behavior emerges 4000 times faster dian in similar approaches

Ob|cctιvc Symbolic regression of environment simulation

Terminal set Integers in the range 0-8192 0 Function set ADD. SUB, MUL, SHL. SHR. XOR, OR, AND

Raw and standardized fitness The sun taken over 50 fitness cases of the absolute value of difference between die actual and desired value predicting die fiuiess

Wrapper None

Parameters

Maximum population size 10000 5 Crossover Prob 90%

Mutation Prob 5%

Selection Tournament Selection

Termination criteria None

Maximum number of generations None

Maximum number of nodes 256 (1024)

TABLE 2 The behavior of die robot is very different between the two systems (memory and non-memory) discussed here The system without memory behaves in a very complex way and gives die impression of a small bug whicli randomly runs around avoiding obstacles, but widi little overall strategy 5 The memory system, on die odier hand, displays a set of very "thought tlirough" behaviors The robot always displays a clear strategy and travels in straight lines or smoodi curves Some of die behaviors evolved show an almost perfect solution to the current task and fitness function

The robot usually demonstrates a limited set of strategies during evolution in our experiments Some of die emerging intelligent strategies might be described as belonging to different behavioral classes (ordered according to increasing success):

1. The straight and fast strategy: This is die simplest "intelligent" behavior. The induction process has only seen die pattern arising from the pleasure part of the fitness function. The model of die robot and its environment thus only contains tlie relationship expressing diat going straight and fast is good. The robot consequently heads into die nearest wall and continues to stand diere spinning its wheels. This strategy sometimes emerges right after the childliood when die noise is removed and die system is solely controlled by inferences from die induced model.

2. The turning behavior: The second simplest su^*ategy is based on die experience dial turning often improves fitness. The robot starts spinning around its own axis and does avoid all obstacles but also ignores die pleasure part of the fitness rewarding it for going straight and fast.

3. The dancing strategy: This strategy uses die state information in die model and navigates to die open space where it starts to move in an irregular circular padi avoiding obstacles. Most of die time die robot moves around keeping a distance to die obstacles big enough to avoid any reaction from its sensors. If this strategy worked in all cases it would be nearly perfect because it keeps obstacles out of reach of the sensors and die robot is totally unexposed to pain. In most cases, however, die robot wanders off its padi and conies too close to an obstacle where it dien is unable to cope with die new situation and experiences collisions.

4. The backing-up su^*ategy: This is die first effective tecl ique diat allows die robot to avoid obstacles while moving around to some extent. Tlie path the robot travels is, however, very limited and it is not a preferred solution.

5. The bouncing strategy. Here die robot gradually turns away from an obstacle as it approaches it. It looks as if the robot bounces like a ball at something invisible close to die obstacle. This behavior gives a minimum speed change in die robot's padi. 6. The perfect or nearly perfect strategy: The robot uses the large free space in die middle of die training environment to go straight and fast, optimizing die pleasure part of the fitness. As soon as die robot senses an object it quickly turns 180 degrees on die spot and continues going straight and fast. This strategy also involves state information because turning 180 degrees takes several events in the robot's perception, and diat cannot be achieved without states. Most of the experiments displayed a simple behavior very early after die childliood, just to realize a more successful pattern a few seconds later and change its strategy correspondingly The change in strategy was always accomplished by a new best individual and fitness value displayed by die GP algoridim.

TABLE 3 illustrates die results of 10 evaluation experiments with e memory based system. The results were produced by timing die robot's behavior in 10 consecutive experiments. In each experiment the robot was watched for 20 minutes before the experiment was terminated. Each time e behavior changed was noted. The table gives the number of the experiment, the strategy displayed when die experiment was terminated and die time when mis strategy first appeared. It is not completely evident what really constitutes an autonomous agent Some would argue diat die autonomy is a property of die controlling algondim while others would argue that physical autonomy is needed

In order to try die ful autonomous paradigm, a special version of die svstem was ported to die 5 micro-controller

Run number Result Time (minutes)

1 perfect 1 5

2 perfect 1 5

•* > perfect 0 5

] () 4 perfect 3 0 perfect 2 0

6 perfect 2 0

7 perfect 1 5

8 perfect 2 0

15 9 perfect 1 0

1 dancing 0 5

TABLE 3 It is possible to download diese systems via die seπal cable to die robot With the accumulators switched on die robot can men be disconnected from die workstation and can run fully autonomous The 0 Motorola 683 1 micro-controller tiicn runs the complete GP learning system

As mentioned earlier, die micro controller has 256 KB of RAM memory The kernel of die GP svstem occupies 32 KB, and each individual 1KB, in the expeπmental setup The complete system without cmoπ consists of 50 individuals and occupies 82KB which is well within the limits of the on-board system The more complex system, learning from memory, has to use a smaller population size than the desired 10000 5 individuals This results m less robust behavior with a more frequent convergence to local optima such as displayed by die first strategies in Figure 15

In eidier case, it is demonstrated herewitii mat die compactness of the compiling GP system enables relatively powerful solutions in weak architectures such as those used in embedded control Summary of Autonomous Agent Control Svstem Experimental Results

It has been demonstrated diat a GP system can be used to control an existing robot in a real-time environment with noisy input. The evolved algorithm shows robust performance even if die robot is lifted and placed in a completely different environment or if obstacles are moved around. It is believed diat die robust behavior of die robot partly could be attributed to die built-in generalization capabilities of die genetic programming system.

We have also cross-compiled the GP-system and run it in die same set-up on die micro-controller on board die robot. This demonstrates die applicability of Genetic Programming to control tasks on low-end architectures. The technique could potentially be applied to many one-chip control applications in, for instance, consumer electronics devices.

It has been demonstrated diat die use of memory with a real robot and a GP based control system could speed up learning by a factor of 40. Furthermore, the strategies evolved with the memory based system display a smootiier, less chaotic behavior, undisturbed by die internal dynamics of die GP system. The memory-based system could also be expected to scale up better because training times are not directly related to die dynamics of die agent and its environment, but instead almost completely depend on the difficulty of die induction problem in die application domain. (g\

In summary, die present invention overcomes the drawbacks of die prior art by eliminating all compiling, inteφreting or other steps that are required to convert a high level programming language instruction such as a LISP S-expression into machine code prior to execution or that are required to access Learned Elements or run-time data in data structures.

This makes possible die practical solutions to problems which could not heretofore be solved due to excessive computation time. For example, a solution to a difficult problem can be produced by die present system in hours, whereas a comparable solution might take years using conventional techniques. Various modifications will become possible for those skilled in die art after receiving die teachings of die present disclosure without departing from the scope thereof.

INDUSTRIAL APPLICABILITY

The present invention has utility in computerized learning which can be used to generate solutions to problems in numerous technical areas, and also to control of an autonomous agent such as an industrial robot.

Claims

WE CLAIM:

1. A computer implemented learning method characterized by die steps of:

(a) providing a computer with an indirectly executable program including: a first insunction that points to and designates machine code stored in a 5 memory as data; a second instruction that points to and designates machine code stored in a memory as directly executable code; a tiiird instruction diat alters machine code pointed to by the first instruction; and o a fourth instruction diat executes machine code pointed to by the second instruction and

(b) controlling die computer to execute die program which performs the steps of:

(b 1 ) creating and storing a machine code entity including a directly executable instruction in a memory; 5 (b2) executing the second instruction to point to the entity;

(b3) executing the fourth instruction using input data to produce a result;

(b4) evaluating the result;

(b5) executing die first instruction to point to the entity;

(b6) executing the third instruction to alter the entity to include a different 0 directly executable instruction; and

(b7) repeating steps (b3) to (b6) until an end criterion is reached.

2. A method as in claim 1, in which: the first instruction points to and designates machine code stored in a memory as data by casting a pointer tiiereto as a data pointer; and 5 die second insunction points to and designates machine code stored in a memory as a directly executable code by casting a pointer tiiereto as a function pointer.

3. A method as in claim 1, in which: die first instruction points to and designates machine code stored in a memory as data by creating a data pointer thereto; and 0 die second instruction points to and designates machine code stored in a memory as directly executable code by creating a function pointer thereto.

4. A method as in claim 1, in which: step (bl) comprises creating the entity as comprising a header and a footer; and step (b6) comprises preventing the header and footer from being altered. 5. A method as in claim 1, in which: step (bl) comprises creating the entity as comprising a return instruction; and step (b6) comprises preventing the retum instruction from being altered.

6. A method as in claim 1 , in which die learning algorithm in step (b4) and the alteration algorithm in step (b6) in combination comprise a genetic algorithm.

7. A metiiod as in claim 1, in which step (b l ) comprises creating the entity as comprising a plurality of directly executable instructions.

8. A mediod as in claim 7, in which step (bl) comprises randomly selecting die directly executable instructions from a set of directly executable instructions.

9. A mediod as in claim 7, in which step (b6) comprises genetically altering the directly executable instructions. 10. A mediod as in claim 7, in which: one of the directly executable instructions comprises an operator and an operand; and step (b6) comprises genetically mutating tlic operand.

11. A mediod as in claim 7, in which two of die directly executable instructions comprise operators respectively; and step (b6) comprises genetically crossing die operators.

12. A method as in claim 1, in which: step (bl) comprises randomly selecting the directly executable instruction from a set of directly executable instructions.

13. A method as in claim 1, in which step (b6) comprises genetically altering the directly executable instruction.

14. A metiiod as in claim 1, in which: tlie directly executable instruction comprises an operator and an operand; and step (b6) comprises genetically mutating the operand.

15. A computer implemented learning mediod characterized by the steps of: (a) providing a computer widi an indirectly executable program; and

(b) controlling tlie computer to execute the program which performs tlie steps of:

(bl) creating a directly executable machine code entity including a directly executable program instruction and a directly executable retum instruction;

(b2) executing the entity using input data to produce a result; (b3) evaluating the result;

(b4) altering only tlic program instruction to include a different directly executable program instruction; and

(b6) repeating steps (b2) to (b4) until an end criterion is reached.

16. A metiiod as in claim 15, in which step (b4) comprises preventing tlie return instruction from being altered.

17. A metiiod as in claim 15, in which step (b4) comprises allowing the return instruction to be altered, and then restoring die return instruction to its original state.

18. A metiiod as in claim 15, in which step (b4) comprises separating die program instmction from die entity, altering die program insunction to include die different program instmction, and returning the different program instmction to die entity.

19. A metiiod of manipulating and executing computer machine code, comprising die steps of:

(a) providing a computer with an indirectly executable program including: a first instmction that points to and designates machine code stored in a memory as data; a second instmction diat points to and designates machine code stored in a memory as directly executable code; a third instmction that alters machine code pointed to by die first insunction; and a fourth insunction diat executes machine code pointed to by the second instmction; and (b) controlling die computer to execute die program which performs die steps of:

(bl) creating and storing a machine code entity including a directly executable instmction in a memory;

(b2) executing the second instmction to point to the entity; (b3) executing the fourth insunction to execute the entity; (b4) executing the first instmction to point to the entity; and

(b5) executing the tiiird instmction to alter the entity to include a different directly executable instmction.

20. A method as in claim 19, in which: step (bl) comprises creating the entity as including a directly executable program instmction and a directly executable return instmction; and step (b5) comprises preventing the retum instmction from being altered.

21. A computer learning system characterized by comprising: a memory for storing an indirectly executable computer program including: a first instmction that points to and designates machine code stored in the memory as data; a second instruction uiat points to and designates machine code stored in die memory as directly executable machine code; a third instinction tiiat alters machine code pointed to by the first instmction; and a fourth instmction that executes machine code pointed to by the second instmction; and a processor for executing die program; die processor, memory and program operating in combination for performing die steps of

(a) creating and stoπng a machine code entity including a directly executable instruction in die memory, (b) executing tiie second insunction to point to the entity,

(c) executing the instmction using input data to produce a result,

(d) evaluating die result,

(e) executing die first instmction to point to die enti ,

(0 executing the third instmction to alter the entity using an alteration algonthm to include a different directly executable instmction, and

(g) repeating steps (b) to (f) until an end cnteπon is reached

22 A system as in claim 21, in which the first instruction points to and designates machine code stored m die memory as data b_\ casting a pointer tiiereto as a data pointer, and die second insunction points to and designates machine code stored m die memory as directly executable code by casting a pointer tiiereto as a function pointer

23 A system as in claim 21, in which tlie first instmction points to and designates machine code stored in the memorv as data by creating a data pointer tiiereto, and die second instmction points to and designates machine code stored in die memorv as directly executable code by creating a function pointer thereto

24 A system as in claim 21, which step (a) compnses creating the entity as compnsmg a header and a footer, and step (0 compnses preventing die header and footer from being altered 25 A svstem as in claim 21, in which step (a) compπses creating the entity as compnsmg a retum insunction, and step (f) compnses preventing die retum instruction from being altered 26 A system as in claim 21, in which die learning algonthm in step (d) and die alteration algonthm in step (f) in combination compnse a genetic algonthm 27 A svstem as in claim 21, in which step (a) compπses creating the entity as compnsmg a plurality of directly executable instructions

28 A system as in claim 27, in which step (a) compnses randomly selecting the directly executable instructions from a set of directly executable instructions

29 A system as in claim 27, in which step (0 compπses genetically altenng die directly executable instructions

30 A system as in claim 27, in which one of the directly executable instructions compnses an operator and an operand, and step (0 compnses genetically mutating the operand 31 A svstem as in claim 27, in which two of the directly executable instructions compnse operators respectively and step (f) compnses genetically crossing die operators 32 A system as in claim 21 in which step (a) compπses randomly selecting die directly executable instmction from a set of directly executable instructions

33 A system as in claim 21, in which step (f) compnses genetically altenng the directly executable instmction

34 A s stem as in claim 21 , in which die directh executable instmction compnses an operator and an operand, and step (0 compπses genetically mutating the operand

3 A system as in claim 21 , in which the memoπ compπses a mam memorv and a processor memorv that is part of the processor and step (a) compπses stoπng the entity in the processor memoπ

36 A system as in claim 35, compnsmg an integrated circuit chip in which die processor and die processor memory are foπned on die chip

37 A system as in claim 36, which at least a portion of the program is stored processor mcmoπ 38 A svstem as in claim 37, in which die processor memory compπses a non-volatile memory section, and said at least a portion of die program is stored in the non-volatile memoπ section 39 A mediod of creating and executing machine code using a digital computer which mcludes a processor and a memoπ' characteπzed by die steps of (a) selecting a directly executable machine code program instmction,

(b) determining a binaπ number whicli represents die program instmction to die processor

(c) selecting a location in d e memory,

(d) creating a directly executable machme code entity which mcludes die binary number, (e) stoπng die entity as data m said location,

(0 providing an indirectly executable insunction which causes die processor to directly execute machine code stored in said location, and

(g) controlling die processor to process the indirectly executable instmction, resulting in die processor directly executing die entity as including the program inst ction 40 A method as in claim 39, in which step (0 compπses providing die indirectly executable instmction such that said processing thereof in step (g) includes compiling

41 A method as in claim 39, in which step (f) compnses providmg die indirectly executable instruction such dial said processing tiiereof in step (g) includes interpreting

42 A metiiod as in claim 39, in which step (f) compnses creating die indirectly executable instmction such diat said processing tiiereof in step (g) includes assembling

43 A metiiod as in claim 39, furdier compnsmg die step of (h) repeating step (g) at least one time

44 A metiiod as in claim 39 further compnsmg the step of

(h) evaluating a result produced by executing the program insunction in step (g)

45 A mediod as in claim 39, further compnsmg tiie step, performed after step (g), of (h) altenng the entity as data such diat die bmaπ number is changed to represent a different directly executable machine code program insunction to die processor

46 A method as in claim 45, further compnsmg die step, performed after step (h) of

(i) controlling die processor to process die indirectly executable instmction, resulting in the processor directly executing die entity as including die program insunction

47 A mediod as in claim 46, further compnsmg die step of (j) evaluating a result produced bv executing die program instmction in step (I)

48 A metiiod as in claim 39, further compnsmg the steps, perfoπned after step (g), of (h) copying die entity to produce a copied entity, and

(l) alteπng die copied entity as data such that the binary number is changed to represent a different directly executable machine code program insunction to the processor 49 A mediod as in claim 39, further compnsmg the steps of

(h) evaluating a result produced by executing die program insunction in step (g), (I) altenng the entity as data such diat the bmaπ number is changed to represent a different directly executable machine code program insunction to die processor,

(l) controlling tlie processor to process die indirectly executable insunction, resulting m die processor directly executing die entity as including tlie program instmction, and

(k) evaluating a result produced by executing die program instmction in step (j)

50 A mediod as in claim 49, further compnsmg die step of

(1) companng die evaluation of step (k) with die evaluation of step (h)

51 A method as in claim 50, furdier compπsmg die step of (m) selecting a subsequent action m accordance with said compaπson of step (1)

52 A mediod as in claim 39, in which step (d) compnses creatmg the entity to furdier include a retum inst ction

53 A method as in claim 52, further compnsmg die steps, performed after step (g), of (h) alteπng die entity as data such diat the binaπ number is changed to represent a different directly executable machine code program insunction to the processor, and

(I) preventing die retum instinction from being altered while performing step (h)

54 A method as in claim 52, further compnsmg the steps, performed after step (g), of (li) altering die entity as data such that the binary number is changed to represent a different directly executable machine code program instmction to the processor, and die retum instinction is allowed to be altered: and

(i) restoring die retum insunction to original state after alteration in step (h) 55 A mediod as in claim 39, in which step (d) comprises creating me entity to further include at least one of a header and a footer.

56 A method as in claim 55, furdier comprising die steps, perfoπned after step (g), of (h) altering die entity as data such that die binary number is changed to represent a different directly executable machine code program instmction to die processor: and (i) preventing said at least one of a header and a footer from being altered while performing step (h)

57 A method as in claim 55, further comprising die steps, performed after step (g), of (h) altering die entity as data such diat die binary number is changed to represent a different directly executable machine code program inst ction to die processor, and said at least one of a header and a footer is allowed to be altered: and

(i) restoring said at least one of a header and a footer to original state after alteration in step (h)

58 A mediod as in claim 55, further comprising the steps, performed after step (g), of (h) separating die bin-u-y number from the entity, (i) altering tiie binary number to produce a different binary number which represents a different directly executable machine code program insunction to die processor; and

(j) replacing die binary number with die different binary number in die entity. 59. A mediod as in claim 39, in which step (d) comprises creating die program instinction to include ran time data 60 A method as in claim 59, further comprising the step, performed after step (g), of

(h) altering die entity as data such tiiat die mn time data is changed 61. A mediod as in claim 60, further comprising the step, performed after step (h), of

(i) controlling tiie processor to process the indirectly executable insunction, resulting in the processor directly executing the entity as including the program instinction. 62 A metiiod as in claim 61, in which step (h) further comprises altering the entity as data such that the binary number is changed to represent a different directly executable machine code program instmction to the processor

63 A metiiod as in claim 61, further comprising die step of.

(j) evaluating a result produced by executing die program insunction in step (i) 64 A mediod as in claim 59, further comprising the steps, performed after step (g). of

(h) copying the entity to produce a copied entity; and (i) altering the copied entity as data such that the mn time data is changed 65 A method as in claim 59, further comprising die steps of:

(h) evaluating a result produced by executing die program insunction in step (g); (i) altering tiie entity as data such diat the mn time data is changed; (j) controlling die processor to process the indirectly executable instmction, resulting in die processor directly executing the entity as including die program instinction; and

(k) evaluating a result produced by executing die program instinction in step (j)

66 A mediod as in claim 65, further comprising the step of:

(1) comparing die evaluation of step (k) with the evaluation of step (h).

67 A metiiod as in claim 66, further comprising die step of: (m) selecting a subsequent action in accordance widi said comparison of step (1)

68 A mediod as in claim 39, in whicli step (a) comprises selecting the program instmction from a set of directly executable program instructions.

69 A method as in claim 39, further comprising die step, perfoπned before step (a), of selecting a subset of program instructions from a set of directly executable program instructions; and step (a) comprises selecting the program instmction from said subset.

70 A metiiod as in claim 39, in which: step (d) comprises creating die program instmction to include run time data; and the mediod further comprises tlie step of: (h) repeating step (g) for different input values of tiie mn time data

71. A mediod as in claim 70, further comprising the step of:

(i) determining a value of die run time data which produces a predetermined result.

72. A mediod as in claim 70, in which step (h) further comprises repeating step (g) for different program instructions. 73. A metiiod as in claim 39, in which die method furdier comprises die step, perfonned after step (g), of:

(h) creating a representation of tlie entity in a selected computer programming language

74. A method as in claim 73, in which step (h) comprises decompiling the entity.

75. A method as in claim 39, in which: step (d) comprises creating the program instmction to include mn time data; and tiie metiiod further comprises the steps of.

(h) determining if execution of step (g) produced a predetermined result; and

(i) if step (g) produced said predetermined result, creating a representation of the entity including said input value of die mn time data in a selected computer programming language. 76. A method as in claim 75, further comprising tiie step of:

(j) repeating step (g) for different input values of tiie mn time data;

(k) deteπ ining a value of the run time data whicli produced a predetermined result; and