USRE27703E - Configuration control in multiprocessors - Google Patents

Abstract

THE PRESENT MULTIPROCESSOR APPARATUS INCLUDES, IN EFFECT, A DISTRIBUTED INTERCONNECTION SYSTEM SUCH THAT FAILURE OF A PORTION PF THE INTERCONNECTION SYSTEM DOES NOT COMPLETELY DISABLE THE APPARATUS. EACH ELEMENT-OF A PLURALITY OF COMPUTING ELEMENTS, A PLURALITY OF STORAGE ELEMENTS, AND A PLURALITY OF OTHER INFORMATION PROCESSING ELEMENTS OF THE TOTAL MULTIPROCESSOR APPARARUS-IS EQUIPPED WITH AN INDIVIDUAL CONFIGURATION CONTROL REGGISTER FOR SELECTIVELY CONTROLLING THE FLOW OF INFORMATION BETWEEN THE RESPECTIVE ELEMENTS AND OTHER ELEMENTS OF THE APPARATUS. FOR CONTROLLING THE INTERCONNECTION SYSTEM REPRESENTED BY THE CONFIGURATION CONTROL REGISTERS, WITH THE REDUNDANCY NECESSARY FOR RELIABLE ''''FAIL-SAFE'''' OPERATION OF THE SYSTEM, EACH OF A PLURALITY OF COMPUTING ELEMENTS IS PROVIDED WITH MEANS FOR INDEPENDENTY DEVELOPING, SELECTING AND CONDITIONING SIGNALS AND WITH MEANS FOR BROADCASTING THESE SIGNALS TO ALL ELEMENTS OF THE APPARATUS, INCLUDING THE ORIGINATING ELEMENT. THE SELECTING SIGNALS ARE ULTIZED AT THE RECEIVING ELEMENTS TO SELECTIVELY LIMIT APPLICATION OF THE CONDITIONING SIGNALS ONLY TO CONFIGURATION CONTROL REGISTERS OF ELEMENTS DESIGNATED BY THE SELECTING SIGNALS, SUCH ELEMENTS HAVING BEEN PREDETERMINED BY THE ORIGINATING COMPUTING ELEMENT. THE CONDITIONING SIGNALS SERVE TO CONDITION THE CONTROL REGISTERS OF THE SELECTED ELEMENTS TO DESIRED STATES OF INFORMATION FLOW CONTROL ALSO PREDETERMINED BY THE ORIGINATING COMPUTING ELEMENT.

Description

July 24, 1973 s, STAFFORD EI'AL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2. 1965 12 Sheets-Sheet 1 mhzmiu fi 02:31:90 muff-.0

WNW

328: 4 Q: 2 a 5 L 35 225E Jo EEO Jo 6 why-4a mm IFO 3: Pzwzud 052K200 INVENTORS THDMAS S. STAFFDkD DONALD L. BURNSTINE GERARD T. PAUL JOHN R, ROGASM ATTDRNEY w bi 32% W 2 JQ; Q

mt K J: mofilzo July 24, 1973 "r. s. STAFFORD ETAL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 3. 1965 12 Sheets-Sheet 5 FIG. 6 COR CE smus 90 90 E S60 STORAGE COMPUTER (5) (SC) P ELEMENT P SE ELEMENTP 100E P (SE) (CE) fi/i/ V 121254 H23456782EHOH|212543////(1254 CCR SE 7'// A s so P P CE P IOCE P I M24234 1 2/12545 1254 FIG. 8 CCR/IOCE /l s 50 P 5;: P SE cE P P CCR/TCU July 24, 1973 T. s. STAFFORD ErAL Re. 27,703

CONFIGURATION CDNTROL IN MULTIPROCESSORS Original Filed June 2, 1965 12 Sheets-Sheet 7 ELEMENT FIG H RECONFIG. RESPONSE llFimiw CM/CEg/ AND TEST SWITCH OFF I0 P NF FQ 4 FROM VEi W SCON(T0 LINE 120,

11 PROGRAM CONTROLS\ TEST,

121 (FIGJO g0 UNF OFF MATCHL AND \NTE RRU PT CONTROLS ELC FROM OTHER ELEMENTS July 24, 1973 "r. s. STAFFORD ETAL Re. 27,703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2, 1965 12 Sheets-Sheet 8 aecomnc. RESPONSE {SE v H v 1 0R M50 T FIG. 12

I 520 OR cm/ce L 30 1 SM/CE1IBIT s5 I CCR/SE1/BiT s0 M mo v ANDS 0/65 SM/CEg/BIT SE1 6 5 36 CCR/SEq/BIT 50 AND ANDS 7 502 50s SMICE3/EIT as, as CCR/SE1/BHSC5 AND AN DS swam/an s5 "N306 307 CCR/SEq/BIT SE4 SE1 TEST LATCH OFF AND SE4 TEST LATCH 0N SE1 KEYS 1 CR8 cue/s5 s so P P c? P 10 P :[2 M21514 A1 //2 /1i2I5{4l3p 1M5 [L lgemsr 4 cmcun AND an TEST swncu on SE TET wr F L ccR/sE, OUTPUT T0 SE1 RECEIVING ems SE1 TEST LATCH ION OFF July 24, 1913 T. S. STAFFORD ET AL CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2, 1965 soon m 4 I I L! RECUNFIC. RESPONSE ([01 2 12 Sheets-Shea t 9 R F l G. i 3

OR CHI/(3E1 SM/CE an 10 1 1 I 36 CCR 10 mm 50 s AND ANDS W CE 2 SM 05 an 6 56 CCR 10 mm AND ANDS W05 5 F.- sn/ce /en 10 8 4 CUR/.[DHBII 505 AND V ANDS f as SH/CE4/BIT I01 CCR/[01/BlT 504 AND 7 ANDS 10, TEST mow OFF 10 TEST men ON 56 1o 1 ms I 0 RS "56* T 00 my l- COR/l0 1 s 50 P SE P SF. if P a 1 z 1 i 213 [4 1 1M5 14 5|sl1 [a 2 9W1 M1 z {5M5 97 4 101 32 4 BYPASS 4 51 52 cmcun 5 1 T I won ON AND 0 E6 8 10 15s? swncn OFF I CUR/[0g OUTPUT 10: TEST man {on on y 973 T. s. STAFFORD E Re. 27,703

CONFIGURATION CONTROL IN MIILTIPROCESSORS Original Filed June 2, 1965 12 Sheets-Sheet 1O FIG. 14 INTERNAL OPERATIONS PERFORMED BY CE,UNDER SE PRG CONTROL, IN CONNECTION WITH RECONFIGURATION INITIAI'E PROGRAM INTERRUPTION IN RESPONSE To ELC 0R SUPERVISOR-T CALL INSTRUCTION ENCOUNTERED TR CURRENT PROGRAM COLLECT CURRENT PROGRAM STATUS WORD IPSWI FROM LOCAL BUFFER 226 REGISTERS AND STORE AT UNIQUE LOCATION IN A PREDETERMINED SE FETcR RER PSW FROM ANOTHER UNIQUE LOCATION IN sE ACCORDING T0 CLASS OF INTERRUPTION DISTRIBUTE REw PSW T0 LOCAL BUFFER REGISTERS AND PROCEED TO FIRST INSTRUCTION OF NEW PROGRAM AT SE ADDRESS REFERERcER IN 228 NEW Psw 229\ INVESTIGATE ELEMENT ccR STATES VIA STATUS TABLE sToREn AT PREDETERMINED ADDRESSES IN SE INCOMPLETE PROCEED" CONFIGURATION COMPLETE EXIT FETCH scoR INSTRUCTION: RI R2 (240 FROM SELECTED ADDRESS IN SE TO LOCAL BUFFERS; VERIFY SUPV. MODE BIT IN PSW; CHECK CM IN RI FOR PERMISSIBLE SC BIT CONFIGURATION; CHECK OWN OCR/CE STATUS BITS AND OWN CE TEST SWITCH; SET SCON LATCH I SIMULTANEOUSLY TRANSFER CM FROM R1 AND SM/BIT CE FROM R2 TO OWN CCR RECEIVING GATES IF OWN STATUS BITS ARE OD OR II, AND TO ALL OTHER CCR RECEIVING GATES IF, IN

ADDITION, OWN CE TEST SWITCH IS OFF I I REsET SM BITS IN R2 IN ACCORDANCE WITH ELEMENT RECONFICURATION RESPONSE U k UPDATE STATUS BITS DE RESPONDING ELEMENTS IN STATUS 233 TABLE July 24, 1973 T. s. STAFFORD ETAL 27.703

CONFIGURATION CONTROL IN MULTIPROCESSORS Original Filed June 2, 1.965 12 Sheets-Sheet 1:

FIG. 16

ELEMENT CCR/BIT s CCR/BIT s PAN; 1 1

TCU1 1 1 United States Patent Oflice Re. 27,703 Reissuecl July 24, 1973 27,703 CONFIGURATION CONTROL IN MULTIPROCESSORS Thomas S. Stafford, Wappingers Falls, Donald C. Bumstine, Hyde Park, Gerard T. Paul, Poughkeepsie, and John R. Rogaski, Woodstock, N.Y., by International Business Machines Corp., Armonk. N.Y., assignee Original No. 3,386,082, dated May 28, 1968, Ser. No. 460,776, June 2, 1965. Application for reissue Sept. 5, 1972, Ser. No. 288,182

Int. Cl. G061 9/00, 11/00 US. Cl. 340-1725 18 Claims Matter enclosed in heavy brackets appears in the original patent but forms no part of this reissue spec fication; matter printed in italics indicates the additions made by reissue.

ABSTRACT OF THE DISCLOSURE The present multiprocessor apparatus includes, in eflect, a distributed interconnection system such that failure of a portion of the interconnection system does not completely disable the apparatus. Each element-of a plurality of computing elements, a plurality of storage elements, and a plurality of other information processing elements of the total multiprocessor apparatus-is equipped with an individual Configuration Control Register for selectively controlling the flow of information between the respective element and other elements of the apparatus. For controlling the interconnection system represented by the Configuration Control Registers, with the redundancy necessary for reliable fail-safe operation of the system, each of a plurality of computing elements is provided with means for independently developing, selecting and conditioning signals and with means for broadcasting these signals to all elements of the apparatus, including the originating element. The selecting signals are utilized at the receiving elements to selectively limit application of the conditioning signals only to Configuration Control Registers of elements designated by the selecting signals, such elements having been predetermined by the originating computing element. The conditioning signals serve to condition the Control Registers of the selected elements to desired states of information flow control also predetermined by the originating computing element.

The invention hereof pertains to the control of inter connections between independent elements of a multiprocessing complex. In particular, it pertains to a system of connection controls which can be used to reliably maintain continuity of data processing operations and eflicient distribution of element work load assignments within a multiprocessing complex in a variety of problem situations.

In certain real-time data processing situations, for example in the processing of air traflic control data, it is necessary to maintain substantial continuity of operation in order to avoid loss of data when circuit failures or other causes of processing bottlenecks are imminent or have actually occurred. It is conventional in this regard to provide duplicate system elements which may be maintained available, on a standby or other basis, to be switched into an active processing system as a substitute for a failing element or to back up an element which is being overloaded with too many tasks. The weak link in this procedure has been the means used to accomplish the switching in of the back-up element and, if need be, the disconnection of the failing element. Hitherto, the practice has been to provide a centralized arrangement of master switching controls which, in response to signals indicating an element failure, or other condition requiring attention can act to switch in back-up elements so as to reorganize the processing system, or systems, then in operation. One problem associated with this form of control is that the master switching controls themselves are susceptible to failures capable of interrupting operations and causing irreparable loss of both data and data processing time.

It is, of course, possible to provide for manual disconnection of the master controls and the, inter-element switches when a failure occurs. However, when time is of the essence, manual control is inadequate since seconds, or even minutes, might elapse between the time of occurrence of a failure and the disconnection of the master controls, during which time thousands of misrouted data transfers may occur wtih consequent loss of much data and disruption of data processing operations currently in progress.

Accordingly, an object hereof is to provide an improved system for controlling the configuration of interconnections between elements of a multiprocessing complex.

Another object is to provide a system for controlling interconnections between elements of a multiprocessing complex which can be adapted to selectively reorganize interconnections between elements to maintain processing operations within the complex at a prescribed level of efiiciency, continuity and reliability.

Another object is to provide a system for efiiciently controlling interconnections between elements of a multiprocessing complex which can be expected to effectively maintain continuity of data processing operation during ordinary system failures.

Yet another object is to provide a system for etficiently controlling interconnections between elements of a multiprocessing complex which can be expected to reliably maintain continuity of processing operation during malfunctioning of any element of the complex including an element which can exercise supervisory control over all interconnections within the complex.

Another object is to provide a system for efficiently controlling interconnections between elements of a multiprocessing system including check features which preclude erroneous initiation of a change in interconnections or undesirable acquiescence by an element in a proffered change.

A feature of the subject configuration control system resides in the provision, within each of a plurality of computing elements of a multiprocessing complex, of means capable, under prescribed conditions, of effecting changes in the states of receptiveness of any elements of the complex signals transmitted thereto from other elements of the complex. Subject to certain restrictions any computing element can transmit configuration control signals which are selectively received by the other elements of the complex and determine the states of receptiveness of the receiving elements. It is thus unnecessary to disconnect power from a faulty element to render other elements insensitive to it and by appropriately configuring states of receptiveness, the processing tasks of a faulty element can be reassigned to a health standby element, or added to the duties of an actively functioning element. Since such configuring can be accomplished automatically in response to a fault indicating signal it is possible to maintain continuity of processing when a failure occurs. It is even possible to allow a faulty element to continue to transmit signals externally while it is undergoing diagnostic testing or other maintenance procedures without risk of disrupting data processing operations currently in progress. of perhaps equal significance, processing bottlenecks, due to assignment of too many tasks to one computing element can be anticipated and another computing element can be programmed to automatically step in and reconfigure the system so as to relieve the overloaded element while the latter remains free to continue its necessary activities without interruption.

Another feature herein is that internal to each element of the complex there is provided a configuration control register (abbreviated CCR) having output control consections to gates which receive and pass signals transmitted by other elements. Thus, the receptiveness of an element to transmissions of other elements is a function of the present state of its CCR. The inputs of any CCR, subject to the output states of certain bits therein, are accessible to each computing element of the complex. Thus the content of any CCR, including a computing element CCR, may be selectively changed by any one of the :omputing elements.

Another feature is that each element can transmit an :lement check signal (ELC) in response to which the :omputing elements can initiate a reconfiguration program leading to modification of selected CCRs and consequent reorganization of the complex. In response to an ELC those computing elements enabled by bits in their re- ;pective CCRs to receive from the element transmitting :he ELC select one of their number to issue a program of reconfiguration signals. These signals establish new ini'ormation settings in selected CCRs, including, if need Je, the CCR associated with the element issuing the ELC tignal. The issuance of an ELC signal may be conditioned 1pon the results of parity checks, power supply checks, ogic checks, work load checks or checks of any other :lement condition which is sufficiently pertinent to sysem operations to warrant consideration of a reconfiguraion program.

The program instruction executed by a computing elenent to reorganize element system assignments within he complex-i.e. to modify selected CCRs--is referred vo herein as SCON (abbreviation for Set Configuraion"). Programs containing such instructions are strategi- :ally located in a plurality of storage elements of the com lex on a redunant basis so that failure of any one stortge element will not prevent reconfiguration of the com- :lex.

In response to an element check or other condition callng for possible reconfiguration of a system, one of the :nabled computing elements determines the condition of he complex, and if circumstances warrant further action, t retrieves one or more appropriate SCON instructions 'rom an associated storage element and executes them. Jnder the direction of the selected SCON instruction, the :omputing element issues a reconfiguration signal, known B a configuration mask (CM), and an element selection :ignal, or selection mask (SM). The CM signals are seectively accepted as new CCR entries by the elements iesignated by SM.

The foregoing and other objects and features of the nvention will be more fully understood and appreciated lpon consideration of the following description thereof .aken with reference to the accompanying drawings wherein:

FIGS. 1-4, when arranged according to the plan shown 11 FIG. 5, constitute a schematic illustration on a gen- :ral level of a multiprocessing complex including a coniguration control system organized in accordance with be present teachings;

FIGS. 6-9, inclusive, provide diagrammatic illustraions of the format of the configuration mask informaion stored in the configuration control registers (CCRs) tssociated with the different types of elements shown in he complex of FIGS. l-4;

FIG. 10 is a schematic of a representative computing :lement configuration control register (CCR/CB and iccess controls therefor;

FIG. 11 is a schematic drawing illustrating parts of a epresentative computing element CB which can conrol the acquisition and execution of SCON instructions For reorganizing the multiprocessing complex hereof;

FIG. 12 is a schematic illustration of pertinent parts of a representative storage element configuration control register (CCR/SE and associated access controls therefor:

FIG. 13 is a schematic of pertinent parts of a representative input/output control element configuration control register (CCR/IOCE and associated access controls;

FIG. 14 is a schematic flow diagram of a sequence of operations executed internally by a computing element in connection with the handling of a SCON instruction;

FIG. 15 contains a table which defines the operational capabilities of each element with reference to certain status bits (S) contained in the CCR internal to the element;

FIG. 16 illustrates a typical stored status table listing the current status bits of all element CCRs of a complex. This table may be used as a stored reference by a CE in connection with the Instruction Fetching routine to determine its selection of a SCON instruction for execution.

GENERAL DESCRIPTION As shown in FIGS. 1-4, a multiprocessing complex of the type under consideration herein typically may include a plurality of computing elements (CE), shown at 1 and 2 in FIG. 1, and may include a plurality of other specialized elements, such as storage elements (SE), shown at 3 and 4 in FIG. 2, input-output control elements (IOCE or IO) shown at 5 and 6 in FIG. 3, and peripheral adapter elements 7, 8 (FIG. 4).

The adapter elements are each connected to a plurality of peripheral devices such as tape stores, communication terminals, printing devices, and the like. The tape store adapter units are hereinafter designated by the symbol TCU (abbreviation for tape control unit) and other peripheral device adapters are designated by the letters PAM (abbreviation for peripheral adapter module).

As may be inferred from the descriptive terminology used above, CEs perform processing operations on internal programs and information supplied by SEs and IOCEs provide a link for exchanging information between the SES and external input and output devices via the additional bufl'ering afforded by the adapter elements.

Each element is provided with a configuration control register 10 (abbreviated CCR), two sets of receiving gates 11, 12 and other internal components 13 collectively denoted other parts. These other parts may include individual power supplies. To simplify FIGS. 1-4, only the internal organizations of representative elements, CB SE IOCE and TCU are shown. It should be understood that the internal organizations of other elements in each category would be similar to those of the representative elements.

The receiving gates 11 control the introduction of new information settings into the respective CCRs, and the receiving gates 12 control the reception of other information by the other internal circuits 13 of the respective elements. The receiving gates 11 and 12 of each element are controlled by various digit outputs of the associated CCR as indicated by lines 15, 16 and 17. Thus, all incoming information signals proffered to an element can be selectively received or rejected by the element in accordance with the contents of its associated CCR. An exception to this is found in the IOCE elements. Information profiered to an IOCE 5, 6 by a peripheral adapter unit 7, 8 is permitted to bypass the receiving gates 12 controlled by CCR/IOCE. This is because there are other internal receiving gates in each IOCE, not controlled by CCR/IOCE, which nevertheless prevent unauthorized reception of information transmitted from an adapter.

The other parts" 13 of each SE, 3, 4, include a fast access store, such as a magnetic core matrix memory, and access controls for reading and writing information relative thereto.

The other parts 13 of each CE include computing and control circuits, for operating upon stored program instructions and data stored in an SE, for executing SC'ON instructions and selectively issuing configuration control information to the CCRs, and for handling the acknowledging responses of elements intended to have their CCRs reconfigured by such information.

The other parts of each IOCE include one or more channel systems for selectively conveying information bidirectionally between peripheral device adapters (TCU, PAM) and SEs, according to the needs of the CEs.

Data and control signals not directly connected with configuration control are exchanged between a CE and and SE via group bus 20, one of the tributary buses 21 (FIG. 1) or 22 (FIG. 2) feeding into bus 20, and one of the branch buses 23 (FIG. 2) or 24 (FIG. 1) extending from the group bus.

Data and control signals not directly connected with configuration control are handled between CE and IOCE units via group bus 25 (FIGS. 1 and 3), one of the tributary buses 26 (*FIG. I) or 27 (FIG. 3) feeding into group bus 25, and one of the branch buses 28 (FIG. 3) or 29 (FIG. 1) respectively, extending out of the group bus.

Data and control signals not directly connected with configuration control are handled between SE and IOCE units via group bus 35, one of the tributary buses 36 (FIG. 2) or 37 (FIG. 3), and one of the respective branch buses 38 (FIG. 3) or 39 (FIG. 2).

One other path for the handling of data and control signals is that between the IOCE units (FIG. 3) and peripheral adapter units (FIG. 4). It includes paths defined by group bus 45, one of the tributary buses 46 (FIG. 3) or 47 (FIG. 4), and a respective branch bus 48 (FIG. 4) or 49 (FIG. 3).

Each element of a group of elements having tributary buses feeding a group bus can transmit signals selectively to any element of a group having branch buses extending from the same group bus. Thus, any of the m computing elements, CE can selectively send information to and receive information from any one of the n storage elements SE via group bus 20 and appropriate ones of the tributary and branch buses 21-24. Hereafter, the foregoing connections will be referred to as data paths to distinguish them from configuration control information handling connections between CEs and CCRs, referred to as configuration control paths. The latter paths are described in greater detail below.

Means not shown are provided for resolving conflicts created by conflicting calls for service by two or more elements from a single element; for example, simultaneous calls by two or more CEs for access to the same SE. Such means act to assign priority to a unique one of the calling elements according to a predetermined plan. Since this priority assignment plan is not pertinent to the configuration control system of the present invention it is deemed unnecessary to provide further details thereof in this discussion.

The following brief example will sufiice to complete the description of the data paths. Suppose CB is engaged in executing a program stored in SE and wishes to retrieve its next instruction from an Instruction Address IA in SE CB raises a call line, in a path 21, 20, 23, between CE and SE and transmits a representation of the address IA, along other lines in the same path. At some later time SE runs through a non-destructive read/ regenerate cycle of access to internal address location IA. Upon readout of the contents of IA, SE raises a response line in the data path 22, 20, 24 between SE and CB while at the same time transmitting the contents of IA to a special buffer register in CB via other lines in the same path. The response signal is used by CE; to reset its call signal and the exchange is completed.

Signals participating in the establishment of new configuration control settings in CCR register are handled through a configuration control bus system including a main group bus 60 (FIG. 1), subgroup buses 61 (FIG. 2), 62 (FIG. 3), 63 (FIG. 4), and a plurality of tributary and branch buses and lines as described below. While any element may initiate a request (ELC) for action which may result in a new CCR setting, only a CE (FIG. 1) can issue the actual CCR setting information, hereinafter termed configuration mask (abbreviated CM). Thus, each CE is provided with a CM output bus such as the bus 71 extending from CB (FIG. l), and each element CCR is provided with a CM input bus 72 for carrying a CM from any CE to the receiving gates 11 of that CCR.

An abbreviated form of notation is employed herein to characterize the information units pertinent to a reconfiguration process. In general this notation has the form: A/B/C, where A represents the information unit, B represents the issuing source, and C specifies particular bits in A. Thus, the notation employed for example to designate particular bits a, b of a 36-bit CM word issued by CB would be CM/CE /bits a, b.

As indicated above, a CM word issued by a CB unit via a CM output bus, such as 71 (FIG. 1) would be applied in parallel to the CCR receiving gates 11 of all elements in FIGS. l-4 via respective CM branch buses 72 extending in parallel to all the inputs of such gates. Thus, the notation CM/CE at 72 indicates that all CM/CE outputs extend as inputs via 72.

Those elements which are intended to receive an issued CM are designated by corresponding bits in a coincidentally issued Selection Mask word (SM). The latter is provided by the CE issuing CM on its associated SM/CE output bus 73. Thus, the gates 11 of CCR/CB (FIG. 1), CCR/SE (FIG. 2), CCR/IOCE, (FIG. 3), and CCR/TCU (FIG. 4), are respectively selected by the conditions of bits designated bit CE bit SE bit I0 and bit TCU in an SM word. 'Since such SM words may issue from any of in source CEs, a separate SM bit line is extended from the bus 73 of each CE to each group of element CCR gates 11. Thus, the lines 74 extending to the gates 11 of CCR/CE in FIG. 1 can convey a CB selecting bit of an SM word issued by any CE (including CB to the receiving gates 11 of CE,, as indicated by the notation SM/CE /bits CB Similarly, the lines 74 extending to SE (FIG. 2) convey SE selecting bits of SM words to the SE, receiving gates 11 as indicated by the notation SM/CE /bits SE Likewise in FIG. 3, the notation SM/CE /bits I0 adjacent bus 74 indicates that the I0 selecting bits of all SM words are conveyed to the gates 11 of IOCE Finally, in FIG. 4, SM/CE /bits TCU indicates that a TCU; selecting bit in each SM word is conveyed thereby to gates 11 of TCU It should be understood that the SM selecting bits are binary-valued signals which in different binary states will tend to actuate or not actuate respective gates 11. It may further be inferred by noting the lines 15 and 16 extending from each CCR to the gates 11 that each CCR can selectively accept or reject a proffered CM (Le. a CM accompanied by an active selecting bit of an SM on a line 74) in accordance with the information currently stored in the CCR. The lines 15 issue from a pair of status bit stages (S S of each CCR, the combined states of which determine four element states: 00, 01, 10, and 11. The lines 16 issued from individual supervisory control bit stages (SC of the respective CCR, and correspond individually to the different CE elements of the complex. Signals on line 16 control acceptance of incoming CMs proffered by SM selecting bits of respective CE units. The status bits S determine whether the associated CCR is in a state permitting acceptance of a proffered CM, and if the state is such a permissive one, the control bits SC selectively determine the CEs from which a CM may be accepted (i.e. the element selecting bits of SM to which the gates 11 may react). An element accepting reconfiguration signals its acceptance of the proifered CM to the issuing CE via an individual response line 75. Failure to receive such a response from any element designated by SM is taken as an indication of program error, or other failure, by the issuing CE.

Over-riding control over the S and SC bits in each element CCR is afforded by reset (R) input lines 76 which are coupled to reset output lines 77 issuing from the CEs (FIG. 1) by manual reset controls 78, and SC bypass circuits which are not shown in FIGS. l-4. The latter circuits which are considered in detail in connection with the discussion herein of FIG. 10 are effective to force all SC bit outputs of CCR to the 1 condition when all SC bits are coincidentally set to 0. The effect thus produced is the same as if all SC bit states in the CCR had been set to 1. Were it not for this, if all SC bits in CCR became coincidentally set to 0 due to some faulty condition the CCR of the associated element would become isolated from further reconfiguration control.

Each element is further provided with an Element Check signalling line (ELC) 79 by means of which the existence therein of an error or other condition requiring attention is made known to the CEs. Recall that the CE states (S) are so maintained that at least two CEs will be able to respond to the ELC. Thus invariably one CE will respond to the ELC, execute SCON, assemble CM and SM, and thereby reconfigure the complex. In response to an ELC signal which is a pulse of short duration, subject to certain checks and priority resolution procedure described hereafter, one of the enabled CEs, other than a CE originating an ELC, will perform a program routine to evaluate the cause of the error or failure which gave rise to the ELC siganl. This one CE will then assemble and forward SM and CM words under control of SCON instructions through the bus system described above. These words condition the selected CCRs and thereby reconfigure the complex. Assuming that the SCON instructions are appropriately programmed the data processing operations in progress prior to the ELC may be continued without significant interruption.

The response and ELC output lines 75 and 79, of all elements are extended in parallel via branch buses 80 (FIG. 1) to the receiving gates 12 of each CE.

The general effects characteristic of a reconfiguration process as executed by the apparatus shown in FIGS. l-4 may now be summarized as follows:

First, one or more CEs are made aware by an ELC or an internal condition, such as a decoded program instruction or manual switch setting, of a problem condition. After executing several internal checks to determine whether they can legitimately issue reconfiguration information, the CEs attempt to gain access to a predetermined one of several SEs containing information pertient to reconfiguration. By executing a priority resolution procedure these CEs enable a unique one of their number to address the SE and withdraw therefrom a status table containing a listing of the status bits (S) currently stored in all CCRs of the complex. Utilizing this status intelligence the enabled CE exchanges other program information with an associated SE to determine a configuration of the complex suitable for maintaining continued processing. Branch decisions in this program guide the controls of the enabled CE to selectively address and execute one or more SCON (Set Configuration) instructions in SE appropriate to the circumstances. Each SCON instruction is retrieved and used by the enabled CE to control a corresponding internal subroutine during which appropriate CM and SM words are assembled in CB registers. Thereupon, the CM word is broadcast towards all elements via bus 60 and proffered to selected elements according to the coincident states of corresponding selecting bits of SM. Those elements designated by SM which are permitted by the S and SC bits in their CCRs to accept the proffered CM do so and acknowledge their acceptance by a signal sent to the issuing CE via the response line 75. The response signal is utilized by means not shown in the issuing CE to reset the corresponding SM selecting bit in the CE register preloaded with the SM word.

When the mask issuing CE determines, by internal controls further considered herein, that all elements designated for reconfiguration by one or more SMs have accepted the correspondingly proffered CM, and therefore verified that the complex has been suitably reconfigred, it may terminate its execution of SCON instructions and resume operation as part of a processing subsystem within the complex, retrieving its next program instruction from an SE to which it is configured by a corresponding SE bit in its CCR.

It will be noted that the output lines 17 of each CCR control the receiving gates 12, of the associated elements, through which all information, other than configuration control information, or adapter information proffered to an IOCE, reach the internal parts 13 of the associated element. Thus, it will be understood that outputs of any CCR connecting to respective lines 17 control reception of information transmitted to the associated element from other elements and thereby the CCR contents in effect determine all signal flow between elements of the complex. A feature of this reception control is that a faulty element, even if it should become isolated from reconfiguration control and continue to transmit erroneous information, can still be cut off from all "healthy" elements of the complex by reconfiguration; i.e. by changing a bit in each healthy elements CCR.

CCR FORMATS Considering FIGS. 6l3, with reference to the complex shown in FIGS. l-4, organizational details of the subject configuration control system may now be explained FIGS. 69 show the relative formats of the control information stored in the respective CCRs of the CE, SE, 10, and adapter elements. The information in each CCR consists of selected parts of a basic 36-bit control word or CM format.

In the particular embodiment of a multiprocessing complex described herein, there are provided four CEs, a maximum of 12 SEs, and a maximum of 3 IOCEs, (Le. m=4, n=12, and 13:3). Accordingly, each CCR contains four supervisory control bit stages (SC the outputs of which control acceptance of a new CM proffered by corresponding ones of CE Each CCR also contains a pair of status bit stages (S and from one to four parity bit stages (P Each parity bit stage holds the anticipated correct parity of the 8 consecutive bits to the left thereof in the CCR and may be compared to a computed parity as a check on the correctness of those 8-bits.

In CCR/CE (FIG. 6) there are 12 bits stages, designated CCR/CE /bits SE, which have outputs controlling the receipt by the associated CE of intelligence forwarded by respective storage elements SE of the complex. There are also 4 bit stages CE controlling receipt by the particular CE of intelligence conveyed from correspondingly numbered CEs, and bit stages IOCE E (or 10 for controlling acceptance by the particular CE of signals originated by corresponding IOCEs. The seven blank bit positions 90, distinguished by a pattern of parallel slanted lines, are considered spare or dont care positions since the outputs therefrom do not control receiving gates.

CCR/SE (FIG. 7), in addition to S, SC and P bit stages contain bit stages CE, and 10 corresponding to the CE and I0 bit stages in CCR/CE (FIG. 6) for controlling acceptance by SE of signals conveyed from receptive CEs and IOCEs. All other bit positions are spare (unused). Since the eight positions 91 to the left of P are blank all of the nine stages 91 and P are unnecessary and may be omitted.

CCR/IOCE (FIG. 8) in addition to bit stages denoted S, SC and P, contain bit stages SE and CE, corresponding to correspondingly designated SE and CE bit stages in CCR/CE, for controlling acceptance by IOCE of signals conveyed from respective SEs and CEs. All other bit positions are spare. Since stage P and the associated eight stages 92 are not used, all nine of the stages may be omitted.

The CCR of each peripheral adapter, for example, CCR/TCU (FIG. 9), in addition to S, SC and P bit stages contains bit stages IO in positions corresponding to those held by the IO bits in CCR/CE, for controlling acceptance by the respective TCU of signals supplied by IOCE respectively. All other stages contain unused or spare bits. Since the stages 93 and 94 respectively associated with P and P are unused, all eighteen of these stages are unnecessary and may be omitted.

Considering the CCR formats (FIGS. 6-9) in relation to the associated elements of the complex as characterized in FIGS. l-4, a number of pertinent observations may be made. An element (for example an SE) may be prevented by a CE bit in its CCR from receiving ordinary communications from a particular CE and yet be permitted by an SC bit in its CCR to accept a CM proffered by the same CE. Secondly, since the CCRs of all elements have similar control information formats-for example, the status and SC bits invariably occupy the first six register positions in all CCR registers, and CE bits, where available, are invariably located in the last four places of a group associated with parity stage P -a single pair of CM and SM words can be used to set a plurality of CCRs when circumstances permit (e.g. when the associated elements are grouped together in a subsystem). Thus, the network of gates and the programming required for entering information into a CCR may be simplified. Finally, it is repeated that while the adapter registers, such as CCR/TCU, have 10 bits to control reception of IOCE originated information, the IOCE CCRs do not contain adapter bits for controlling information flow in the reverse direction. The reason for this is that the ordinary reciprocal controls between IOCE and adapter units are so arranged that if one cannot receive a communication from the other, it cannot provide a key sequence of responses necessary to actuate the other. It is, therefore, sufficient to provide CCR control in one direction only.

CCR/CE AND SETTING LOGIC Referring to FIG. 10, the configuration control register CCR/CB of a representative computing element CH and the circuits controlling access thereto which are substantially identical in internal organization to similarly functioning circuits in the other CEs (GE are seen to comprise detail features, as follows: For admitting new configuration control settings (CMs), proffered by any of the CEs including CE to the 36-stage register 100, which is the CCR/CB associated with CE there are provided four AND circuits 101-104. The outputs of these control four respective groups of AND circuits 105-108. Each group of circuits 105-108 comprises 36 individual AND circuits for handling the individual bits of a CM signal presented to the group. Groups 105 to 108 are coupled to receive CM signals originating at respective CEs, CE, to CH via respective groups of input wires 110 to 113. OR gates 115 combine respective outputs from each group of AND circuits 105-108 to provide a single 36-bit input path feeding into register 100.

AND circuits 101-104 are partially controlled by CE; bits in selection mask signal words (SM) issued by respective CEs, CE As an additional partial control, signals corresponding to the outputs of bit stages SC of register 100, except under conditions noted below, are applied to respective ones of circuits 101-104. As a third partial control, SCON enabling signals applied via line 119 to AND circuit 101, and SCON enabling signals transmitted by other CEs are applied via line 120 to the other three AND circuits 102-104. A fourth control input 121, to each AND circuit 101-104, is a TEST LATCH OFF signal provided by CE, control circuits shown in FIG. 11. This line is disabled only when the values of both status bits, S and S in CCR/SE are coincidentally 0 while a CB Test Switch (FIG. 11) is in an ON position, and it is restored to an enabled cond tion when the switch is thereafter placed in an OFF position.

An additional group of 36 inputs to OR circuits 11.5 not previously mentioned is supplied by lines 125. Signals on these lines correspond to respective manually keyed inputs 126 of 36 AND circuits indicated generally at 127, when the latter are enabled by a control signal on a common input line 129. Line 129 couples to the ON output of the TEST LATCH feeding line 121, so that the circuits 127 are enabled only when the line 121 becomes disabled; i.e. only when bits S and S in CCR/CB are both 0 while the CE, Test Switch is ON. It is, therefore, clear that the contents of register can be manually changed through action of keys or switch levers only under certain predetermined status bit and test switch conditions, this feature serving as an interlock or precautionary measure precluding accidental isolation of an element from CE, as a result of accidental manipulation of a CCR/CB bit setting key.

The outputs of AND circuits 101-104 are also extended to OR circuit 133 which couples to the CE configuration response line 75, also shown in FIG. 1. Upon application of a CE, selecting bit of any selecting mask SM to one of the AND circuits 101-104, and translation of a signal through the circuit, a response is conveyed via line 75 to the CE issuing the SM. This response is used to reset the CE, selecting bit in the register holding SM whereby the issuing CE is informed that CE; has accepted a proffered CM in CCR/CB When energized, reset line 135 sets each of bits SC to 1, and each status bit 5,, S to 0, so that if the CE, tcst switch is OFF, AND circuits 101-104 will all be enabled, permitting any CE to reconfigure CCR/CE.

A bypass circuit 137 translates the outputs of bit positions SC of register 100 to the output bus 138. This circuit includes four OR circuits 139-142, effective to directly translate respective SC outputs of register 100, so long as one or more of these outputs are in the 1 state, and an OR circuit 143 and complementing circuit 144 providing an indirect second input to OR circuits 139-142, which is active whenever all bit stages SC in register 100 simultaneously contain zeroes. AND circuits 101- 104 cannot all be simultaneously disabled by CCR/CE bits SC since the corresponding inputs are taken from the bypass circuit 137, and the four outputs of the latter invariably include a 1" signal. Thus, register 100 cannot be accidentally isolated from all CEs. As an additional feature, the circuits 137 produce an all ones" output, upon detection of a CCR parity error, as suggested by inline 145 joining the output of complementing circuit Each of the other CEs, CH while not illustrated in detail should be understood to comprise CCR gates and controls corresponding in function and organization to those shown in FIG. 10 for CB except that the identitles of inputs to the circuits corresponding to AND circuits 101-104 and 127 of CE, are different for each of the CEs. For CB the AND circuits in the positions of AND circuits 101, 103 and 104 would be controlled jointly by CB TEST LATCH OFF and SCON signals origmating at other CEs, and individually by bits 0E in SM/CE SM/CE and SM/CE, respectively, and by bits 8C S6 and SC, from respective stages of CR/CE2- The AND circuit in the position of AND 102 would have inputs SCON/CB SM/CE /bit CB and CCR/CE /bit 8C The AND circuit corresponding to AND circuit 127 would be conditioned to pass CB data key signals by CE: TEST LATCH ON, and the output of the OR circuit corresponding to OR circuit 133 would represent the reconfiguration response of CE For CB the AND circuits corresponding to 101, 102 and 104 would be controlled jointly by C5 TEST LATCH OFF and SCON signals produced by CE CB or CB and individually by bits CE, of SM/CE SM/CE and SM/CE respectively, and outputs of bit stages SC SC; and 5G,, respectively of COR/CB The inputs to the cir- :uit corresponding to AND circuit 103 would be SCON/ CB3, CF13, and The it! puts to the circuit corresponding to AND circuit 127 would be CB TEST LATCH ON and CB DATA KEYS.

For CE, the counterparts of AND 101, 102 and 103 would have joint inputs CB TEST LATCH OFF and SCON (controlled by CE CB or CB as well as first individual inputs SM/cE /bit CB SM/ CE bit CE.,, and SMICE /bit CB respectively, and second individual inputs CCR/CE /bit S CCR/CEq/bit C and CCR/ CE /bit SC respectively. The counterparts of AND 104 would have inputs SCON/CB SM/CE /bit C13,, and CCR/CEq/bit SC.,. The counterpart of AND 137 would nave inputs CE, TEST LATCH ON and CE, DATA KEYS.

ISSUANCE OF SCON BY CE Consideration is next given with reference to FIGS. 11 and 14 to the other internal parts 160 of CE, and by analogy to correspondingly functioning parts in the other :Es. These include circuits which participate in the acquiaition and execution of SCON instructions and the collecion and distribution of CM and SM words to selected CCRs throughout the complex, with consequent reorganization of the associated elements within the complex. the parts 160 are grouped into four main subdivisions iesignated receiving gates 161 (same as gates 12 in FIG. l), computing and parity checking circuits 162, interrupt :ontrols 163 and instruction decoding and other controls l64. The latter includes CE, Test Switch 165, other manial controls 166, and other electronic controls yet to be liscussed. The circuits 162 carry out computing and parity rhecking functions on information handled within 160 in- :luding checks on the information content of the configuation control register 0 (CCR/CH of FIG. 10. The :ircuits 164 provide all of the sequential control signals .nd include sufficient local buffer storage registers to en- .ble CB to carry out its computing and data handling unctions effectively.

The other manual controls 166 include switches for :ontrolling power sources, single cycle operation, and Ither functions requiring manual control. The controls 66 are effective only when a bistable test latch 167 is in he ON state as indicated by a predetermined signal level n line 168. Latch 167 is set to the ON state by an active Iutput from AND circuit 169. The latter occurs only then test switch 165 is in the ON condition and concidentally an enabling output is received from AND ircuit 170 in response to 0 indication from stages 8, and 2 of CCR/CE, (FIG. 10). Once in the ON state latch 67 can be reset to OFF only by setting the test switch 65 to its OFF position. Thus, the bits S and 8;, can be aried for test purposes While the test latch remains in the )N state.

The OFF and ON outputs of test latch 167 also extend a the lines 121 and 129 respectively shown in FIG. 10. Vhenever the condition of the test latch is reversed from )N to OFF line 171 in FIG. 10 is energized to reset bit tates S and S of CCR/CB to 0, whereby CCR/CB rerains accessible to either manual reconfiguration by turnig the test switch 165 back to ON, or by programmed :configuration as considered below.

In addition to the test latch there is provided a SCON Set Configuration) enabling latch 175 which can be set an ON condition by an output from AND circuit 176. fircuit 176 is partially enabled by one signal 177 of a roup of sequential microprogram signals schematically :presented at 178. These signals are derived during. a CB iicroprogram of operations culminating in the fetching to 013 of a SCON instruction from one of the working storages SE as described below. The signal 177 is invariably followed by a signal 179 which establishes latch in the OFF or RESET state. In its ON state latch 175 provides a SCON enabling signal to input line 119 of AND circuit 101 in FIG. 10. If, in addition, test latch 167 is OFF, a remote SCON enabling signal is translated through AND circuit 180 to the receiving gates of all other element CCRs. In each CE the remote SCON signals of the other CEs are passed through an OR circuit connection such as 181 (FIG. 11) to a control line such as 120 (FIG. 10).

Other conditions for completing the actuation of AND circuit 176 to set latch 175 are determined by the signal on line 184 (CCR/CE /bit SC and the outputs of OR circuits 185 and 186. The output of OR circuit 185 is active if all of SC bits of the configuration mask CM/CE; conveyed to bus 71 (FIG. 10) are not simultaneously in the 0 state. The output of OR circuit 186 is active if either AND circuit 187 or AND circuit 170 is energized or if microprogram control line 189 is excited. AND circuit 187 or 170 will be enabled if the values of bits S, and S are both simultaneously equal to 1 or simultaneously 0, respectively. Thus, OR circuit 186 and AND circuits 187 and 170 together comprise an inverse exclusive OR circuit acting upon the outputs of stages S and S of CCR/CB Line 189 is energized during manually initiated loading in an SE of a program scheduled for utilization by GE. Hence, this line is identified by the symbol IPL (abbreviation for Initial Program Loading Signal).

It should therefore be understood that SCON latch 175 is set to ON if microprogram signal 177 occurs while bit SC, of CCR/CB and at least one of the bits SC, in CM/CE are coincidentally in a 1 condition and either bits 8, and S in CCR/CB are set to identical conditions or control line 189 is excited. It should also be understood that latch 175 is reset to OFF by a signal on line 179 invariably following the signal 177. While in the ON condition latch 175 controls translation of SM selecting bits via line 119, FIG. 10, and via lines corresponding to 120 of FIG. 10 in all other elements.

The handling of mask words CM and SM within the circuits 160 is characterized schematically as follows. CM and SM are acquired by programming from SE units permitted access to 160 by gates 161. CM and SM are forwarded under microprogram control into selected registers and 196 respectively, which are designated R and K; respectively. This is indicated schematically by respective broken lines 197 and 198. CM and SM are parity checked through circuit 162 as indicated by broken lines 200 and 201, and are also coupled to outgoing buses corresponding to buses 71 and 73 shown in FIG. 1. As previously indicated the positional format of the CM intelligence is the same for all elements with different bits masked upon entry into different element CCRs. Thus, each CM contains status bits (S), supervisory control bits (SC), SE bits, CE bits, and IO bits and parity bits in the positions shown in FIG. 6, but the element CCRs receiving a CM will receive selected portions of a CM and different CMs may be sent out to different elements during a reconfiguration program, as will be described below.

The format of SM is indicated at 205. SM is a 36-bit selection control word including eleven non-selecting bits consisting of four parity bits P and seven spare bit positions indicated at 207 and 208, and twenty-five selection control bits which are coupled to individual element CCR access gates such as AND circuit 101 in FIG. 10, whereby selective translation to each CCR of a simultaneously issued CM may be effected. Thus, SM contains six peripheral device adapter selecting bits 209, twelve CCR/ SE selecting bits 210, four CCR/CE selecting bits 211, and three CCR/IO selecting bits 212. The values of these bits are 1 for selection and 0 for non-selection of the associated element CCRs.

It would be helpful at this point to consider the program and microprogram processes by which a CB can reconfigure the complex. Referring to FIG. 14, this process is characterized in a nine step flow chart in which the sequential sub-program or microprogram steps taken by a CB are numbered 225-233, and discussed below in that order.

Each of the CEs under consideration is designed to operate as an independent central processing unit (CPU) on instructions of the IBM System/360 instruction set. This set of instructions and their implementation by a processor are characterized, for example, in co-pending patent application Ser. No. 357,372 of G. M. Amdahl et al., filed Apr. 6, 1964, which is assigned to the assignee of the present application. Thus, each CE is a processor having internal sets of controls for executing subprograrns or microprograms required for interpretation and execution of instructions in the above set. It is further noted that each CE is provided with additional subprogram or microprogram controls for interpreting and executing a SCON instruction, as defined and characterized below, which is not a member of the IBM System/ 360 instruction set.

It is assumed that prior to step 225 (FIG. 14) the CCR organization of the complex of FIGS. 1-4 is determinable. This means that if the complex is first starting into operation it would be necessary to operate manual reset switches in all elements to establish initial states of reccptiveness in the CCRs of the CEs appropriate for initiating the reconfiguration sequence defined in FIG. 14. It is assumed that the complex is in operation prior to step 225 and a condition has occurred indicating possible need for reorganization.

In step 225 of FIG. 14, CE initiates a microprogram interrupt sequence of a predetermined class having sensed a signal associated with that class of interruption. This may involve sensing of an element check request (ELC) issued by an element other than the sensing CE or a SUPERVISOR (ALL instruction belonging to the above universal set. The CE begins its interruption sub-routine after completing its last previous operation. As part of this sub-routine the last instruction address and certain other items of information are assembled from local registers into a 64-bit program status word (PSW) which is stored (step 226) at a unique location in an SE with which the CE is then capable of communicating by virtue of the known status of reception control settings in the CCR/CE and CCR/SE registers. The unique PSW location corresponds to the type or class of interruption.

At step 227 a new PSW is fetched to CE from another unique location in that SE also corresponding to the class of interruption.

At 228 the new PSW is set up in local buffer registers of the CE and the CE proceeds to recover the first instruction of a new program series stored in an SE, in accordance with instruction address intelligence contained in the new PSW. In this new series the CE is directed (step 229) to test the status bits of all elements of the complex, by referring to a status bit table in a predetermined series of SE locations. This table is shown in FIG. 16 and discussed below with reference to that figure. Depending on the present status of the complex and the particular type of condition which caused the interruption of step 225 program branches are taken to direct the CE to either terminate the current program series (exit 237) and begin an ordinary program sequence or to continue in sequence to address a SCON reconfiguration instruction located in SE.

Assuming that the sequence leading to step 230 is followed, a SCON instruction is fetched from SE to local buffers of CE and the PSW in CE is checked to assure that the supervisory mode bit is appropriately set. The SCON instruction has an 18-bit format, parts of which are indicated at 240. There are eight bits (241) defining an operation code (OP CODE), four bits 242 designating a first local register R in CE, and four hits 243 designating 14 a second local register R in CB. Not shown at 240 are two parity bits indicating the correct parity of the other sixteen bits.

The registers denoted by R and R are the registers 195 and 196 shown in FIG. 11. These are preset under program control with respective 36-bit words representing a configuration mask (CM) and selection mask (SM) appropriate to the selected SCON instruction. Such loading may take place as part of step 229 or it may have been effected at some previous time as part of a program process of collecting several CM and SM words in several storage locations or registers.

During this phase of the process the SC bits (SC, in the CM word in R are checked, to make sure that they define a legal configuration, by OR circuit 185 and AND circuit 176 of FIG. 11. coincidentally, a current SC bit and status bits 8 S in CCR/CE, the SC bits in CM, and the current state of the CE test switch (165, FIG. 11) are checked by AND circuits 187 and 170, OR circuit 186, and AND circuit 176 (all FIG. 11) to verify that the CE is in a state in which it is permitted to execute SCON as discussed hereafter. If all checks are successful SCON latch (FIG. 11) is set, and simultaneously (231) the CM intelligence accompanied by SM selecting bits is transferred out via appropriate buses (extensions of 71, 73, FIG. 11) and presented selectively to the individual CCR receiving gates of the complex. As each element response is received (232) via bus 250, FIG. 11, the corresponding SM selecting bit in R is reset. Hence, execution of the SCON instruction may be considered complete when all bits in R are set to 0. By storing the SM word in another register, CE can up-date the CCR status bits of the responding elements in the above-mentioned status table (step 233).

At this point it is necessary to determine whether the system has been completely reconfigured or only partially reconfigured. Hence, it is necessary to revert CE to state 29 (FIG. 14) and to again conditionally exit, or proceed to another SCON instruction 231. Eventually, after one or more passes through loop 229-233 an exit condition is reached terminating the configuring program.

The status table shown in FIG. 16 includes a pair of current status bits, S S for each element of the complex, the values of which correspond to conditions stored in the status bit stages of respective element CCRs. To condition program operation 229 on the states of these hits it is necessary for the programmer to keep in mind the desired end configuration of CCR bits and to program towards that end.

A word of explanation is in order at this point. In the exemplary embodiment under consideration there are five distinct states shown in the table of FIG. 15, and a programming convention used in the exemplary present organization is that all elements in the same state will receive only from each other. While this is not an essential restriction it simplifies the reconfiguration program considerably since the configuration mask (CM) for any element can be composed by entering ones in the CM bit positions assigned to other elements in the same state and zeroes in all other bit positions. Without this restriction it would be necessary to program more of the CCR information into the status table of FIG. 16.

Now let us consider an example as follows. Assume that CB CB SE I0 and all adapters except TCU are in ACTIVE states with their respective pairs of status bits set to the condition 11 (note FIG .16). This means that all of the real time processing is being handled by these elements, that each element in this group can receive intelligence from any other element of the group to which it has a data path connection, due to corresponding 1 bits in the CCRs, and that each element in the group is insensitive, by virtue of 0 bits in its CCR, to communications from elements not in the group. Now assume CE has failed and it is desired to reassign CE; from the ACTIVE group to the TEST group while bringing in CB presently in REDUNDANT state (5 :1, 5 as a replacement for CE. This may be accomplished by means of two SCON instructions executed in two passes through loop 229233 of FIG. 14, as follows:

CB may execute the reconfiguration program.

The CM word in register R of CE in the first execution of step 231 (FIG. 14) would have the form:

The SM word in register R of CE, would have the form:

PAM TCU sscg szp sz cz to have the form:

s so 2-; 52 P2 51: CE 13 IO 11 so our% 00000011 00001001 010 And the SM word for correctly distributing this CM throughout the complex would be:

PAM TQU 1 53 P SE 10 P Assuming that the fault in CE is not located in CCR/ SE or its access network, the above CM will be cor- 'ectly accepted by CE and the reconfiguration process nay then be terminated. It might be mentioned that this assumption is a realistic one since this small segment at CE will rarely fail. Upon termination of reconfigura- :ion CE will be ready to undergo manual tests or autonatic tests under program control of CE When CE 1238 been repaired it will usually be desirable to reconigure CR, or CE; into the REDUNDANT group.

The question of priority relating to simultaneous reuests for service by several elements was briefly touched ipon above. It is worthwhile to note that IO and CE mits will generally not impose significant conflicting re- ;uest loads on an SE since the 10 elements act in re- ;ponse to I/O instructions executed by the CEs. A priorty resolution problem also occurs when several CEs at- ;empt to share access to one SE simultaneously or in nterleaved passes. This is a problem only when a CE 5 executing an extended program involving many refer- :nces to SE, for example a data sorting program. It is lot a problem for the procedure involved in FIG. 14.

Nevertheless it is noted for the sake of completeness hat hardware is provided in each SE for selecting a inique access request from among any plurality of coending requests.

It may be observed that in the example above correipondingly ordered SC and CE bits in the second CM CEP,

word do not have equal values. This results from a spe- :ialized programming restriction which is stated as folows: A CM issued to form an ACTIVE or REDUN- DANT group will have SC bits set to l in the positions corresponding to all ACTIVE and REDUNDANT CEs (this permits a REDUNDANT CE upon receipt of ELC from a CB to reconfigure the complex by first resetting the CCR state bits of the receiving CE to the ACTIVE state (11) as indicated in Note 4 (FIG. 15). A CM issued to form a TEST group may contain ls in SC bit positions assigned to CEs in the TEST state as well as in other positions to permit recall of TEST elements to other states by CEs in other states. The question occurs, what happens when an additional non-failing element is required in the TEST group. The answer is that only a CE in the ACTIVE group could safely make such a reassignment since only such a CE can be presumed to be aware of current processing needs. It is advisable to add that an element in TEST state may be less available" to the ACTIVE system than an element in REDUNDANT state since it may be accumulating important test data in a manner not susceptible to interruption, whereas processing in the REDUNDANT group is of the type which is considered instantaneously interruptable; for example, program debugging routines.

It should be obvious that if all CCR bits were stored with the status table of FIG. 16 considerably more sophisticated reconfiguration schedules could be followed. However, the present plan of organization simplifies the reconfiguration programming etIort by limiting the number of passes through the reconfiguration subroutine to at most four (i.e. the number of distinct states defined by the CCR status bits: ACTIVE (11), REDUNDANT (01), REDUNDANT (10), and TEST (00) and is adequate for most multiprocessing purposes.

CCR/ SE Referring to FIG. 12, CCR/SE, is a register 300 in which the twelve SE bit positions are unused since one SE cannot communicate with other SEs. The groups of AND circuits 305-308 controlled by outputs of respective AND circuits 301-304 are adapted to selectively translate respective CM words CM/CE CM/CE CM/ CE; and CM/CE to OR circuits 315. AND circuits 301- 304 are individually controlled by SE bits in SM words issued by CE respectively, and by bits 5C 8C 8C and SC in CCR/SE (register 300). AND circuits 301- 304 are further jointly controlled by a TEST switch 316 acting through a TEST LATCH 317. With latch 3H7 OFF circuits 301-304 are each partially enabled. Latch 317 is set OFF when switch 316 is OFF and 0N when AND circuit 318 responds to the occurrence of: TEST switch 316 ON and CCR/SE /bit S =CCR/SE /bit 8 :0.

Manually keyed CCR changes are entered in CCR/SE via ANDs 327, corresponding functionally to ANDs 127 of FIG. 10. OR 333 and bypass circuit 337 perform functions corresponding to those respectively provided by OR 133 and bypass circuit 137 of FIG. 10. Check circuits not shown provide the parity and other checks relevant to issuance of ELC/SE (on a line also not shown). OR circuit 350 conveys SCON signals from CE 2' 3 or 4 via line 320 to ANDs 301-304.

As suggested in FIG. 13, the CCR circuits for a representative IO are substantially similar to the CCR/SE circuits of FIG. 12, except that the IO bit locations in CCR/IO are unused. This is because IOs need not exchange information between each other in the present organization. Of course, this is not an essential restriction since it may be desirable in some complexes to permit transfers between IOs.

Referring to FIG. 15, in the subject organization an element can be established in any of five distinct states: ACTIVE (8 :1, 8 :1), REDUNDANT (8 :0, 5 :1), REDUNDANT (8 :0, 8 :0), TEST WITH LATCH OFF (8 :0, 8 :0), and TEST WITH LATCH ON (S -=0, 8 :0). REDUNDANT-10, differs from RE- DUNDANT-01 only in that the former state alfords some

Images