US5841973A - Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory - Google Patents

Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory Download PDF

Info

Publication number
US5841973A
US5841973A US08/615,694 US61569496A US5841973A US 5841973 A US5841973 A US 5841973A US 61569496 A US61569496 A US 61569496A US 5841973 A US5841973 A US 5841973A
Authority
US
United States
Prior art keywords
processing element
message
message queue
field
destination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US08/615,694
Inventor
Richard E. Kessler
Steven M. Oberlin
Steven L. Scott
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Morgan Stanley and Co LLC
Hewlett Packard Enterprise Development LP
Original Assignee
Cray Research LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cray Research LLC filed Critical Cray Research LLC
Priority to US08/615,694 priority Critical patent/US5841973A/en
Assigned to CRAY RESEARCH, INC. reassignment CRAY RESEARCH, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OBERLIN, STEVEN M., SCOTT, STEVEN L., KESSLER, RICHARD E.
Application granted granted Critical
Publication of US5841973A publication Critical patent/US5841973A/en
Assigned to SILICON GRAPHICS, INC. reassignment SILICON GRAPHICS, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CRAY RESEARCH, L.L.C.
Assigned to FOOTHILL CAPITAL CORPORATION reassignment FOOTHILL CAPITAL CORPORATION SECURITY AGREEMENT Assignors: SILICON GRAPHICS, INC.
Assigned to U.S. BANK NATIONAL ASSOCIATION, AS TRUSTEE reassignment U.S. BANK NATIONAL ASSOCIATION, AS TRUSTEE SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS, INC.
Assigned to GENERAL ELECTRIC CAPITAL CORPORATION reassignment GENERAL ELECTRIC CAPITAL CORPORATION SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS, INC.
Assigned to MORGAN STANLEY & CO., INCORPORATED reassignment MORGAN STANLEY & CO., INCORPORATED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GENERAL ELECTRIC CAPITAL CORPORATION
Assigned to SILICON GRAPHICS INTERNATIONAL, CORP. reassignment SILICON GRAPHICS INTERNATIONAL, CORP. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SGI INTERNATIONAL, INC., SILICON GRAPHICS, INC. ET AL.
Assigned to SILICON GRAPHICS INTERNATIONAL, INC. reassignment SILICON GRAPHICS INTERNATIONAL, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS, INC.
Assigned to SGI INTERNATIONAL, INC. reassignment SGI INTERNATIONAL, INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS INTERNATIONAL, INC.
Assigned to SILICON GRAPHICS INTERNATIONAL CORP. reassignment SILICON GRAPHICS INTERNATIONAL CORP. MERGER (SEE DOCUMENT FOR DETAILS). Assignors: SGI INTERNATIONAL, INC.
Assigned to CRAY RESEARCH, L.L.C. reassignment CRAY RESEARCH, L.L.C. MERGER (SEE DOCUMENT FOR DETAILS). Assignors: CRAY RESEARCH, INC.
Assigned to MORGAN STANLEY SENIOR FUNDING, INC. reassignment MORGAN STANLEY SENIOR FUNDING, INC. SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS INTERNATIONAL CORP.
Anticipated expiration legal-status Critical
Assigned to SILICON GRAPHICS, INC. reassignment SILICON GRAPHICS, INC. ORDER...AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, ENCUMBRANCES, AND INTERESTS. Assignors: MORGAN STANLEY & CO., INCORPORATED
Assigned to SILICON GRAPHICS, INC. reassignment SILICON GRAPHICS, INC. ORDER...AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, ENCUMBRANCES, AND INTERESTS. Assignors: FOOTHILL CAPITAL CORPORATION
Assigned to SILICON GRAPHICS, INC reassignment SILICON GRAPHICS, INC ORDER. . .AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, CLAIMS, ENCUMBRANCES, AND INTERESTS Assignors: U.S. BANK NATIONAL ASSOCIATION, AS TRUSTEE
Assigned to SILICON GRAPHICS INTERNATIONAL CORP. reassignment SILICON GRAPHICS INTERNATIONAL CORP. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT
Assigned to HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP reassignment HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SILICON GRAPHICS INTERNATIONAL CORP.
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00Digital computers in general; Data processing equipment in general
    • G06F15/16Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G06F15/163Interprocessor communication
    • G06F15/173Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
    • G06F15/17356Indirect interconnection networks
    • G06F15/17368Indirect interconnection networks non hierarchical topologies
    • G06F15/17381Two dimensional, e.g. mesh, torus

Definitions

  • the present invention relates generally to the field of high-speed digital data processing systems, and more particularly, to a messaging facility for a distributed memory multiprocessor computer system.
  • Multiprocessor computer systems comprise a number of processing element nodes connected together by an interconnect network. Each processing element node includes at least one processing element.
  • the interconnect network transmits packets of information or messages between processing element nodes. Packets comprise multiple physical transfer units (phits). A phit is typically the width of a network physical communication link or physical channel between processing element nodes.
  • Multiprocessor computer systems having hundreds or thousands of processing element nodes are typically referred to as massively parallel processing (MPP) systems.
  • MPP massively parallel processing
  • each processing element is considered a stand-alone computer with its own central processor, local memory, and associated control logic.
  • each processing element can only address its own local memory and cannot directly read or write the local memory associated with another processing element.
  • Each processing element reads data from another processing element's memory by sending a message and input/output (I/O)-like packet, to the target processing element requesting that some data from its memory be formatted and sent back to the requesting processing element, or vice versa for writes.
  • I/O input/output
  • Message passing is a well-known and prevalent MPP system programming model because multicomputers are relatively easy to build.
  • the ease of construction of a multicomputer arises from the use of commodity microprocessors in an environment that closely resembles their "natural habitat" (i.e., that hardware and software implementation envisioned by the microprocessor designers), that is, a network of small autonomous computers.
  • every processing element can directly address all of memory, including the memory of another (remote) processing element, without involving the processor at that processing element.
  • I/O processing element-to-remote-memory communications
  • reads or writes to another processing element's memory are accomplished in the same manner as reads or writes to the local memory.
  • Multiprocessors have an ease-of-programming advantage over multicomputers.
  • each communication between processing element's must be explicitly coordinated and initiated by the programmer.
  • communications arise naturally and implicitly by the specification of variables and constants in program statements. If the data represented by the variable or constant name happens to reside in the memory of a remote processing element, an access of the data there is automatically initiated the same manner as if the data were in local memory.
  • This messaging facility enables the passing of packets of data from one processing element to another without having an explicit destination address in the target processing element's memory.
  • a message is a special cache-line-size remote write that has as its destination a hardware-managed message queue in the memory of the receiving processing element. Arriving messages are placed in the message queue in the order that they arrive by hardware mechanisms that also provide the lowest-level communication handshaking protocol service. Flow control between processing elements is accomplished by the queue management hardware, with software intervention used as necessary to deal with the error cases caused by queue overflows, and the like.
  • the preferred embodiment of the messaging facility described in the U.S. Pat. No. 5,581,705 has a fixed sized message queue in a designated area of the local memory.
  • interrupt circuitry in the destination node generates an interrupt to the processor in the destination node each time a message is written into the message queue in the destination node.
  • a shared memory message queue for example, is very inefficient because each processor needs to acquire access or rights to the shared memory message queue prior to altering information in the shared memory message queue, such as with a locking or similar semaphore type operation.
  • the present invention provides a messaging facility in a multiprocessor computer system having a plurality of processing elements interconnected by an interconnect network.
  • Each processing element includes a processor and a local memory. Globally addressable portions of local memory of each processing element form a distributed memory.
  • the messaging facility includes assembly circuitry in a source processing element for assembling a message to be sent from the source processing element to a destination processing element based on information provided from the source processing element's processor.
  • a network router transmits the assembled message from the source processing element to the destination processing element via the interconnect network.
  • a message queue in the local memory of the destination processing element stores the transmitted message.
  • a control word stored in the local memory of the destination processing element includes a limit field designating a size of the message queue and a tail field designating an index into the corresponding message queue to indicate a first location in the message queue where the transmitted message is to be stored.
  • Shell circuitry in the destination processing element atomically reads and updates the tail field. In this atomic operation, the tail field is read to determine where to store the transmitted message and updated to point to a second location in the message queue where a next transmitted message is to be stored.
  • the messaging facility includes interrupt circuitry.
  • the control word preferably includes a threshold field designating an interrupt location in the message queue.
  • the interrupt circuitry generates an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field.
  • the destination processor maintains a head pointer designating an index into the corresponding message queue to indicate a location where the message is stored in the message queue for reading by the destination processing element's processor.
  • the assembly circuitry includes a plurality of external registers.
  • the message is assembled in an aligned block of external registers within the plurality of external registers.
  • the assembly circuitry includes a plurality of write buffers internal to the source processing element's processor.
  • the destination processing element polling the control word stored in the local memory of the destination processing element to ascertain if the transmitted message has been stored in the message queue.
  • the destination processing element includes shell circuitry to generate a send-accepted packet to be provided to the source processing element based on the tail field being less than the limit field to indicate that the designation processing element accepted the transmitted message.
  • the shell circuitry generates a send-rejected packet to be provided to the source processing element based on the tail field being equal to or greater than the limit field to indicate that the message queue is full and that the designation processing element rejected the transmitted message.
  • the messaging facility of the present invention can include multiple message queues in the local memory of the destination processing element's local memory.
  • the number of message queues in the destination processing element's local memory is preferably defined by software running on the destination processor.
  • an interrupt status register stores a multiple messages bit to indicate if the tail field is equal to the interrupt location indicated by the threshold field in multiple message queues.
  • the size of each message queue is variable and the local microprocessor does not have to be interrupted with each message arrival.
  • the messaging facility according to the present invention still provides an efficient one-way transversal of the interconnect network, and does not require the complicated locking mechanism such as is required with a shared memory message queue.
  • the present invention provides flexibility in that the local processor can poll its queue to determine when a message has arrived.
  • the flexibility of the messaging facility according to the present invention permits an interrupt to be generated in a variety of situations such as at the receipt of each message, at any specified number of messages, or when the message queue is full.
  • the messaging facility according to the present invention can provide an arbitrary number of queues in each processing element.
  • FIG. 1 is a simplified model of a massively parallel processing (MPP) system.
  • MPP massively parallel processing
  • FIG. 2 is a simplified block of a MPP system.
  • FIG. 3 is a block diagram of a processing element included in the MPP system of FIGS. 1 and 2.
  • FIG. 4 is diagram illustrating a portion of a message queue used in the MPP system of FIGS. 1 and 2.
  • FIG. 5 is a diagram of message queue control word used to define the message queue of FIG. 4.
  • FIG. 6 is a diagram illustrating a message send E-register command used to transfer a message from source E-registers to a slot in a destination message queue.
  • FIG. 7 is a diagram illustrating a Message address register format.
  • FIG. 8 is a diagram illustrating generating the message slot global virtual address (GVA).
  • FIG. 9A illustrates a message queue before a message store in a scenario where a tail field value is less than a threshold field value.
  • FIG. 9B illustrates a message queue after the message store the scenario where a tail field value is less than a threshold field value.
  • FIG. 10A illustrates a message queue before a message store in a scenario where a tail field value is equal to a threshold field value.
  • FIG. 10B illustrates a message queue after the message store the scenario where a tail field value is equal to a threshold field value.
  • FIG. 11A illustrates a message queue before a message store in a scenario where a tail field value is greater than a threshold field value.
  • FIG. 11B illustrates a message queue after the message store the scenario where a tail field value is greater than a threshold field value.
  • FIG. 1 illustrates a simplified model of the components of a massively parallel processing (MPP) system generally indicated at 20.
  • MPP system 20 typically comprises hundreds or thousands of processing element nodes, such as processing element node 22.
  • An interconnect network 24 provides communication paths among the processing element nodes in MPP system 20.
  • interconnect network 24 forms a three-dimensional matrix of paths which connect the processing element nodes in the x, y, and z dimensions as indicated by arrows 26.
  • FIG. 2 A simplified block diagram of MPP system 20 is illustrated in FIG. 2.
  • each of the N processing elements, such as processing element 22, in MPP system 20 include a processor 30 connected to a local memory 32.
  • MPP system 20 is a MIMD massively parallel processing system with a physically distributed, globally addressable memory.
  • Each processor 30 has a favored, low latency, high bandwidth path to an associated local memory 32, and a longer latency lower bandwidth access to the memory banks 32 associated with other processors 30 over the interconnect network 24.
  • the interconnect network 24 is comprised of a three-dimensional torus which, when connected, creates a three-dimensional matrix of processing elements.
  • the torus design has several advantages, including high speed information transfers and the ability to avoid bad communication links.
  • a toroidal interconnect network is also scalable in all three dimensions. Interconnect networks of this nature are described in more detail in the following co-pending and commonly assigned U.S. patent applications: U.S. Pat. No. 5,533,198, entitled “DIRECTION ORDER ROUTING IN MULTIPROCESSING SYSTEMS,” by Gregory M. Thorson, filed Nov. 30, 1992 and U.S. Pat. No. 5,701,416, entitled “ADAPTIVE ROUTING MECHANISM FOR TORUS INTERCONNECTION NETWORK,” by Thorson et al., filed Apr. 13, 1995, both of which are incorporated herein by reference.
  • FIG. 3 illustrates a simplified block diagram of one embodiment of processor element 22.
  • processor element 22 includes a high-performance reduced instruction set computer (RISC) microprocessor 30 such as the EV5 microprocessor (DEC P/N 21164) or the EV56 microprocessor (DEC P/N 21164A), both available from Digital Equipment Corporation.
  • RISC reduced instruction set computer
  • Each microprocessor 30 is coupled to a local memory 32 that is a distributed portion of the globally-addressable system memory. Local memories 32 can be formed, for instance, by stacking a plurality of dynamic random access memories (DRAM) integrated circuits 34 configured as banks of memory operating under control of memory bank controllers 36.
  • DRAM dynamic random access memories
  • Microprocessor 30 is also connected to a nonvolatile memory 38, such as an erasable programmable read only memory (EPROM) used to store relatively unchanging information such as, for instance, boot program code.
  • EPROM erasable programmable read only memory
  • Each processing element 22 further includes shell circuitry 40 that implements synchronization and communication functions facilitating interactions between processor elements 30.
  • Shell circuitry 40 includes external registers circuit 42 and an interconnection network router 44 used to connect multiple processing elements 22 in a three-dimensional toroidal "fabric."
  • the toroidal interconnect network carries all data communicated between processing elements 22 and those memories 32 that are not local to the particular processing element 22.
  • external registers circuit 42 permits asynchronous (i.e., independent of the local processor) movement of data, including blocks of data, between processor 30 and remote memories 32 associated with other processing elements 22 and between local memory 32 and remote processors 30 associated with other processing elements 22.
  • external registers circuit 42 is optionally used to provide flexible addressing modes that permit a high degree of control over the redistribution of data between shared portions of the system memory.
  • shell circuitry 40 also includes an input/output circuit 46 used to communicate over a high-speed channel with one or more input/output devices (not shown).
  • an input/output circuit 46 used to communicate over a high-speed channel with one or more input/output devices (not shown).
  • Such a high-speed channel is described a commonly assigned patent application Ser. No. 08/614,854, entitled “RECONFIGURABLE RING BASED NETWORK SYSTEM,” filed on even date herewith, and which is herein incorporated by reference.
  • microprocessor 30 is connected through external register circuit 42 to memory 32.
  • External register circuit 42 includes a plurality of external registers (E-registers) 48.
  • Microprocessor 30 is capable of reading from and writing to E-registers 48 via external data path 50 and external address path 52.
  • external register circuit 42 is capable of reading from and writing to memory 32 via memory data path 54 and memory address path 56.
  • microprocessor 30 can be implemented in a commodity microprocessor, such as the EV5 or the EV56 available from Digital Equipment Corporation. Commodity microprocessor implementations impose limitations on the types of application behaviors that can be supported efficiently.
  • External register circuit 42 provides a means to extend the capabilities of the commodity microprocessor with minimal requirements imposed on microprocessor 30.
  • E-registers 48 are mapped directly into the memory space of microprocessor 30. This provides high-bandwidth and low-latency load and store access to the registers themselves.
  • special memory-mapped store operations are used to provoke the E-registers to reference normal memory locations. These special operations include the ability to move data to/from E-registers 48 and memory locations in memory 32, to perform memory synchronization operations, initiate message transfers (to, for instance, other microprocessors), and to perform local operations that otherwise cannot be performed quickly within microprocessor 30.
  • E-registers 48 serve as source or destination for additional operands and can be used to store the results of operations on operands supplied by microprocessor 30 or even for the results of operations on those operands previously stored in E-registers 48.
  • a memory location in memory 32 local to a particular processing element can be accessed in two ways: a direct (cacheable) load or store by the processor, or an indirect (non-cacheable) operation through the external registers (E-registers) 48 in external register circuit 42.
  • the virtual address used to access the memory location can be the same in both cases.
  • Processor load and store instructions can reference only local memory.
  • An on-chip DTB translates the virtual addresses to local physical addresses when the processor directly issues loads and stores.
  • Shell circuitry 40 translates the virtual address to a (either local or remote) physical address when the processor issues E-register GET and PUT references.
  • Each packet preferably comprises a header.
  • the header includes routing information that steers the packet through the network, destination information that indicates which processing element should receive the packet, and control information that instructs the processing element that receives the packet to perform an operation.
  • the header may also contain source information that indicates which processing element created the packet and may contain memory address information.
  • a packet optionally comprises a body. The body of a packet typically includes data to be transferred between processing element nodes or system data from or to a host system.
  • the network routers such as network router 44, at each processing element node 22 transfer packets through communication links in interconnect network 24 based on the routing information contained in each header of each packet.
  • the header and the body have variable lengths and transfer over the communication links one phit at a time.
  • MPP system 20 preferably uses wormhole routing. Preferred methods of routing the packets with wormhole routing are described in the above incorporated co-pending and commonly assigned U.S. patent application Ser. No. 07/983,979, entitled “DIRECTION ORDER ROUTING IN MULTIPROCESSING SYSTEMS," by Gregory M. Thorson, filed Nov. 30, 1992 and U.S. patent application Ser. No. 08/421,566, entitled “ADAPTIVE ROUTING MECHANISM FOR TORUS INTERCONNECTION NETWORK,” by Thorson et al., filed Apr. 13, 1995.
  • a message is a packet of information that one processing element, such as processing element 22, sends to another processing element.
  • shell circuitry 40 in the destination processing element stores the message in a designated location in the associated local memory 32 of the destination processing element. This designated location in memory 32 is referred to as a message queue, such as message queue 60 shown in FIG. 2.
  • message queue 60 contains more than a specified amount of messages, as specified by software, shell circuitry 40 interrupts microprocessor 30, if software enables the interrupt. Microprocessor 30 then reads the messages from message queue 60.
  • the messaging facility of the present invention utilizes several hardware and software components. These components are described in more detail below and include: the message queue; a message queue control word (MQCW); a software-maintained head pointer; a SEND E-register command; a interrupt status (IR -- STATUS) register; and a message address (MSG -- ADDR) register.
  • MQCW message queue control word
  • IR -- STATUS interrupt status register
  • MSG -- ADDR message address
  • message queue 60 is a portion of local memory 32 that software designates as the destination for messages.
  • Software defines at least one message queue, and possibly several individual message queues for each processing element 22.
  • Software also defines the size of each message queue.
  • Message queue 60 for example, stores from one to 2,097,151 messages in one embodiment of the present invention.
  • each message comprises 64 bytes of information.
  • Each 64-byte address location in the message queue 60 is referred to as a slot, such as slot 64 illustrated in FIG. 4.
  • the MQCW is a 64-bit word of information that software stores in bits 63:0! of a first slot 62 in message queue 60.
  • Shell circuitry 40 and software use the information stored in the MQCW to perform messaging functions.
  • FIG. 5 illustrates a typical format of one embodiment of a MQCW illustrated generally at 70.
  • MQCW 70 comprises four fields including: a limit field 72, a tail field 74, a threshold field 76, and a signal (SIG) bit 78.
  • Limit field 72, tail field 74, and threshold field 76 represent offset values with respect to the address of MQCW 70.
  • Limit field 72 indicates a number of slots in message queue 60. For example, when limit field 72 is set to 50, message queue 60 contains 50 slots for 50 messages. Software sets the limit field value of limit field 72, which in one embodiment varies from between one and 2,097,151 to permit message queue 60 to store from one to 2,097,151 messages.
  • Tail field 74 indicates a slot, such as slot 64, in message queue 60 which is to receive the next message.
  • the shell circuitry stores the message in the slot pointed to by the tail field 74 of the MQCW. After storing the message in the indicated slot, shell circuitry 40 increments the tail field by one.
  • shell circuitry 40 atomically reads and updates the tail field. In this atomic operation, the tail field is read to determine where to store the transmitted message and updated to point to a second location in the message queue where a next transmitted message is to be stored.
  • Threshold field 76 indicates which slot, such as slot 64, is to cause shell circuitry 40 to set an interrupt for the associated microprocessor 30. For example, when threshold field 76 is set to 30, shell circuitry 40 sets an interrupt for the associated microprocessor 30 when tail field 74 is equal to 30.
  • SIG bit 78 indicates whether tail field 74 is equal to threshold field 76.
  • Shell circuitry 40 sets SIG bit 78 to a one, when tail field 74 is equal to threshold field 76.
  • Software resets SIG bit 78 to zero by writing a zero to the MQCW. Because hardware modifies MQCW 70 in an atomic read-modify-write operation, software manipulation of MQCW 70 does not conflict with the hardware updates of the MQCW.
  • Each microprocessor 30 maintains its own head pointer with software.
  • the software-maintained head pointer value indicates the slot, such as slot 64, of the associated message queue 60 that is to be next read by the software of the local microprocessor 30.
  • MPP system 20 hardware does not use, modify, or provide storage for the head pointer.
  • FIG. 6 illustrates a message send (SEND) E-register command used to transfer a 64-byte (8-word) message from eight contiguous source E-registers of E-registers 48 to a 64-byte slot 64 in a destination processing element message queue 60.
  • the "e” represents the E-register address, where the eight contiguous source E-registers begin at E-register address e and run through E-register address e+7.
  • the SEND E-register command is described in more detail in the above-incorporated patent application entitled "METHOD AND APPARATUS FOR EXTENDING MEMORY REFERENCE CAPABILITIES OF A MICROPROCESSOR.”
  • the IR -- STATUS register comprises a message address valid bit and a multiple messages bit used for messaging.
  • the message address valid bit stored in the IR-STATUS register indicates that the tail value indicated by tail field 74 for a given message queue 60 is equal to the threshold value indicated by threshold field 76 for the given message queue 60.
  • the given message queue 60 is identified by the MSG -- ADDR register discussed in more detail below.
  • the message address valid bit remains set to one until the bit is reset by software.
  • the multiple messages bit indicates that the tail value of tail field 74 is equal to the threshold value of threshold field 76 in more than one message queue 60.
  • the multiple messages bit remains set to one until software resets the bit.
  • FIG. 7 illustrates the MSG -- ADDR register format.
  • the MSG -- ADDR register comprises a global virtual address (GVA) of the MQCW for the particular message queue 60 that set the message address valid bit stored in the IR -- STATUS register.
  • GVA global virtual address
  • the GVA in the MSG -- ADDR register remains locked in the MSG -- ADDR register until software resets the message address valid bit of the IR -- STATUS register.
  • Shell circuitry 40 stores a new GVA into the register the next time shell circuitry 40 sets the message address valid bit to one.
  • software Before a destination processing element 22 can receive messages, software must define at least one message queue 60 for that processing element 22.
  • Software defines a given message queue 60 by storing a MQCW to a location in local memory 32. The values in the MQCW fields define the given message queue 60.
  • software sets limit field 72 of MQCW 70 to 50 to define a message queue 60 which is capable of storing as many as 50 messages.
  • Software sets tail field 74 to a value greater than zero. If software were to set tail field 74 to zero, the first message would overwrite MQCW 70 because tail field 74 is an offset with respect to the MQCW address. Normally, software initially sets tail field 74 to one.
  • threshold field 76 is set to any number greater than zero up to the limit field value.
  • the threshold value stored in threshold field 76 must be at least one slot larger than the tail value stored in tail field 74 to generate an interrupt.
  • software before sending a message, software stores data for a 64-byte message in eight contiguous E-registers of E-registers 48 that align on 64-byte boundary in the E-register address space.
  • software uses STORE E-register commands, to transfer data from a source microprocessor 30 to the E-registers 48.
  • the STORE E-register command is described in more detail in the above-incorporated patent application entitled "METHOD AND APPARATUS FOR EXTENDING MEMORY REFERENCE CAPABILITIES OF A MICROPROCESSOR.”
  • SEND E-register command After storing data representing the message to be sent in the eight source E-registers of E-registers 48, software issues a SEND E-register command.
  • the SEND command signals shell circuitry 40 in the source processing element 22 to send the stored message to the destination processing element 22.
  • a SEND E-register command address opcode argument indicates to shell circuitry 40 that the E-register command is a SEND, and also indicates whether the SEND command references a segment translation table (STT).
  • shell circuitry 40 adds the index offset to the base virtual address to create a user virtual address. Shell circuitry 40 references the STT to convert the user virtual address to a GVA. When it sends a message, software sets SEND E-register command address arguments so that the GVA is the GVA for the MQCW 70 in the destination processing element 22.
  • shell circuitry 40 adds the index offset to the base virtual address to create the GVA.
  • software sets SEND E-register command address arguments so that the GVA is the GVA for the MQCW 70 in the destination processing element 22.
  • shell circuitry 40 After shell circuitry 40 receives all of the SEND E-register command address arguments from the associated microprocessor 30, the shell circuitry sets E-register state (ERS) codes for the eight contiguous source E-registers of E-registers 48 to indicate that the E-register is empty. Shell circuitry 40 then creates a message packet and sends the message packet to the destination processing element 22 along interconnect network 24.
  • E-register state ERS
  • Shell circuitry 40 does not change the value of the ERS codes again until it receives a SEND-accepted or a SEND-rejected response packet.
  • shell circuitry 40 sets the ERS codes to indicate that the E-register is full and no errors occurred during the transfer, which indicates that the message was accepted by the destination processing element 22.
  • the shell circuitry sets the ERS codes to indicate that the E-register is full, but that a previous message SEND command that used the E-register was rejected, which indicates that the message queue was full and the message was not accepted by the destination processing element 22.
  • software can resend the message.
  • the Ser. No. 08/166,443 application teaches that messages can be transmitted simply by writing a cache line to a special remote address. Messages are distinguished from ordinary remote cache-line writes by the mapping of the destination address. Special addresses are reserved in the physical memory address range to be interpreted as message transmission triggers. If a store is performed to the special address, hardware decodes the destination address to indicate that the store is actually a message send. The processing element shell circuitry then creates a message packet and sends the packet into the interconnection network for routing to the destination processing element.
  • the microprocessor checks all 4 buffers for a possible match between the address of the current store and one that may be pending in the write buffers. If a match is found, the write data is merged into its proper place in the cache line as indicated by the word-within-cache-line bits in the address. If there was data at that spot in the write buffer from a previous store with the same cache line address and word-within-cache-line pointer, the previous data is lost.
  • the DECChip microprocessor used in the MPP system described in the Ser. No. 08/166,443 application keeps the data in the write buffers for an indeterminate amount of time, unless specific steps are taken to flush them to the memory subsystem.
  • One or more write buffers will flush if: a store is made to an address that matches none of the cache-line addresses currently in the write buffers, a memory barrier instruction is issued, or an exception or interrupt occurs. Barring any of these occurrences, the write buffers will still drain eventually, but the timing of the actual write is difficult to predict.
  • shell circuitry 40 in a destination processing element 22 When shell circuitry 40 in a destination processing element 22 receives a message, the shell circuitry references a location in local memory 32, which is identified by the message packet as the MQCW 70 location. After reading MQCW 70, shell circuitry 40 compares tail field 74 to limit field 72. When the tail value of tail field 74 is equal to the limit value of limit field 72, shell circuitry 40 creates a SEND-rejected packet and returns the SEND-rejected packet to the processing element 22 that created the message. The SEND-rejected packet indicates that the message queue is full and cannot accept another message.
  • shell circuitry 40 When the tail value of tail field 74 is less than the limit value of limit field 72, shell circuitry 40 returns a SEND-accepted packet to the processing element 22 that created the message. Shell circuitry 40 then stores the message in message queue 60 in the destination processing element 22 at the slot pointed to by the tail pointer indicated by tail field 74. Shell circuitry 40 then increments the tail pointer by adding one to tail field 74 and stores the updated tail field value in the MQCW 70 in local memory 32 of the destination processing element 22.
  • the tail value of tail field 74 is less than the threshold value of threshold field 76.
  • limit field 72 is equal to four
  • tail field 74 is equal to one
  • threshold field 76 is equal to three.
  • shell circuitry 40 shifts the tail value six positions to the left and adds the shifted value to the GVA of the MQCW. The resulting sum is the byte-address GVA for the destination slot in the message queue.
  • shell circuitry 40 sets bits 5:0! of the resulting address to zeros. By setting bits 5:0! to zero, a 64-byte aligned GVA is created for the destination slot in the message queue.
  • FIG. 9A illustrates the message queue before the message store
  • FIG. 9B illustrates the message queue after the message store.
  • Shell circuitry 40 stores the message in the slot that is identified by the message slot GVA and increments the tail value by one as illustrated in FIGS. 9A and 9B. To complete the message store operation, shell circuitry 40 compares the tail value to the threshold value. Since in this scenario, the tail value is less than the threshold value, shell circuitry 40 does not set a message interrupt for the local microprocessor 30.
  • the tail value stored in tail field 74 is incremented and becomes equal to the threshold value stored in threshold field 76.
  • limit field 72 is equal to four
  • tail field 74 is equal to two
  • threshold field 76 is three.
  • shell circuitry 40 shifts the tail value six positions to the left and adds the shifted value to the GVA of the MQCW. This sum represents the GVA for the destination slot in the message queue 60 in local memory 30. After the GVA of the MQCW is added to the shifted tail value, shell circuitry 40 sets bits 5:0! of the resulting address to zeros. This creates a 64-byte aligned GVA for the destination slot in the message queue.
  • FIG. 10A illustrates the message queue before a message is stored
  • FIG. 10B illustrates the message queue after a message is stored for the second scenario.
  • Shell circuitry stores the message in the slot identified by the message slot GVA and increments the tail value by one as indicated by the move of the tail pointer from slot 2 to 3 as illustrated in FIGS. 10A and 10B.
  • shell circuitry 40 After the message is stored in the message queue, shell circuitry 40 compares the tail value to the threshold value. Since in the second scenario the tail value is equal to the threshold value after the tail value is incremented as a result of the message store, shell circuitry 40 sets SIG bit 78 of the MQCW for the message queue to one to indicate that the tail value is equal to the threshold value.
  • Shell circuitry 40 then examines the value of the message address valid bit stored in the IR -- STATUS register. When the message address valid bit is set to zero, shell circuitry 40 sets the message address valid bit to one, stores the GVA for the MQCW in a MSG -- ADDR register, and locks the MSG -- ADDR register. If software enables the interrupt, the message address value bit equal to one condition sets an interrupt for the microprocessor. When the message address valid bit is set to one, shell circuitry 40 checks the value of the multiple messages bit of the IR -- STATUS register. If the multiple message bit is set to zero, shell circuitry 40 sets the multiple messages bit to one. In either case, after the shell circuitry examines the value of the message address valid bit of the IR -- STATUS register, and performs a necessary operation based on the state of this bit, the message store operation is complete.
  • the tail value of tail field 74 is greater than the threshold value of threshold field 76.
  • limit field 72 is four
  • tail field 74 is three
  • threshold field 76 is two.
  • shell circuitry 40 shifts the tail value six positions to the left and adds this value to the GVA of the MQCW.
  • the resulting sum is a GVA for the destination slot in the message queue 60 of the local memory 32.
  • shell circuitry 40 sets bits 5:0! of the resulting address to zero. Setting bits 5:0! to zero creates a 64-byte aligned GVA for the destination slot in the message queue.
  • FIG. 11A illustrates the message queue before a message store
  • FIG. 11B illustrates the message queue after a message store for the third scenario.
  • Shell circuitry 40 stores the message in the slot as identified by the message slot GVA and increments the tail value by one as illustrated in FIGS. 11A and 11B.
  • shell circuitry 40 compares the tail value to the threshold value. Since in this example the tail value is greater than the threshold value, shell circuitry 40 does not set a message interrupt for the microprocessor. As a result, shell circuitry 40 does not modify SIG bit 78 of the MQCW.
  • SIG bit 78 has been reset by software
  • the SIG bit for the message queue remains set to one.
  • SIG bit 78 has been reset by software, the SIG bit remains set to zero.
  • Shell circuitry 40 does not modify the message address valid bit or the multiple messages bit of the IR -- STATUS register. Therefore, the message address valid bit and/or the multiple messages bit of the IR -- STATUS register remains set to one, if they have not been reset by software. If these bits have been reset by software, these bits remain set to zero. In any case, this completes the message store operation for the third scenario.
  • reading a message from the message queue can best be illustrated with a variety of scenarios which show how software reads messages out of the message queues. These scenarios provide examples for illustrative purposes only, and therefore, the example scenarios do not necessarily reflect the actual methods used by software in the preferred embodiment. In all of these scenarios, software enables the message address valid and the multiple messages interrupts in the IR -- STATUS register. Alternatively, software polls the MQCW tail pointer stored in tail field 74 to determine when messages have arrived.
  • the message address valid bit is equal to one and the multiple messages bit is equal to zero.
  • the following steps describe a process software can use to read messages out of the message queue.
  • software defines a message queue with limit field 72 set to 30, tail field 74 set to one, and threshold field 76 set to 25.
  • the local microprocessor 30 receives an interrupt from shell circuitry 40.
  • the operating system software running in microprocessor 30 reads the value of the IR -- STATUS register to obtain the values of the message address valid interrupt and the multiple messages interrupt bits.
  • the operating system software then reads the value stored in the MSG -- ADDR register, which represents the value of the GVA of the message queue that caused shell circuitry 40 to set the message address valid interrupt.
  • the operating system software then resets the message address valid interrupt to zero by writing a one to the message address valid bit of the IR -- STATUS register.
  • the operating system software then signals user software that a message queue contains messages and provides user software with the GVA of the MQCW.
  • User software then performs an atomic SWAP command on the MQCW.
  • the swaperand value of the SWAP command comprises new MQCW values which provide a limit field 72 of 60, a tail field 74 of 31, a threshold field 76 of 55, and a SIG bit 78 of zero.
  • User software then retrieves the value of the software-contained head pointer for the message queue.
  • the head pointer indicates the slot location of the next message to be read out of the message queue.
  • User software then sends a command to shell circuitry 40 to read the message that is referenced by the head pointer.
  • the command can be any command that accesses local memory 32.
  • the command can be a direct local memory load command or a get vector E-register command.
  • User software then processes the message. After processing the message, user software increments the value of the head pointer by one slot and compares the value of the head pointer to the tail pointer. When the head pointer is less than the tail pointer, user software again sends a command to shell circuitry 40 to read the message that was referenced by the head pointer, processes the message, and increments the value of the head pointer by one slot. User software repeats this process until, after being incremented by one slot, the head pointer is equal to the tail pointer to indicate that user software has read all the messages out of the message queue.
  • microprocessor 30 receives an interrupt due to this message queue exceeding the threshold value, operating system and user software repeats the above steps, except that when the SWAP command is performed, user software sets the new MQCW values to set limit field 72 to 30, tail field 74 to one, threshold field 76 to 25, and SIG bit 78 to zero. This process creates two virtual message queues that use the same MQCW.
  • the message address valid interrupt bit is equal to one and the multiple messages interrupt bit is equal to one.
  • the following steps describe a process software can use to read messages out of the message queue for this scenario.
  • user software defines multiple message queues and each message queue is initially set to have limit field 72 equal to 30, and threshold field 76 equal to 25.
  • microprocessor 30 receives an interrupt from shell circuitry 40.
  • Operating system software running in the local microprocessor 30 reads the value of the IR -- STATUS register to read the message address valid interrupt bit and the multiple messages bit.
  • Operating system software then reads the value stored in the MSG -- ADDR register to obtain the value of the GVA of the first message queue that causes shell circuitry 40 to set the message interrupt.
  • Operating system software resets the message address valid and multiple messages interrupts to zero by writing a one to the message address valid and multiple messages bits of the IR -- STATUS register.
  • the operating system software then signals the user software that multiple message queues contain messages and provides user software with the GVA of the first MQCW. For this first MQCW, SIG bit 78 is set to one.
  • SIG bit 78 When SIG bit 78 is set to zero, software does not read messages from that message queue and instead retrieves the GVA for the MQCW of another message queue. When SIG bit 78 is set to one, software continues with the following steps.
  • the swaperand value for the SWAP command comprises new MQCW values which set limit field 72 to 60, tail field 74 to 31, threshold field 76 to 55, and SIG bit 78 to zero.
  • User software then retrieves the value of the software-contained head pointer for this message queue.
  • the head pointer indicates a slot location and the next message to be read out of the message queue.
  • User software then sends a command to shell circuitry 40 to read the message that is referenced by the head pointer.
  • this command may be any command that accesses local memory such as a direct local memory load command or a get vector E-register command.
  • User software then processes the message. After processing the message, user software increments the value of the head pointer by one slot and compares the value of the head pointer to the tail pointer. When the head pointer is less than the tail pointer, user software again sends a command to shell circuitry 40 to read the message that is referenced by the head pointer, processes the message, and increments the value of the head pointer by one slot, and compares the value of the head pointer to the tail pointer. User software repeats these steps until the head pointer is equal to the tail pointer, at which point, user software has read all the messages out of the message queue.
  • User software then reads the second MQCW from memory and examines the SIG bit for the next defined message queue.
  • the above steps performed for the first message queue are repeated for the second message queue until user software has read all the messages out of the second message queue as indicated when the head pointer is equal to the tail pointer.
  • the above steps are repeated by user software for each message queue defined in the given processing element 22.
  • the message address valid interrupt bit is equal to zero and the multiple messages bit is equal to one.
  • operating system software and user software running in the local microprocessor 30 perform the same functions and steps described above for the second scenario where the message address valid bit is equal to one and the multiple messages bit is equal to zero, except that the MSG -- ADDR register does not contain a valid GVA. Because the MSG -- ADDR register does not contain a valid GVA, the operating system software does not read the value stored in the MSG -- ADDR register representing the value of the GVA of the first message queue that causes shell circuitry 40 to set the message interrupt.
  • the operating system software instead of the operating system software signalling user software that multiple message queues contain messages and providing user software with the GVA of the first MQCW, the operating system software instead only signals user software that multiple message queues contain messages.
  • the messaging facility according to the present invention provides an arbitrary number of queues in each processing element.
  • the size of each message queue is variable.
  • the operating system operating on the local microprocessor does not have to be interrupted with each message arrival.
  • the messaging facility according to the present invention still provides an efficient one-way transversal of the network of a MPP system, and does not require the complicated locking mechanism such as is required with a shared memory message queue.
  • the present invention provides flexibility in that the local processor can poll its queue to determine when a message has arrived.
  • the flexibility of the messaging facility according to the present invention permits an interrupt to be generated in a variety of situations such as at the receipt of each message, at any specified number of messages, or when the message queue is full.
  • the threshold value of the threshold field in the MQCW permits great flexibility in deciding when the local microprocessor 30 is to be interrupted in response to messages being read into an associated message queue.

Abstract

A messaging facility in a multiprocessor computer system includes assembly circuitry in a source processing element for assembling a message to be sent from the source processing element to a destination processing element based on information provided from a processor in the source processing element. A network router transmits the assembled message from the source processing element to the destination processing element via an interconnect network. A message queue in a local memory of the destination processing element stores the transmitted message. A control word stored in the local memory of the destination processing element includes a limit field designating a size of the message queue and a tail field designating an index into the corresponding message queue to indicate a location in the message queue where the transmitted message is to be stored. Shell circuitry in the destination processing element atomically reads and updates the tail field.

Description

STATEMENT REGARDING GOVERNMENT RIGHTS
The present invention was made with government support under MDA 972-95-3-0032, awarded by ARPA. The Government has certain rights in this invention.
FIELD OF THE INVENTION
The present invention relates generally to the field of high-speed digital data processing systems, and more particularly, to a messaging facility for a distributed memory multiprocessor computer system.
BACKGROUND OF THE INVENTION
Multiprocessor computer systems comprise a number of processing element nodes connected together by an interconnect network. Each processing element node includes at least one processing element. The interconnect network transmits packets of information or messages between processing element nodes. Packets comprise multiple physical transfer units (phits). A phit is typically the width of a network physical communication link or physical channel between processing element nodes. Multiprocessor computer systems having hundreds or thousands of processing element nodes are typically referred to as massively parallel processing (MPP) systems.
Many MPP systems are defined as multicomputers, wherein each processing element is considered a stand-alone computer with its own central processor, local memory, and associated control logic. In a multicomputer, each processing element can only address its own local memory and cannot directly read or write the local memory associated with another processing element. Each processing element reads data from another processing element's memory by sending a message and input/output (I/O)-like packet, to the target processing element requesting that some data from its memory be formatted and sent back to the requesting processing element, or vice versa for writes. Thus in a multicomputing system, each remote reference is essentially an I/O operation. This style of interprocessor communications is called "message passing." Message passing is a well-known and prevalent MPP system programming model because multicomputers are relatively easy to build. The ease of construction of a multicomputer arises from the use of commodity microprocessors in an environment that closely resembles their "natural habitat" (i.e., that hardware and software implementation envisioned by the microprocessor designers), that is, a network of small autonomous computers.
For many applications, however, a multiprocessor MPP system is more desirable than a multicomputing MPP system. In a multiprocessor MPP, every processing element can directly address all of memory, including the memory of another (remote) processing element, without involving the processor at that processing element. Instead of treating processing element-to-remote-memory communications as an I/O operation, reads or writes to another processing element's memory are accomplished in the same manner as reads or writes to the local memory.
Multiprocessors have an ease-of-programming advantage over multicomputers. In a multicomputer, each communication between processing element's must be explicitly coordinated and initiated by the programmer. In a multiprocessor, however, communications arise naturally and implicitly by the specification of variables and constants in program statements. If the data represented by the variable or constant name happens to reside in the memory of a remote processing element, an access of the data there is automatically initiated the same manner as if the data were in local memory.
Software techniques are known which allow the emulation of a multiprocessor and its associated ease-of-programming on a multicomputer. Such software emulators translate the application program remote reads and writes into explicit I/O-like message-passing operations in a way that is hidden from the application programmer. Software control of a global address space using message passing is very inefficient, however, because I/O-like operations have a large software start-up overhead associated with each message. Successful message passing programming models on multicomputers rely on relatively large granularity programming, passing large amounts of data with each communication, to amortize the startup overhead over many elements of data.
An efficient message-passing facility for a multiprocessor MPP system is described in co-pending and commonly assigned U.S. patent application Ser. No. 08/166,443 filed on Dec. 13, 1995 entitled "MESSAGING FACILITY FOR DISTRIBUTIVE MEMORY MASSIVELY PARALLEL PROCESSING SYSTEM," now U.S. Pat. No. 5,581,705 and which is incorporated herein by reference. This messaging facility provides certain hardware primitives to allow one processing element to send data to another processing element's dedicated message queue and interrupt the target processing element upon message arrival. The hardware supported messaging facility permits emulation of a multicomputing MPP system on a multiprocessing MPP system. This messaging facility enables the passing of packets of data from one processing element to another without having an explicit destination address in the target processing element's memory. A message is a special cache-line-size remote write that has as its destination a hardware-managed message queue in the memory of the receiving processing element. Arriving messages are placed in the message queue in the order that they arrive by hardware mechanisms that also provide the lowest-level communication handshaking protocol service. Flow control between processing elements is accomplished by the queue management hardware, with software intervention used as necessary to deal with the error cases caused by queue overflows, and the like.
The preferred embodiment of the messaging facility described in the U.S. Pat. No. 5,581,705, however, has a fixed sized message queue in a designated area of the local memory. In addition, there is only one message queue in each processing element node. Furthermore, interrupt circuitry in the destination node generates an interrupt to the processor in the destination node each time a message is written into the message queue in the destination node. There is, therefore, a need for a messaging facility in a multiprocessor computer system which uses variable size message queues and has the capability of having a variable number of message queues in each processing element node. There is also the need for a messaging facility that does not require interrupting the processor in the destination node each time a message is written into the destination queue, but which is still efficient in passing messages between processing element nodes. A shared memory message queue, for example, is very inefficient because each processor needs to acquire access or rights to the shared memory message queue prior to altering information in the shared memory message queue, such as with a locking or similar semaphore type operation.
SUMMARY OF THE INVENTION
The present invention provides a messaging facility in a multiprocessor computer system having a plurality of processing elements interconnected by an interconnect network. Each processing element includes a processor and a local memory. Globally addressable portions of local memory of each processing element form a distributed memory. The messaging facility includes assembly circuitry in a source processing element for assembling a message to be sent from the source processing element to a destination processing element based on information provided from the source processing element's processor. A network router transmits the assembled message from the source processing element to the destination processing element via the interconnect network. A message queue in the local memory of the destination processing element stores the transmitted message. A control word stored in the local memory of the destination processing element includes a limit field designating a size of the message queue and a tail field designating an index into the corresponding message queue to indicate a first location in the message queue where the transmitted message is to be stored. Shell circuitry in the destination processing element atomically reads and updates the tail field. In this atomic operation, the tail field is read to determine where to store the transmitted message and updated to point to a second location in the message queue where a next transmitted message is to be stored.
In one preferred embodiment of the present invention, the messaging facility includes interrupt circuitry. In this embodiment, the control word preferably includes a threshold field designating an interrupt location in the message queue. The interrupt circuitry generates an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field.
In one embodiment of the messaging facility, the destination processor maintains a head pointer designating an index into the corresponding message queue to indicate a location where the message is stored in the message queue for reading by the destination processing element's processor.
In one embodiment of the messaging facility of the present invention, the assembly circuitry includes a plurality of external registers. The message is assembled in an aligned block of external registers within the plurality of external registers. In another embodiment, the assembly circuitry includes a plurality of write buffers internal to the source processing element's processor.
In one embodiment of the messaging facility, the destination processing element polling the control word stored in the local memory of the destination processing element to ascertain if the transmitted message has been stored in the message queue.
In a preferred embodiment of the messaging facility of the present invention, the destination processing element includes shell circuitry to generate a send-accepted packet to be provided to the source processing element based on the tail field being less than the limit field to indicate that the designation processing element accepted the transmitted message. The shell circuitry generates a send-rejected packet to be provided to the source processing element based on the tail field being equal to or greater than the limit field to indicate that the message queue is full and that the designation processing element rejected the transmitted message.
The messaging facility of the present invention can include multiple message queues in the local memory of the destination processing element's local memory. The number of message queues in the destination processing element's local memory is preferably defined by software running on the destination processor. In one embodiment, an interrupt status register stores a multiple messages bit to indicate if the tail field is equal to the interrupt location indicated by the threshold field in multiple message queues.
In the message queue according the present invention, the size of each message queue is variable and the local microprocessor does not have to be interrupted with each message arrival. Yet, the messaging facility according to the present invention still provides an efficient one-way transversal of the interconnect network, and does not require the complicated locking mechanism such as is required with a shared memory message queue. While not requiring dedicated hardware message queues, the present invention provides flexibility in that the local processor can poll its queue to determine when a message has arrived. Alternatively, the flexibility of the messaging facility according to the present invention permits an interrupt to be generated in a variety of situations such as at the receipt of each message, at any specified number of messages, or when the message queue is full. Furthermore, the messaging facility according to the present invention can provide an arbitrary number of queues in each processing element.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a simplified model of a massively parallel processing (MPP) system.
FIG. 2 is a simplified block of a MPP system.
FIG. 3 is a block diagram of a processing element included in the MPP system of FIGS. 1 and 2.
FIG. 4 is diagram illustrating a portion of a message queue used in the MPP system of FIGS. 1 and 2.
FIG. 5 is a diagram of message queue control word used to define the message queue of FIG. 4.
FIG. 6 is a diagram illustrating a message send E-register command used to transfer a message from source E-registers to a slot in a destination message queue.
FIG. 7 is a diagram illustrating a Message address register format.
FIG. 8 is a diagram illustrating generating the message slot global virtual address (GVA).
FIG. 9A illustrates a message queue before a message store in a scenario where a tail field value is less than a threshold field value.
FIG. 9B illustrates a message queue after the message store the scenario where a tail field value is less than a threshold field value.
FIG. 10A illustrates a message queue before a message store in a scenario where a tail field value is equal to a threshold field value.
FIG. 10B illustrates a message queue after the message store the scenario where a tail field value is equal to a threshold field value.
FIG. 11A illustrates a message queue before a message store in a scenario where a tail field value is greater than a threshold field value.
FIG. 11B illustrates a message queue after the message store the scenario where a tail field value is greater than a threshold field value.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.
MPP SYSTEM
FIG. 1 illustrates a simplified model of the components of a massively parallel processing (MPP) system generally indicated at 20. MPP system 20 typically comprises hundreds or thousands of processing element nodes, such as processing element node 22. An interconnect network 24 provides communication paths among the processing element nodes in MPP system 20. In the model of system 20 illustrated in FIG. 1, interconnect network 24 forms a three-dimensional matrix of paths which connect the processing element nodes in the x, y, and z dimensions as indicated by arrows 26.
A simplified block diagram of MPP system 20 is illustrated in FIG. 2. As illustrated in FIG. 2, each of the N processing elements, such as processing element 22, in MPP system 20 include a processor 30 connected to a local memory 32. In one embodiment, MPP system 20 is a MIMD massively parallel processing system with a physically distributed, globally addressable memory. Each processor 30 has a favored, low latency, high bandwidth path to an associated local memory 32, and a longer latency lower bandwidth access to the memory banks 32 associated with other processors 30 over the interconnect network 24.
In the preferred embodiment, the interconnect network 24 is comprised of a three-dimensional torus which, when connected, creates a three-dimensional matrix of processing elements. The torus design has several advantages, including high speed information transfers and the ability to avoid bad communication links. A toroidal interconnect network is also scalable in all three dimensions. Interconnect networks of this nature are described in more detail in the following co-pending and commonly assigned U.S. patent applications: U.S. Pat. No. 5,533,198, entitled "DIRECTION ORDER ROUTING IN MULTIPROCESSING SYSTEMS," by Gregory M. Thorson, filed Nov. 30, 1992 and U.S. Pat. No. 5,701,416, entitled "ADAPTIVE ROUTING MECHANISM FOR TORUS INTERCONNECTION NETWORK," by Thorson et al., filed Apr. 13, 1995, both of which are incorporated herein by reference.
FIG. 3 illustrates a simplified block diagram of one embodiment of processor element 22. In this embodiment, processor element 22 includes a high-performance reduced instruction set computer (RISC) microprocessor 30 such as the EV5 microprocessor (DEC P/N 21164) or the EV56 microprocessor (DEC P/N 21164A), both available from Digital Equipment Corporation. Each microprocessor 30 is coupled to a local memory 32 that is a distributed portion of the globally-addressable system memory. Local memories 32 can be formed, for instance, by stacking a plurality of dynamic random access memories (DRAM) integrated circuits 34 configured as banks of memory operating under control of memory bank controllers 36. Microprocessor 30 is also connected to a nonvolatile memory 38, such as an erasable programmable read only memory (EPROM) used to store relatively unchanging information such as, for instance, boot program code.
Each processing element 22 further includes shell circuitry 40 that implements synchronization and communication functions facilitating interactions between processor elements 30. Shell circuitry 40 includes external registers circuit 42 and an interconnection network router 44 used to connect multiple processing elements 22 in a three-dimensional toroidal "fabric." The toroidal interconnect network carries all data communicated between processing elements 22 and those memories 32 that are not local to the particular processing element 22. In one embodiment, external registers circuit 42 permits asynchronous (i.e., independent of the local processor) movement of data, including blocks of data, between processor 30 and remote memories 32 associated with other processing elements 22 and between local memory 32 and remote processors 30 associated with other processing elements 22. In addition, external registers circuit 42 is optionally used to provide flexible addressing modes that permit a high degree of control over the redistribution of data between shared portions of the system memory.
As is illustrated FIG. 3, shell circuitry 40 also includes an input/output circuit 46 used to communicate over a high-speed channel with one or more input/output devices (not shown). Such a high-speed channel is described a commonly assigned patent application Ser. No. 08/614,854, entitled "RECONFIGURABLE RING BASED NETWORK SYSTEM," filed on even date herewith, and which is herein incorporated by reference.
In the embodiment of processing element 22 illustrated in FIG. 4, microprocessor 30 is connected through external register circuit 42 to memory 32. External register circuit 42 includes a plurality of external registers (E-registers) 48. Microprocessor 30 is capable of reading from and writing to E-registers 48 via external data path 50 and external address path 52. Likewise, external register circuit 42 is capable of reading from and writing to memory 32 via memory data path 54 and memory address path 56.
As mentioned above, microprocessor 30 can be implemented in a commodity microprocessor, such as the EV5 or the EV56 available from Digital Equipment Corporation. Commodity microprocessor implementations impose limitations on the types of application behaviors that can be supported efficiently. External register circuit 42 provides a means to extend the capabilities of the commodity microprocessor with minimal requirements imposed on microprocessor 30.
E-registers 48 are mapped directly into the memory space of microprocessor 30. This provides high-bandwidth and low-latency load and store access to the registers themselves. In addition, in one embodiment, special memory-mapped store operations are used to provoke the E-registers to reference normal memory locations. These special operations include the ability to move data to/from E-registers 48 and memory locations in memory 32, to perform memory synchronization operations, initiate message transfers (to, for instance, other microprocessors), and to perform local operations that otherwise cannot be performed quickly within microprocessor 30. E-registers 48 serve as source or destination for additional operands and can be used to store the results of operations on operands supplied by microprocessor 30 or even for the results of operations on those operands previously stored in E-registers 48. More detail of the description, use, and advantages of E-registers is described in a commonly assigned patent application Ser. No. 08/615,671, entitled "METHOD AND APPARATUS FOR EXTENDING MEMORY REFERENCE CAPABILITIES OF A MICROPROCESSOR," filed on even date herewith, and which is herein incorporated by reference.
In one embodiment of the processing element illustrated in FIG. 3, a memory location in memory 32 local to a particular processing element can be accessed in two ways: a direct (cacheable) load or store by the processor, or an indirect (non-cacheable) operation through the external registers (E-registers) 48 in external register circuit 42. The virtual address used to access the memory location can be the same in both cases. Processor load and store instructions can reference only local memory. An on-chip DTB translates the virtual addresses to local physical addresses when the processor directly issues loads and stores. Shell circuitry 40 translates the virtual address to a (either local or remote) physical address when the processor issues E-register GET and PUT references.
Information is transferred between the processing element nodes 22 via the interconnect network 24 in the form of a packet. Each packet preferably comprises a header. The header includes routing information that steers the packet through the network, destination information that indicates which processing element should receive the packet, and control information that instructs the processing element that receives the packet to perform an operation. The header may also contain source information that indicates which processing element created the packet and may contain memory address information. A packet optionally comprises a body. The body of a packet typically includes data to be transferred between processing element nodes or system data from or to a host system.
The network routers, such as network router 44, at each processing element node 22 transfer packets through communication links in interconnect network 24 based on the routing information contained in each header of each packet. The header and the body have variable lengths and transfer over the communication links one phit at a time. MPP system 20 preferably uses wormhole routing. Preferred methods of routing the packets with wormhole routing are described in the above incorporated co-pending and commonly assigned U.S. patent application Ser. No. 07/983,979, entitled "DIRECTION ORDER ROUTING IN MULTIPROCESSING SYSTEMS," by Gregory M. Thorson, filed Nov. 30, 1992 and U.S. patent application Ser. No. 08/421,566, entitled "ADAPTIVE ROUTING MECHANISM FOR TORUS INTERCONNECTION NETWORK," by Thorson et al., filed Apr. 13, 1995.
MESSAGING FACILITY
Messaging Components
A message is a packet of information that one processing element, such as processing element 22, sends to another processing element. After receiving a message, shell circuitry 40 in the destination processing element stores the message in a designated location in the associated local memory 32 of the destination processing element. This designated location in memory 32 is referred to as a message queue, such as message queue 60 shown in FIG. 2. When message queue 60 contains more than a specified amount of messages, as specified by software, shell circuitry 40 interrupts microprocessor 30, if software enables the interrupt. Microprocessor 30 then reads the messages from message queue 60.
The messaging facility of the present invention utilizes several hardware and software components. These components are described in more detail below and include: the message queue; a message queue control word (MQCW); a software-maintained head pointer; a SEND E-register command; a interrupt status (IR-- STATUS) register; and a message address (MSG-- ADDR) register.
Referring to FIG. 2, message queue 60 is a portion of local memory 32 that software designates as the destination for messages. Software defines at least one message queue, and possibly several individual message queues for each processing element 22. Software also defines the size of each message queue. Message queue 60, for example, stores from one to 2,097,151 messages in one embodiment of the present invention.
One form of message queue 60 is illustrated generally in FIG. 4. In this embodiment, each message comprises 64 bytes of information. Each 64-byte address location in the message queue 60 is referred to as a slot, such as slot 64 illustrated in FIG. 4. In the embodiment illustrated, the MQCW is a 64-bit word of information that software stores in bits 63:0! of a first slot 62 in message queue 60. Shell circuitry 40 and software use the information stored in the MQCW to perform messaging functions.
FIG. 5 illustrates a typical format of one embodiment of a MQCW illustrated generally at 70. MQCW 70 comprises four fields including: a limit field 72, a tail field 74, a threshold field 76, and a signal (SIG) bit 78. Limit field 72, tail field 74, and threshold field 76 represent offset values with respect to the address of MQCW 70.
Limit field 72 indicates a number of slots in message queue 60. For example, when limit field 72 is set to 50, message queue 60 contains 50 slots for 50 messages. Software sets the limit field value of limit field 72, which in one embodiment varies from between one and 2,097,151 to permit message queue 60 to store from one to 2,097,151 messages.
Tail field 74 indicates a slot, such as slot 64, in message queue 60 which is to receive the next message. When shell circuitry 40 receives a message, the shell circuitry stores the message in the slot pointed to by the tail field 74 of the MQCW. After storing the message in the indicated slot, shell circuitry 40 increments the tail field by one. Thus, shell circuitry 40 atomically reads and updates the tail field. In this atomic operation, the tail field is read to determine where to store the transmitted message and updated to point to a second location in the message queue where a next transmitted message is to be stored.
Threshold field 76 indicates which slot, such as slot 64, is to cause shell circuitry 40 to set an interrupt for the associated microprocessor 30. For example, when threshold field 76 is set to 30, shell circuitry 40 sets an interrupt for the associated microprocessor 30 when tail field 74 is equal to 30.
The state of SIG bit 78 indicates whether tail field 74 is equal to threshold field 76. Shell circuitry 40 sets SIG bit 78 to a one, when tail field 74 is equal to threshold field 76. Software resets SIG bit 78 to zero by writing a zero to the MQCW. Because hardware modifies MQCW 70 in an atomic read-modify-write operation, software manipulation of MQCW 70 does not conflict with the hardware updates of the MQCW.
Each microprocessor 30 maintains its own head pointer with software. The software-maintained head pointer value indicates the slot, such as slot 64, of the associated message queue 60 that is to be next read by the software of the local microprocessor 30. MPP system 20 hardware does not use, modify, or provide storage for the head pointer.
After reading a message from message queue 60, software controlling the local microprocessor 30 increments the head pointer to point to the next slot in the message queue. When the head pointer points to the same slot as the tail pointer indicated by tail field 74, software has then read all of the messages out of message queue 60.
FIG. 6 illustrates a message send (SEND) E-register command used to transfer a 64-byte (8-word) message from eight contiguous source E-registers of E-registers 48 to a 64-byte slot 64 in a destination processing element message queue 60. In FIG. 6, the "e" represents the E-register address, where the eight contiguous source E-registers begin at E-register address e and run through E-register address e+7. The SEND E-register command is described in more detail in the above-incorporated patent application entitled "METHOD AND APPARATUS FOR EXTENDING MEMORY REFERENCE CAPABILITIES OF A MICROPROCESSOR."
In one embodiment, the IR-- STATUS register comprises a message address valid bit and a multiple messages bit used for messaging. The message address valid bit stored in the IR-STATUS register indicates that the tail value indicated by tail field 74 for a given message queue 60 is equal to the threshold value indicated by threshold field 76 for the given message queue 60. The given message queue 60 is identified by the MSG-- ADDR register discussed in more detail below. The message address valid bit remains set to one until the bit is reset by software. The multiple messages bit indicates that the tail value of tail field 74 is equal to the threshold value of threshold field 76 in more than one message queue 60. The multiple messages bit remains set to one until software resets the bit.
FIG. 7 illustrates the MSG-- ADDR register format. The MSG-- ADDR register comprises a global virtual address (GVA) of the MQCW for the particular message queue 60 that set the message address valid bit stored in the IR-- STATUS register. The GVA in the MSG-- ADDR register remains locked in the MSG-- ADDR register until software resets the message address valid bit of the IR-- STATUS register. Shell circuitry 40 stores a new GVA into the register the next time shell circuitry 40 sets the message address valid bit to one.
Defining a Message Queue
Before a destination processing element 22 can receive messages, software must define at least one message queue 60 for that processing element 22. Software defines a given message queue 60 by storing a MQCW to a location in local memory 32. The values in the MQCW fields define the given message queue 60.
For example, software sets limit field 72 of MQCW 70 to 50 to define a message queue 60 which is capable of storing as many as 50 messages. Software sets tail field 74 to a value greater than zero. If software were to set tail field 74 to zero, the first message would overwrite MQCW 70 because tail field 74 is an offset with respect to the MQCW address. Normally, software initially sets tail field 74 to one.
When software defines a message queue 60, threshold field 76 is set to any number greater than zero up to the limit field value. The threshold value stored in threshold field 76 must be at least one slot larger than the tail value stored in tail field 74 to generate an interrupt.
Sending a Message
Referring to FIG. 6, before sending a message, software stores data for a 64-byte message in eight contiguous E-registers of E-registers 48 that align on 64-byte boundary in the E-register address space. In one embodiment, software uses STORE E-register commands, to transfer data from a source microprocessor 30 to the E-registers 48. The STORE E-register command is described in more detail in the above-incorporated patent application entitled "METHOD AND APPARATUS FOR EXTENDING MEMORY REFERENCE CAPABILITIES OF A MICROPROCESSOR."
After storing data representing the message to be sent in the eight source E-registers of E-registers 48, software issues a SEND E-register command. The SEND command signals shell circuitry 40 in the source processing element 22 to send the stored message to the destination processing element 22. In one embodiment, a SEND E-register command address opcode argument indicates to shell circuitry 40 that the E-register command is a SEND, and also indicates whether the SEND command references a segment translation table (STT).
If software sets the SEND E-register command address opcode argument to indicate that the SEND command will reference the STT, shell circuitry 40 adds the index offset to the base virtual address to create a user virtual address. Shell circuitry 40 references the STT to convert the user virtual address to a GVA. When it sends a message, software sets SEND E-register command address arguments so that the GVA is the GVA for the MQCW 70 in the destination processing element 22.
When software sets the SEND E-register command address opcode argument to indicate that the SEND command will not reference the STT, shell circuitry 40 adds the index offset to the base virtual address to create the GVA. When software sends a message, software sets SEND E-register command address arguments so that the GVA is the GVA for the MQCW 70 in the destination processing element 22.
After shell circuitry 40 receives all of the SEND E-register command address arguments from the associated microprocessor 30, the shell circuitry sets E-register state (ERS) codes for the eight contiguous source E-registers of E-registers 48 to indicate that the E-register is empty. Shell circuitry 40 then creates a message packet and sends the message packet to the destination processing element 22 along interconnect network 24.
Shell circuitry 40 does not change the value of the ERS codes again until it receives a SEND-accepted or a SEND-rejected response packet. When shell circuitry 40 receives a SEND-accepted packet, shell circuitry 40 sets the ERS codes to indicate that the E-register is full and no errors occurred during the transfer, which indicates that the message was accepted by the destination processing element 22. When shell circuitry 40 receives a SEND-rejected packet, the shell circuitry sets the ERS codes to indicate that the E-register is full, but that a previous message SEND command that used the E-register was rejected, which indicates that the message queue was full and the message was not accepted by the destination processing element 22. When this condition occurs, software can resend the message.
The above described embodiment uses E-registers to assemble the messages in the source processing element. Nevertheless, other methods can be used to assemble the message. For example, a suitable assembly means, which comprises write buffers is described in detail in the above incorporated U.S. patent application Ser. No. 08/166,443 filed on Dec. 13, 1995 entitled "MESSAGING FACILITY FOR DISTRIBUTIVE MEMORY MASSIVELY PARALLEL PROCESSING SYSTEM."
The Ser. No. 08/166,443 application teaches that messages can be transmitted simply by writing a cache line to a special remote address. Messages are distinguished from ordinary remote cache-line writes by the mapping of the destination address. Special addresses are reserved in the physical memory address range to be interpreted as message transmission triggers. If a store is performed to the special address, hardware decodes the destination address to indicate that the store is actually a message send. The processing element shell circuitry then creates a message packet and sends the packet into the interconnection network for routing to the destination processing element.
In the MPP system described in the Ser. No. 08/166,443 application, messages are assembled in circuits on the microprocessor chip called "write buffers." The cache line write to the special address must be performed as sequential stores to addresses within the same cache line. The write buffers on the microprocessor attempt to assemble full cache lines before requesting a bus cycle and will succeed in most cases if no intervening memory references are made while sending a message. If the processor is interrupted while constructing the cache line message, the message will be sent as two incomplete cache lines in two separate messages (called message fragments). The half-word mask field in the message header must be examined by software upon receipt of a message to see if the entire message is present. If not, the receiving processor must defer processing of the message fragment until the remaining portions arrive.
In this embodiment, it is important to have an understanding of the functioning of the microprocessor write buffers in order to ensure proper message transmission. There are 4 buffers, each a cache-line in size. At every store, the microprocessor checks all 4 buffers for a possible match between the address of the current store and one that may be pending in the write buffers. If a match is found, the write data is merged into its proper place in the cache line as indicated by the word-within-cache-line bits in the address. If there was data at that spot in the write buffer from a previous store with the same cache line address and word-within-cache-line pointer, the previous data is lost.
The DECChip microprocessor used in the MPP system described in the Ser. No. 08/166,443 application keeps the data in the write buffers for an indeterminate amount of time, unless specific steps are taken to flush them to the memory subsystem. One or more write buffers will flush if: a store is made to an address that matches none of the cache-line addresses currently in the write buffers, a memory barrier instruction is issued, or an exception or interrupt occurs. Barring any of these occurrences, the write buffers will still drain eventually, but the timing of the actual write is difficult to predict.
It can be seen that if multiple messages are sequentially sent to a particular processor, different special cache line addresses must be used for any five message in a row, or unintentional merging of messages in the write buffers can occur.
Those skilled in the art can see that the message assembly functions of the write buffers could also easily be accomplished by buffers implemented external to the microprocessor if the microprocessor did not have write buffers or it was not desirable for some reason to use the on-board write buffers for message assembly.
Receiving and Storing a Message
When shell circuitry 40 in a destination processing element 22 receives a message, the shell circuitry references a location in local memory 32, which is identified by the message packet as the MQCW 70 location. After reading MQCW 70, shell circuitry 40 compares tail field 74 to limit field 72. When the tail value of tail field 74 is equal to the limit value of limit field 72, shell circuitry 40 creates a SEND-rejected packet and returns the SEND-rejected packet to the processing element 22 that created the message. The SEND-rejected packet indicates that the message queue is full and cannot accept another message.
When the tail value of tail field 74 is less than the limit value of limit field 72, shell circuitry 40 returns a SEND-accepted packet to the processing element 22 that created the message. Shell circuitry 40 then stores the message in message queue 60 in the destination processing element 22 at the slot pointed to by the tail pointer indicated by tail field 74. Shell circuitry 40 then increments the tail pointer by adding one to tail field 74 and stores the updated tail field value in the MQCW 70 in local memory 32 of the destination processing element 22.
Three example scenarios, where shell circuitry 40 in the destination processing element 22 accepts a message and stores the message in message queue 60, are described below to illustrate scenarios under the following three conditions: 1) the tail value is less than the threshold value; 2) the tail value is equal to the threshold value; and 3) the tail value is greater than the threshold value.
In the first scenario, the tail value of tail field 74 is less than the threshold value of threshold field 76. In this example scenario, for illustrative purposes limit field 72 is equal to four, tail field 74 is equal to one, and threshold field 76 is equal to three. Referring to FIG. 8, which illustrates generating the message slot GVA, shell circuitry 40 shifts the tail value six positions to the left and adds the shifted value to the GVA of the MQCW. The resulting sum is the byte-address GVA for the destination slot in the message queue. After adding the MQCW GVA to the shifted tail value, shell circuitry 40 sets bits 5:0! of the resulting address to zeros. By setting bits 5:0! to zero, a 64-byte aligned GVA is created for the destination slot in the message queue.
FIG. 9A illustrates the message queue before the message store, and FIG. 9B illustrates the message queue after the message store. Shell circuitry 40 stores the message in the slot that is identified by the message slot GVA and increments the tail value by one as illustrated in FIGS. 9A and 9B. To complete the message store operation, shell circuitry 40 compares the tail value to the threshold value. Since in this scenario, the tail value is less than the threshold value, shell circuitry 40 does not set a message interrupt for the local microprocessor 30.
In the second scenario, the tail value stored in tail field 74 is incremented and becomes equal to the threshold value stored in threshold field 76. In this scenario, initially, limit field 72 is equal to four, tail field 74 is equal to two, and threshold field 76 is three.
Again, referring to FIG. 8 to illustrate generating the message slot GVA, shell circuitry 40 shifts the tail value six positions to the left and adds the shifted value to the GVA of the MQCW. This sum represents the GVA for the destination slot in the message queue 60 in local memory 30. After the GVA of the MQCW is added to the shifted tail value, shell circuitry 40 sets bits 5:0! of the resulting address to zeros. This creates a 64-byte aligned GVA for the destination slot in the message queue.
Similar to FIGS. 9A and 9B illustrating the first scenario, FIG. 10A illustrates the message queue before a message is stored, and FIG. 10B illustrates the message queue after a message is stored for the second scenario. Shell circuitry stores the message in the slot identified by the message slot GVA and increments the tail value by one as indicated by the move of the tail pointer from slot 2 to 3 as illustrated in FIGS. 10A and 10B.
After the message is stored in the message queue, shell circuitry 40 compares the tail value to the threshold value. Since in the second scenario the tail value is equal to the threshold value after the tail value is incremented as a result of the message store, shell circuitry 40 sets SIG bit 78 of the MQCW for the message queue to one to indicate that the tail value is equal to the threshold value.
Shell circuitry 40 then examines the value of the message address valid bit stored in the IR-- STATUS register. When the message address valid bit is set to zero, shell circuitry 40 sets the message address valid bit to one, stores the GVA for the MQCW in a MSG-- ADDR register, and locks the MSG-- ADDR register. If software enables the interrupt, the message address value bit equal to one condition sets an interrupt for the microprocessor. When the message address valid bit is set to one, shell circuitry 40 checks the value of the multiple messages bit of the IR-- STATUS register. If the multiple message bit is set to zero, shell circuitry 40 sets the multiple messages bit to one. In either case, after the shell circuitry examines the value of the message address valid bit of the IR-- STATUS register, and performs a necessary operation based on the state of this bit, the message store operation is complete.
In the third scenario, the tail value of tail field 74 is greater than the threshold value of threshold field 76. For illustrative purposes, in this scenario, limit field 72 is four, tail field 74 is three, and threshold field 76 is two.
Referring back to FIG. 8, shell circuitry 40 shifts the tail value six positions to the left and adds this value to the GVA of the MQCW. The resulting sum is a GVA for the destination slot in the message queue 60 of the local memory 32. After the GVA of the MQCW is added to the shifted tail value, shell circuitry 40 sets bits 5:0! of the resulting address to zero. Setting bits 5:0! to zero creates a 64-byte aligned GVA for the destination slot in the message queue.
As with FIGS. 9A and 9B for the first scenario and as with FIGS. 10A and 10B for the second scenario, FIG. 11A illustrates the message queue before a message store, and FIG. 11B illustrates the message queue after a message store for the third scenario. Shell circuitry 40 stores the message in the slot as identified by the message slot GVA and increments the tail value by one as illustrated in FIGS. 11A and 11B. After the message is stored, shell circuitry 40 compares the tail value to the threshold value. Since in this example the tail value is greater than the threshold value, shell circuitry 40 does not set a message interrupt for the microprocessor. As a result, shell circuitry 40 does not modify SIG bit 78 of the MQCW. Thus, unless SIG bit 78 has been reset by software, the SIG bit for the message queue remains set to one. Similarly, if SIG bit 78 has been reset by software, the SIG bit remains set to zero.
Shell circuitry 40 does not modify the message address valid bit or the multiple messages bit of the IR-- STATUS register. Therefore, the message address valid bit and/or the multiple messages bit of the IR-- STATUS register remains set to one, if they have not been reset by software. If these bits have been reset by software, these bits remain set to zero. In any case, this completes the message store operation for the third scenario.
Reading a Message
As with storing a message in the message queue, reading a message from the message queue can best be illustrated with a variety of scenarios which show how software reads messages out of the message queues. These scenarios provide examples for illustrative purposes only, and therefore, the example scenarios do not necessarily reflect the actual methods used by software in the preferred embodiment. In all of these scenarios, software enables the message address valid and the multiple messages interrupts in the IR-- STATUS register. Alternatively, software polls the MQCW tail pointer stored in tail field 74 to determine when messages have arrived.
When software enables the message address valid and the multiple messages interrupts in the IR_STATUS register, three scenarios exist when the local microprocessor 30 receives an interrupt from shell circuitry 40 which include: 1) the message address valid bit is one and the multiple messages bit is zero; 2) the message address valid bit is one and the multiple messages bit is one; and 3) the message address valid bit is zero and the multiple message bit is one.
In the first scenario, the message address valid bit is equal to one and the multiple messages bit is equal to zero. The following steps describe a process software can use to read messages out of the message queue. In this scenario for illustrative purposes, software defines a message queue with limit field 72 set to 30, tail field 74 set to one, and threshold field 76 set to 25.
Initially, the local microprocessor 30 receives an interrupt from shell circuitry 40. The operating system software running in microprocessor 30 reads the value of the IR-- STATUS register to obtain the values of the message address valid interrupt and the multiple messages interrupt bits. The operating system software then reads the value stored in the MSG-- ADDR register, which represents the value of the GVA of the message queue that caused shell circuitry 40 to set the message address valid interrupt.
The operating system software then resets the message address valid interrupt to zero by writing a one to the message address valid bit of the IR-- STATUS register.
The operating system software then signals user software that a message queue contains messages and provides user software with the GVA of the MQCW. User software then performs an atomic SWAP command on the MQCW. The swaperand value of the SWAP command comprises new MQCW values which provide a limit field 72 of 60, a tail field 74 of 31, a threshold field 76 of 55, and a SIG bit 78 of zero.
User software then retrieves the value of the software-contained head pointer for the message queue. The head pointer indicates the slot location of the next message to be read out of the message queue. User software then sends a command to shell circuitry 40 to read the message that is referenced by the head pointer. The command can be any command that accesses local memory 32. For example, the command can be a direct local memory load command or a get vector E-register command.
User software then processes the message. After processing the message, user software increments the value of the head pointer by one slot and compares the value of the head pointer to the tail pointer. When the head pointer is less than the tail pointer, user software again sends a command to shell circuitry 40 to read the message that was referenced by the head pointer, processes the message, and increments the value of the head pointer by one slot. User software repeats this process until, after being incremented by one slot, the head pointer is equal to the tail pointer to indicate that user software has read all the messages out of the message queue.
The next time microprocessor 30 receives an interrupt due to this message queue exceeding the threshold value, operating system and user software repeats the above steps, except that when the SWAP command is performed, user software sets the new MQCW values to set limit field 72 to 30, tail field 74 to one, threshold field 76 to 25, and SIG bit 78 to zero. This process creates two virtual message queues that use the same MQCW.
In the second scenario, the message address valid interrupt bit is equal to one and the multiple messages interrupt bit is equal to one. The following steps describe a process software can use to read messages out of the message queue for this scenario. For illustrative purposes, user software defines multiple message queues and each message queue is initially set to have limit field 72 equal to 30, and threshold field 76 equal to 25.
Initially, microprocessor 30 receives an interrupt from shell circuitry 40. Operating system software running in the local microprocessor 30 reads the value of the IR-- STATUS register to read the message address valid interrupt bit and the multiple messages bit. Operating system software then reads the value stored in the MSG-- ADDR register to obtain the value of the GVA of the first message queue that causes shell circuitry 40 to set the message interrupt. Operating system software resets the message address valid and multiple messages interrupts to zero by writing a one to the message address valid and multiple messages bits of the IR-- STATUS register. The operating system software then signals the user software that multiple message queues contain messages and provides user software with the GVA of the first MQCW. For this first MQCW, SIG bit 78 is set to one.
User software reads the MQCW from memory 32 and examines SIG bit 78. When SIG bit 78 is set to zero, software does not read messages from that message queue and instead retrieves the GVA for the MQCW of another message queue. When SIG bit 78 is set to one, software continues with the following steps.
User software performs an atomic SWAP command on the MQCW. The swaperand value for the SWAP command comprises new MQCW values which set limit field 72 to 60, tail field 74 to 31, threshold field 76 to 55, and SIG bit 78 to zero.
User software then retrieves the value of the software-contained head pointer for this message queue. The head pointer indicates a slot location and the next message to be read out of the message queue. User software then sends a command to shell circuitry 40 to read the message that is referenced by the head pointer. Again, as with the first scenario, this command may be any command that accesses local memory such as a direct local memory load command or a get vector E-register command.
User software then processes the message. After processing the message, user software increments the value of the head pointer by one slot and compares the value of the head pointer to the tail pointer. When the head pointer is less than the tail pointer, user software again sends a command to shell circuitry 40 to read the message that is referenced by the head pointer, processes the message, and increments the value of the head pointer by one slot, and compares the value of the head pointer to the tail pointer. User software repeats these steps until the head pointer is equal to the tail pointer, at which point, user software has read all the messages out of the message queue.
User software then reads the second MQCW from memory and examines the SIG bit for the next defined message queue. The above steps performed for the first message queue are repeated for the second message queue until user software has read all the messages out of the second message queue as indicated when the head pointer is equal to the tail pointer. The above steps are repeated by user software for each message queue defined in the given processing element 22.
In the third scenario, the message address valid interrupt bit is equal to zero and the multiple messages bit is equal to one. When this scenario occurs, operating system software and user software running in the local microprocessor 30 perform the same functions and steps described above for the second scenario where the message address valid bit is equal to one and the multiple messages bit is equal to zero, except that the MSG-- ADDR register does not contain a valid GVA. Because the MSG-- ADDR register does not contain a valid GVA, the operating system software does not read the value stored in the MSG-- ADDR register representing the value of the GVA of the first message queue that causes shell circuitry 40 to set the message interrupt. In addition, instead of the operating system software signalling user software that multiple message queues contain messages and providing user software with the GVA of the first MQCW, the operating system software instead only signals user software that multiple message queues contain messages.
CONCLUSION
Thus, the messaging facility according to the present invention provides an arbitrary number of queues in each processing element. In addition, the size of each message queue is variable. Furthermore, the operating system operating on the local microprocessor does not have to be interrupted with each message arrival. However, the messaging facility according to the present invention still provides an efficient one-way transversal of the network of a MPP system, and does not require the complicated locking mechanism such as is required with a shared memory message queue. While not requiring dedicated hardware message queues, the present invention provides flexibility in that the local processor can poll its queue to determine when a message has arrived. Alternatively, the flexibility of the messaging facility according to the present invention permits an interrupt to be generated in a variety of situations such as at the receipt of each message, at any specified number of messages, or when the message queue is full. In other words, the threshold value of the threshold field in the MQCW permits great flexibility in deciding when the local microprocessor 30 is to be interrupted in response to messages being read into an associated message queue.
Although specific embodiments have been illustrated and described herein for purposes of description of the preferred embodiment, it will be appreciated by those of ordinary skill in the art that a wide variety of alternate and/or equivalent implementations calculated to achieve the same purposes may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. Those with skill in the mechanical, electromechanical, electrical, and computer arts will readily appreciate that the present invention may be implemented in a very wide variety of embodiments. This application is intended to cover any adaptations or variations of the preferred embodiments discussed herein. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.

Claims (20)

What is claimed is:
1. A messaging facility in a multiprocessor computer system having a plurality of processing elements interconnected by an interconnect network, each processing element including a processor and a local memory, wherein globally addressable portions of local memory of each processing element form a distributed memory, the messaging facility comprising:
assembly means in a source processing element for assembling a message to be sent from the source processing element to a destination processing element based on information provided from the source processing element's processor;
a network router for transmitting the assembled message from the source processing element to the destination processing element via the interconnect network;
a message queue in the local memory of the destination processing element for storing the transmitted message, wherein a control word stored in the local memory of the destination processing element includes a limit field designating a size of the message queue and a tail field designating an index into the corresponding message queue to indicate a first location in the message queue where the transmitted message is to be stored; and
shell circuitry in the destination processing element for atomically reading the tail field to determine where to store the transmitted message and updating the tail field to point to a second location in the message queue where a next transmitted message is to be stored.
2. The messaging facility of claim 1 further comprising interrupt means, and wherein the control word further includes a threshold field designating an interrupt location in the message queue, and wherein the interrupt means generates an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field.
3. The messaging facility of claim 1 wherein the destination processor maintains a head pointer designating an index into the corresponding message queue to indicate a location where the message is stored in the message queue for reading by the destination processing element's processor.
4. The messaging facility of claim 1 further comprising at least a second message queue in the local memory of the destination processing element's local memory.
5. The messaging facility of claim 4 wherein the control word further includes a threshold field designating an interrupt location in the message queue, and wherein the messaging facility further comprises:
interrupt means for generating an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field; and
multiple message status means for indicating if the tail field is equal to the interrupt location indicated by the threshold field in multiple message queues.
6. The messaging facility of claim 1 wherein the assembly means comprises a plurality of external registers, and wherein the message is assembled in an aligned block of external registers within the plurality of external registers.
7. The messaging facility of claim 1 wherein the assembly means comprises a plurality of write buffers internal to the source processing element's processor.
8. The messaging facility of claim 1 wherein the destination processing element includes polling means for polling the control word stored in the local memory of the destination processing element to ascertain if the transmitted message has been stored in the message queue.
9. The messaging facility of claim 1 wherein the shell circuitry in the destination processing element generates a send-accepted packet to be provided to the source processing element based on the tail field being less than the limit field to indicate that the designation processing element accepted the transmitted message.
10. The messaging facility of claim 1 wherein the shell circuitry in the destination processing element generates a send-rejected packet to be provided to the source processing element based on the tail field being equal to or greater than the limit field to indicate that the message queue is full and that the designation processing element rejected the transmitted message.
11. A method of passing messages in a multiprocessor computer system having a plurality of processing elements interconnected by an interconnect network, each processing element including a processor and a local memory, wherein globally addressable portions of local memory of each processing element form a distributed memory, the method comprising the steps of:
assembling a message to be sent from a source processing element to a destination processing element based on information provided from the source processing element's processor;
transmitting the assembled message from the source processing element to the destination processing element via the interconnect network; and
defining a message queue in the local memory of the destination processing element for storing the transmitted message;
defining a control word which includes a limit field designating a size of the message queue and a tail field designating an index into the corresponding message queue to indicate a first location in the message queue where the transmitted message is to be stored;
storing the control word in the local memory of the destination processing element; and
atomically reading the tail field to determine where to store the transmitted message and updating the tail field to point to a second location in the message queue where a next transmitted message is to be stored.
12. The method of claim 11 wherein the control word is further defined to include a threshold field designating an interrupt location in the message queue, and wherein the method further includes the step of generating an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field.
13. The method of claim 11 further comprising the step of maintaining a head pointer designating an index into the corresponding message queue to indicate a location where the message is stored in the message queue for reading by the destination processing element's processor.
14. The method of claim 11 further comprising the step of defining at least a second message queue in the local memory of the destination processing element's local memory.
15. The method of claim 14 wherein the control word is further defined to include a threshold field designating an interrupt location in the message queue, and wherein the method further includes the steps of:
generating an interrupt for the destination processing element's processor if the location indicated by the tail field is equal to the interrupt location indicated by the threshold field; and
indicating if the tail field is equal to the interrupt location indicated by the threshold field in multiple message queues.
16. The method of claim 11 wherein the assembling step includes the step of assembling the message in an aligned block of external registers within a plurality of external registers.
17. The method of claim 11 wherein the assembling step includes the step of writing the message to a plurality of write buffers internal to the source processing element's processor.
18. The method of claim 11 further comprising the step of polling the control word stored in the local memory of the destination processing element to ascertain if the transmitted message has been stored in the message queue.
19. The method of claim 11 further comprising the step of generating a send-accepted packet in the destination processing element to be provided to the source processing element based on the tail field being less than the limit field to indicate that the designation processing element accepted the transmitted message.
20. The method of claim 11 further comprising the step of generating a send-rejected packet in the destination processing element to be provided to the source processing element based on the tail field being equal to or greater than the limit field to indicate that the message queue is full and that the designation processing element rejected the transmitted message.
US08/615,694 1996-03-13 1996-03-13 Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory Expired - Lifetime US5841973A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US08/615,694 US5841973A (en) 1996-03-13 1996-03-13 Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US08/615,694 US5841973A (en) 1996-03-13 1996-03-13 Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory

Publications (1)

Publication Number Publication Date
US5841973A true US5841973A (en) 1998-11-24

Family

ID=24466461

Family Applications (1)

Application Number Title Priority Date Filing Date
US08/615,694 Expired - Lifetime US5841973A (en) 1996-03-13 1996-03-13 Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory

Country Status (1)

Country Link
US (1) US5841973A (en)

Cited By (87)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6055605A (en) * 1997-10-24 2000-04-25 Compaq Computer Corporation Technique for reducing latency of inter-reference ordering using commit signals in a multiprocessor system having shared caches
US6088771A (en) * 1997-10-24 2000-07-11 Digital Equipment Corporation Mechanism for reducing latency of memory barrier operations on a multiprocessor system
US6173375B1 (en) * 1997-02-28 2001-01-09 Lucent Technologies Inc. Method for accessing a shared resource in a multiprocessor system
US20020152373A1 (en) * 2000-09-13 2002-10-17 Chih-Tang Sun Tunnel interface for securing traffic over a network
US20030093626A1 (en) * 2001-11-14 2003-05-15 Fister James D.M. Memory caching scheme in a distributed-memory network
US6629152B2 (en) * 1998-06-29 2003-09-30 International Business Machines Corporation Message passing using shared memory of a computer
US20030223418A1 (en) * 2002-06-04 2003-12-04 Sachin Desai Network packet steering
US20030223361A1 (en) * 2002-06-04 2003-12-04 Zahid Hussain System and method for hierarchical metering in a virtual router based network switch
US20040024771A1 (en) * 2002-08-01 2004-02-05 Oracle International Corporation Buffered message queue architecture for database management systems with transactional enqueue support
US6704833B2 (en) * 2002-01-04 2004-03-09 Hewlett-Packard Development Company, L.P. Atomic transfer of a block of data
US20040078621A1 (en) * 2002-08-29 2004-04-22 Cosine Communications, Inc. System and method for virtual router failover in a network routing system
US20040095934A1 (en) * 2002-11-18 2004-05-20 Cosine Communications, Inc. System and method for hardware accelerated packet multicast in a virtual routing system
US20050093357A1 (en) * 2003-10-29 2005-05-05 Mitsumasa Takeda Structure of backrest assembly for standing ride type driver's platform in industrial vehicle
US20050125430A1 (en) * 2002-08-01 2005-06-09 Oracle International Corporation Asynchronous actions using flashback
US20050198422A1 (en) * 2003-12-18 2005-09-08 Arm Limited Data communication mechanism
US6976260B1 (en) * 1999-09-24 2005-12-13 International Business Machines Corporation Method and apparatus for serializing a message queue in a multiprocessing environment
US20060026358A1 (en) * 2004-07-29 2006-02-02 International Business Machines Corporation Method and apparatus of supporting cacheable registers
US20060080512A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Graphics processor with snoop filter
US20060080398A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Direct access of cache lock set data without backing memory
US20060080511A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Enhanced bus transactions for efficient support of a remote cache directory copy
US20060080508A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Snoop filter directory mechanism in coherency shared memory system
US20060095672A1 (en) * 2003-06-30 2006-05-04 Microsoft Corporation System and method for parallel execution of data generation tasks
US20060098022A1 (en) * 2003-06-30 2006-05-11 International Business Machines Corporation System and method for transfer of data between processors using a locked set, head and tail pointers
US20060140122A1 (en) * 2004-12-28 2006-06-29 International Business Machines Corporation Link retry per virtual channel
US20060140188A1 (en) * 2004-12-28 2006-06-29 International Business Machines Corporation Self-healing link sequence counts within a circular buffer
US20060149787A1 (en) * 2004-12-30 2006-07-06 Kapil Surlaker Publisher flow control and bounded guaranteed delivery for message queues
US20060190660A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Data ordering translation between linear and interleaved domains at a bus interface
US20060190662A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Transaction flow control mechanism for a bus bridge
US20060190668A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Computer system architecture
US20060190655A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Apparatus and method for transaction tag mapping between bus domains
US20060190667A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Pipeline bit handling circuit and method for a bus bridge
US20060190659A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corportion Computer system bus bridge
US20060190651A1 (en) * 2005-02-24 2006-08-24 International Business Machiness Corporation Method and system for ordering requests at a bus interface
US7111072B1 (en) * 2000-09-13 2006-09-19 Cosine Communications, Inc. Packet routing system and method
US7174372B1 (en) 2000-09-13 2007-02-06 Fortinet, Inc. System and method for managing router metadata
US7177311B1 (en) 2002-06-04 2007-02-13 Fortinet, Inc. System and method for routing traffic through a virtual router-based network switch
US20070055825A1 (en) * 2001-02-24 2007-03-08 Blumrich Matthias A Managing coherence via put/get windows
US20070101341A1 (en) * 2005-10-07 2007-05-03 Oracle International Corporation Event locality using queue services
US20070104119A1 (en) * 2000-09-13 2007-05-10 Fortinet, Inc. System and method for managing and provisioning virtual routers
US20070115979A1 (en) * 2004-11-18 2007-05-24 Fortinet, Inc. Method and apparatus for managing subscriber profiles
US20070121659A1 (en) * 2005-11-30 2007-05-31 Broadcom Corporation Ring-based cache coherent bus
US7275124B2 (en) 2005-02-24 2007-09-25 International Business Machines Corporation Method and system for controlling forwarding or terminating of a request at a bus interface based on buffer availability
US20080002811A1 (en) * 2006-06-29 2008-01-03 Allison John W Treatment delivery optimization
US7340535B1 (en) 2002-06-04 2008-03-04 Fortinet, Inc. System and method for controlling routing in a virtual router system
US7346910B1 (en) * 2000-05-26 2008-03-18 International Business Machines Incorporation Administration of groups of computer programs, data processing systems, or system resources
US7376125B1 (en) 2002-06-04 2008-05-20 Fortinet, Inc. Service processing switch
US20080117917A1 (en) * 2004-11-18 2008-05-22 Fortinet, Inc. Method and apparatus for managing subscriber profiles
US7389358B1 (en) 2000-09-13 2008-06-17 Fortinet, Inc. Distributed virtual system to support managed, network-based services
US7444398B1 (en) 2000-09-13 2008-10-28 Fortinet, Inc. System and method for delivering security services
US20090013113A1 (en) * 2007-07-03 2009-01-08 Henry Russell J Methods and systems for interprocessor message exchange between devices using only write bus transactions
US7539744B2 (en) 2000-09-13 2009-05-26 Fortinet, Inc. Network operating system for maintaining redundant master control blade management information
US7574495B1 (en) 2000-09-13 2009-08-11 Fortinet, Inc. System and method for managing interworking communications protocols
US7577794B2 (en) 2004-10-08 2009-08-18 International Business Machines Corporation Low latency coherency protocol for a multi-chip multiprocessor system
US7580373B2 (en) 2001-06-28 2009-08-25 Fortinet, Inc. Identifying nodes in a ring network
US7584390B2 (en) 1997-12-22 2009-09-01 Pact Xpp Technologies Ag Method and system for alternating between programs for execution by cells of an integrated circuit
US20090225754A1 (en) * 2004-09-24 2009-09-10 Fortinet, Inc. Scalable ip-services enabled multicast forwarding with efficient resource utilization
US7620678B1 (en) * 2002-06-12 2009-11-17 Nvidia Corporation Method and system for reducing the time-to-market concerns for embedded system design
US7650448B2 (en) 1996-12-20 2010-01-19 Pact Xpp Technologies Ag I/O and memory bus system for DFPS and units with two- or multi-dimensional programmable cell architectures
US7657861B2 (en) 2002-08-07 2010-02-02 Pact Xpp Technologies Ag Method and device for processing data
US7657877B2 (en) 2001-06-20 2010-02-02 Pact Xpp Technologies Ag Method for processing data
US20100153965A1 (en) * 2008-12-16 2010-06-17 International Buisness Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on inter-thread communications
US20100153541A1 (en) * 2008-12-16 2010-06-17 International Buisness Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on processor workload
US20100153542A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on broadcast information
US20100153966A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster using local job tables
US7782087B2 (en) 2002-09-06 2010-08-24 Martin Vorbach Reconfigurable sequencer structure
US7822968B2 (en) 1996-12-09 2010-10-26 Martin Vorbach Circuit having a multidimensional structure of configurable cells that include multi-bit-wide inputs and outputs
US7822881B2 (en) 1996-12-27 2010-10-26 Martin Vorbach Process for automatic dynamic reloading of data flow processors (DFPs) and units with two- or three-dimensional programmable cell architectures (FPGAs, DPGAs, and the like)
US7840842B2 (en) 2001-09-03 2010-11-23 Martin Vorbach Method for debugging reconfigurable architectures
US7844796B2 (en) 2001-03-05 2010-11-30 Martin Vorbach Data processing device and method
US7996827B2 (en) 2001-08-16 2011-08-09 Martin Vorbach Method for the translation of programs for reconfigurable architectures
US8058899B2 (en) 2000-10-06 2011-11-15 Martin Vorbach Logic cell array and bus system
US8099618B2 (en) 2001-03-05 2012-01-17 Martin Vorbach Methods and devices for treating and processing data
US8127061B2 (en) 2002-02-18 2012-02-28 Martin Vorbach Bus systems and reconfiguration methods
US8156284B2 (en) 2002-08-07 2012-04-10 Martin Vorbach Data processing method and device
US8209653B2 (en) 2001-09-03 2012-06-26 Martin Vorbach Router
US8230411B1 (en) * 1999-06-10 2012-07-24 Martin Vorbach Method for interleaving a program over a plurality of cells
US8250503B2 (en) 2006-01-18 2012-08-21 Martin Vorbach Hardware definition method including determining whether to implement a function as hardware or software
US8301872B2 (en) 2000-06-13 2012-10-30 Martin Vorbach Pipeline configuration protocol and configuration unit communication
USRE44365E1 (en) 1997-02-08 2013-07-09 Martin Vorbach Method of self-synchronization of configurable elements of a programmable module
US8503463B2 (en) 2003-08-27 2013-08-06 Fortinet, Inc. Heterogeneous media packet bridging
US8686549B2 (en) 2001-09-03 2014-04-01 Martin Vorbach Reconfigurable elements
US8686475B2 (en) 2001-09-19 2014-04-01 Pact Xpp Technologies Ag Reconfigurable elements
US8812820B2 (en) 2003-08-28 2014-08-19 Pact Xpp Technologies Ag Data processing device and method
US20140330919A1 (en) * 2003-03-05 2014-11-06 Websense, Inc. Asynchronous mechanism and message pool
US8914590B2 (en) 2002-08-07 2014-12-16 Pact Xpp Technologies Ag Data processing method and device
US9037807B2 (en) 2001-03-05 2015-05-19 Pact Xpp Technologies Ag Processor arrangement on a chip including data processing, memory, and interface elements
WO2017206101A1 (en) * 2016-06-01 2017-12-07 Beijing Yi Zhang Yun Feng Technology Co., Ltd. Synchronization of messages based on queues

Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1988001341A1 (en) * 1986-08-19 1988-02-25 Gerhard Haubenwallner Internal combustion engine
WO1988008652A1 (en) * 1987-04-27 1988-11-03 Thinking Machines Corporation Method and apparatus for routing message packets
US4933933A (en) * 1986-12-19 1990-06-12 The California Institute Of Technology Torus routing chip
US4965793A (en) * 1989-02-03 1990-10-23 Digital Equipment Corporation Method and apparatus for interfacing a system control unit for a multi-processor
US4980852A (en) * 1986-12-30 1990-12-25 International Business Machines Corporation Non-locking queueing mechanism for enabling a receiver device to read from a queue without access synchronization with a sending device
US4995056A (en) * 1989-01-13 1991-02-19 International Business Machines Corporation System and method for data communications
US5008882A (en) * 1987-08-17 1991-04-16 California Institute Of Technology Method and apparatus for eliminating unsuccessful tries in a search tree
US5027330A (en) * 1988-12-30 1991-06-25 At&T Bell Laboratories FIFO memory arrangement including a memory location fill indication
US5036459A (en) * 1989-03-09 1991-07-30 U.S. Philips Corporation Multi-processor computer system with distributed memory and an interprocessor communication mechanism, and method for operating such mechanism
US5068784A (en) * 1988-07-28 1991-11-26 Oki Electric Industry Co., Ltd. Method and system for monitoring the number of available buffers
US5105424A (en) * 1988-06-02 1992-04-14 California Institute Of Technology Inter-computer message routing system with each computer having separate routinng automata for each dimension of the network
EP0501524A2 (en) * 1983-05-31 1992-09-02 W. Daniel Hillis Parallel processor
US5157692A (en) * 1989-03-20 1992-10-20 Fujitsu Limited System for controlling communication between parallel computers
US5161156A (en) * 1990-02-02 1992-11-03 International Business Machines Corporation Multiprocessing packet switching connection system having provision for error correction and recovery
US5218676A (en) * 1990-01-08 1993-06-08 The University Of Rochester Dynamic routing system for a multinode communications network
US5412782A (en) * 1992-07-02 1995-05-02 3Com Corporation Programmed I/O ethernet adapter with early interrupts for accelerating data transfer
US5521916A (en) * 1994-12-02 1996-05-28 At&T Corp. Implementation of selective pushout for space priorities in a shared memory asynchronous transfer mode switch
US5581705A (en) * 1993-12-13 1996-12-03 Cray Research, Inc. Messaging facility with hardware tail pointer and software implemented head pointer message queue for distributed memory massively parallel processing system
EP0460599B1 (en) * 1990-06-06 1999-02-10 TM Patents L.P. Massively parallel processor including queue-based message delivery system

Patent Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0501524A2 (en) * 1983-05-31 1992-09-02 W. Daniel Hillis Parallel processor
WO1988001341A1 (en) * 1986-08-19 1988-02-25 Gerhard Haubenwallner Internal combustion engine
US4933933A (en) * 1986-12-19 1990-06-12 The California Institute Of Technology Torus routing chip
US4980852A (en) * 1986-12-30 1990-12-25 International Business Machines Corporation Non-locking queueing mechanism for enabling a receiver device to read from a queue without access synchronization with a sending device
WO1988008652A1 (en) * 1987-04-27 1988-11-03 Thinking Machines Corporation Method and apparatus for routing message packets
US5008882A (en) * 1987-08-17 1991-04-16 California Institute Of Technology Method and apparatus for eliminating unsuccessful tries in a search tree
US5105424A (en) * 1988-06-02 1992-04-14 California Institute Of Technology Inter-computer message routing system with each computer having separate routinng automata for each dimension of the network
US5068784A (en) * 1988-07-28 1991-11-26 Oki Electric Industry Co., Ltd. Method and system for monitoring the number of available buffers
US5027330A (en) * 1988-12-30 1991-06-25 At&T Bell Laboratories FIFO memory arrangement including a memory location fill indication
US4995056A (en) * 1989-01-13 1991-02-19 International Business Machines Corporation System and method for data communications
US4965793A (en) * 1989-02-03 1990-10-23 Digital Equipment Corporation Method and apparatus for interfacing a system control unit for a multi-processor
US5036459A (en) * 1989-03-09 1991-07-30 U.S. Philips Corporation Multi-processor computer system with distributed memory and an interprocessor communication mechanism, and method for operating such mechanism
US5157692A (en) * 1989-03-20 1992-10-20 Fujitsu Limited System for controlling communication between parallel computers
US5218676A (en) * 1990-01-08 1993-06-08 The University Of Rochester Dynamic routing system for a multinode communications network
US5161156A (en) * 1990-02-02 1992-11-03 International Business Machines Corporation Multiprocessing packet switching connection system having provision for error correction and recovery
EP0460599B1 (en) * 1990-06-06 1999-02-10 TM Patents L.P. Massively parallel processor including queue-based message delivery system
US5412782A (en) * 1992-07-02 1995-05-02 3Com Corporation Programmed I/O ethernet adapter with early interrupts for accelerating data transfer
US5581705A (en) * 1993-12-13 1996-12-03 Cray Research, Inc. Messaging facility with hardware tail pointer and software implemented head pointer message queue for distributed memory massively parallel processing system
US5521916A (en) * 1994-12-02 1996-05-28 At&T Corp. Implementation of selective pushout for space priorities in a shared memory asynchronous transfer mode switch

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
Barbara Chapman, et al., "Programming in Vienna Fortran", Dept. of Statistics and Coputer Science, 121-160.
Barbara Chapman, et al., Programming in Vienna Fortran , Dept. of Statistics and Coputer Science , 121 160. *
Bradley R. Carlile, "Algorithms and Design: The CRAP APP Shared-Memory System", COMPCON Spring '93, San Francisco, CA, 312-320, (Feb. 22, 1993).
Bradley R. Carlile, Algorithms and Design: The CRAP APP Shared Memory System , COMPCON Spring 93, San Francisco, CA , 312 320, (Feb. 22, 1993). *
Erik Debenedictis, et al., "Extending Unix for Scalable Computing", IEEE, 43-53, (Nov. 1993).
Erik Debenedictis, et al., Extending Unix for Scalable Computing , IEEE , 43 53, (Nov. 1993). *
Tom Mcdonald, et al., "Addressing in Cray Research's MPP Fortran", Third Workshop on Compilers for Parallel Computers, 161-172 (Jul. 7, 1992).
Tom Mcdonald, et al., Addressing in Cray Research s MPP Fortran , Third Workshop on Compilers for Parallel Computers , 161 172 (Jul. 7, 1992). *

Cited By (216)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8156312B2 (en) 1996-12-09 2012-04-10 Martin Vorbach Processor chip for reconfigurable data processing, for processing numeric and logic operations and including function and interconnection control units
US7822968B2 (en) 1996-12-09 2010-10-26 Martin Vorbach Circuit having a multidimensional structure of configurable cells that include multi-bit-wide inputs and outputs
US8195856B2 (en) 1996-12-20 2012-06-05 Martin Vorbach I/O and memory bus system for DFPS and units with two- or multi-dimensional programmable cell architectures
US7899962B2 (en) 1996-12-20 2011-03-01 Martin Vorbach I/O and memory bus system for DFPs and units with two- or multi-dimensional programmable cell architectures
US7650448B2 (en) 1996-12-20 2010-01-19 Pact Xpp Technologies Ag I/O and memory bus system for DFPS and units with two- or multi-dimensional programmable cell architectures
US7822881B2 (en) 1996-12-27 2010-10-26 Martin Vorbach Process for automatic dynamic reloading of data flow processors (DFPs) and units with two- or three-dimensional programmable cell architectures (FPGAs, DPGAs, and the like)
USRE44365E1 (en) 1997-02-08 2013-07-09 Martin Vorbach Method of self-synchronization of configurable elements of a programmable module
USRE45223E1 (en) 1997-02-08 2014-10-28 Pact Xpp Technologies Ag Method of self-synchronization of configurable elements of a programmable module
USRE45109E1 (en) 1997-02-08 2014-09-02 Pact Xpp Technologies Ag Method of self-synchronization of configurable elements of a programmable module
US6173375B1 (en) * 1997-02-28 2001-01-09 Lucent Technologies Inc. Method for accessing a shared resource in a multiprocessor system
US6055605A (en) * 1997-10-24 2000-04-25 Compaq Computer Corporation Technique for reducing latency of inter-reference ordering using commit signals in a multiprocessor system having shared caches
US6088771A (en) * 1997-10-24 2000-07-11 Digital Equipment Corporation Mechanism for reducing latency of memory barrier operations on a multiprocessor system
US7584390B2 (en) 1997-12-22 2009-09-01 Pact Xpp Technologies Ag Method and system for alternating between programs for execution by cells of an integrated circuit
US8819505B2 (en) 1997-12-22 2014-08-26 Pact Xpp Technologies Ag Data processor having disabled cores
US6629152B2 (en) * 1998-06-29 2003-09-30 International Business Machines Corporation Message passing using shared memory of a computer
US8468329B2 (en) 1999-02-25 2013-06-18 Martin Vorbach Pipeline configuration protocol and configuration unit communication
US8726250B2 (en) 1999-06-10 2014-05-13 Pact Xpp Technologies Ag Configurable logic integrated circuit having a multidimensional structure of configurable elements
US8230411B1 (en) * 1999-06-10 2012-07-24 Martin Vorbach Method for interleaving a program over a plurality of cells
US8312200B2 (en) 1999-06-10 2012-11-13 Martin Vorbach Processor chip including a plurality of cache elements connected to a plurality of processor cores
US6976260B1 (en) * 1999-09-24 2005-12-13 International Business Machines Corporation Method and apparatus for serializing a message queue in a multiprocessing environment
US7346910B1 (en) * 2000-05-26 2008-03-18 International Business Machines Incorporation Administration of groups of computer programs, data processing systems, or system resources
US8301872B2 (en) 2000-06-13 2012-10-30 Martin Vorbach Pipeline configuration protocol and configuration unit communication
US7912936B2 (en) 2000-09-13 2011-03-22 Nara Rajagopalan Managing interworking communications protocols
US9124555B2 (en) 2000-09-13 2015-09-01 Fortinet, Inc. Tunnel interface for securing traffic over a network
US9853948B2 (en) 2000-09-13 2017-12-26 Fortinet, Inc. Tunnel interface for securing traffic over a network
US9667604B2 (en) 2000-09-13 2017-05-30 Fortinet, Inc. Tunnel interface for securing traffic over a network
US7639632B2 (en) 2000-09-13 2009-12-29 Fortinet, Inc. System and method for managing and provisioning virtual routers
US7444398B1 (en) 2000-09-13 2008-10-28 Fortinet, Inc. System and method for delivering security services
US20020152373A1 (en) * 2000-09-13 2002-10-17 Chih-Tang Sun Tunnel interface for securing traffic over a network
US8069233B2 (en) 2000-09-13 2011-11-29 Fortinet, Inc. Switch management system and method
US9391964B2 (en) 2000-09-13 2016-07-12 Fortinet, Inc. Tunnel interface for securing traffic over a network
US7539744B2 (en) 2000-09-13 2009-05-26 Fortinet, Inc. Network operating system for maintaining redundant master control blade management information
US8260918B2 (en) 2000-09-13 2012-09-04 Fortinet, Inc. Packet routing system and method
US9258280B1 (en) 2000-09-13 2016-02-09 Fortinet, Inc. Tunnel interface for securing traffic over a network
US7111072B1 (en) * 2000-09-13 2006-09-19 Cosine Communications, Inc. Packet routing system and method
US7885207B2 (en) 2000-09-13 2011-02-08 Fortinet, Inc. Managing and provisioning virtual routers
US7174372B1 (en) 2000-09-13 2007-02-06 Fortinet, Inc. System and method for managing router metadata
US8250357B2 (en) 2000-09-13 2012-08-21 Fortinet, Inc. Tunnel interface for securing traffic over a network
US7272643B1 (en) 2000-09-13 2007-09-18 Fortinet, Inc. System and method for managing and provisioning virtual routers
US7574495B1 (en) 2000-09-13 2009-08-11 Fortinet, Inc. System and method for managing interworking communications protocols
US8583800B2 (en) 2000-09-13 2013-11-12 Fortinet, Inc. Packet routing system and method
US7389358B1 (en) 2000-09-13 2008-06-17 Fortinet, Inc. Distributed virtual system to support managed, network-based services
US8320279B2 (en) 2000-09-13 2012-11-27 Fortinet, Inc. Managing and provisioning virtual routers
US20070104119A1 (en) * 2000-09-13 2007-05-10 Fortinet, Inc. System and method for managing and provisioning virtual routers
US20100094980A1 (en) * 2000-09-13 2010-04-15 Fortinet, Inc. Managing and provisioning virtual routers
US9160716B2 (en) 2000-09-13 2015-10-13 Fortinet, Inc. Tunnel interface for securing traffic over a network
US20070121579A1 (en) * 2000-09-13 2007-05-31 Fortinet, Inc. Packet routing system and method
US20090300159A1 (en) * 2000-09-13 2009-12-03 Fortinet, Inc. Managing interworking communications protocols
US8058899B2 (en) 2000-10-06 2011-11-15 Martin Vorbach Logic cell array and bus system
US9047440B2 (en) 2000-10-06 2015-06-02 Pact Xpp Technologies Ag Logical cell array and bus system
US8471593B2 (en) 2000-10-06 2013-06-25 Martin Vorbach Logic cell array and bus system
US8161248B2 (en) 2001-02-24 2012-04-17 International Business Machines Corporation Simplifying and speeding the management of intra-node cache coherence
US7870343B2 (en) * 2001-02-24 2011-01-11 International Business Machines Corporation Managing coherence via put/get windows
US20070055825A1 (en) * 2001-02-24 2007-03-08 Blumrich Matthias A Managing coherence via put/get windows
US8122197B2 (en) 2001-02-24 2012-02-21 International Business Machines Corporation Managing coherence via put/get windows
US20090313439A1 (en) * 2001-02-24 2009-12-17 International Business Machines Corporation Managing coherence via put/get windows
US20110072219A1 (en) * 2001-02-24 2011-03-24 International Business Machines Corporation Managing coherence via put/get windows
US8312301B2 (en) 2001-03-05 2012-11-13 Martin Vorbach Methods and devices for treating and processing data
US9037807B2 (en) 2001-03-05 2015-05-19 Pact Xpp Technologies Ag Processor arrangement on a chip including data processing, memory, and interface elements
US8145881B2 (en) 2001-03-05 2012-03-27 Martin Vorbach Data processing device and method
US9075605B2 (en) 2001-03-05 2015-07-07 Pact Xpp Technologies Ag Methods and devices for treating and processing data
US8099618B2 (en) 2001-03-05 2012-01-17 Martin Vorbach Methods and devices for treating and processing data
US7844796B2 (en) 2001-03-05 2010-11-30 Martin Vorbach Data processing device and method
US7657877B2 (en) 2001-06-20 2010-02-02 Pact Xpp Technologies Ag Method for processing data
US7890663B2 (en) 2001-06-28 2011-02-15 Fortinet, Inc. Identifying nodes in a ring network
US9602303B2 (en) 2001-06-28 2017-03-21 Fortinet, Inc. Identifying nodes in a ring network
US9998337B2 (en) 2001-06-28 2018-06-12 Fortinet, Inc. Identifying nodes in a ring network
US7580373B2 (en) 2001-06-28 2009-08-25 Fortinet, Inc. Identifying nodes in a ring network
US7996827B2 (en) 2001-08-16 2011-08-09 Martin Vorbach Method for the translation of programs for reconfigurable architectures
US8869121B2 (en) 2001-08-16 2014-10-21 Pact Xpp Technologies Ag Method for the translation of programs for reconfigurable architectures
US7840842B2 (en) 2001-09-03 2010-11-23 Martin Vorbach Method for debugging reconfigurable architectures
US8069373B2 (en) 2001-09-03 2011-11-29 Martin Vorbach Method for debugging reconfigurable architectures
US8407525B2 (en) 2001-09-03 2013-03-26 Pact Xpp Technologies Ag Method for debugging reconfigurable architectures
US8429385B2 (en) 2001-09-03 2013-04-23 Martin Vorbach Device including a field having function cells and information providing cells controlled by the function cells
US8686549B2 (en) 2001-09-03 2014-04-01 Martin Vorbach Reconfigurable elements
US8209653B2 (en) 2001-09-03 2012-06-26 Martin Vorbach Router
US8686475B2 (en) 2001-09-19 2014-04-01 Pact Xpp Technologies Ag Reconfigurable elements
US20030093626A1 (en) * 2001-11-14 2003-05-15 Fister James D.M. Memory caching scheme in a distributed-memory network
US6704833B2 (en) * 2002-01-04 2004-03-09 Hewlett-Packard Development Company, L.P. Atomic transfer of a block of data
US8127061B2 (en) 2002-02-18 2012-02-28 Martin Vorbach Bus systems and reconfiguration methods
US7177311B1 (en) 2002-06-04 2007-02-13 Fortinet, Inc. System and method for routing traffic through a virtual router-based network switch
US7668087B2 (en) 2002-06-04 2010-02-23 Fortinet, Inc. Hierarchical metering in a virtual router-based network switch
US7522604B2 (en) 2002-06-04 2009-04-21 Fortinet, Inc. Routing traffic through a virtual router-based network switch
US8064462B2 (en) 2002-06-04 2011-11-22 Fortinet, Inc. Service processing switch
US20090225759A1 (en) * 2002-06-04 2009-09-10 Fortinet, Inc. Hierarchical metering in a virtual router-based network switch
US20030223418A1 (en) * 2002-06-04 2003-12-04 Sachin Desai Network packet steering
US20090238181A1 (en) * 2002-06-04 2009-09-24 Fortinet, Inc. Network packet steering via configurable association of processing resources and network interfaces
US20070109968A1 (en) * 2002-06-04 2007-05-17 Fortinet, Inc. Hierarchical metering in a virtual router-based network switch
US20030223361A1 (en) * 2002-06-04 2003-12-04 Zahid Hussain System and method for hierarchical metering in a virtual router based network switch
US8068503B2 (en) 2002-06-04 2011-11-29 Fortinet, Inc. Network packet steering via configurable association of processing resources and netmods or line interface ports
US8848718B2 (en) 2002-06-04 2014-09-30 Google Inc. Hierarchical metering in a virtual router-based network switch
US9967200B2 (en) 2002-06-04 2018-05-08 Fortinet, Inc. Service processing switch
US20080259936A1 (en) * 2002-06-04 2008-10-23 Fortinet, Inc. Service processing switch
US7203192B2 (en) 2002-06-04 2007-04-10 Fortinet, Inc. Network packet steering
US7161904B2 (en) 2002-06-04 2007-01-09 Fortinet, Inc. System and method for hierarchical metering in a virtual router based network switch
US20070127382A1 (en) * 2002-06-04 2007-06-07 Fortinet, Inc. Routing traffic through a virtual router-based network switch
US8638802B2 (en) 2002-06-04 2014-01-28 Cisco Technology, Inc. Network packet steering via configurable association of packet processing resources and network interfaces
US7376125B1 (en) 2002-06-04 2008-05-20 Fortinet, Inc. Service processing switch
US8306040B2 (en) 2002-06-04 2012-11-06 Fortinet, Inc. Network packet steering via configurable association of processing resources and network interfaces
US7720053B2 (en) 2002-06-04 2010-05-18 Fortinet, Inc. Service processing switch
US9215178B2 (en) 2002-06-04 2015-12-15 Cisco Technology, Inc. Network packet steering via configurable association of packet processing resources and network interfaces
US8111690B2 (en) 2002-06-04 2012-02-07 Google Inc. Routing traffic through a virtual router-based network switch
US7340535B1 (en) 2002-06-04 2008-03-04 Fortinet, Inc. System and method for controlling routing in a virtual router system
US7620678B1 (en) * 2002-06-12 2009-11-17 Nvidia Corporation Method and system for reducing the time-to-market concerns for embedded system design
US20040024771A1 (en) * 2002-08-01 2004-02-05 Oracle International Corporation Buffered message queue architecture for database management systems with transactional enqueue support
US20050125430A1 (en) * 2002-08-01 2005-06-09 Oracle International Corporation Asynchronous actions using flashback
US7814067B2 (en) 2002-08-01 2010-10-12 Oracle International Corporation Asynchronous actions using flashback
US8914590B2 (en) 2002-08-07 2014-12-16 Pact Xpp Technologies Ag Data processing method and device
US8281265B2 (en) 2002-08-07 2012-10-02 Martin Vorbach Method and device for processing data
US7657861B2 (en) 2002-08-07 2010-02-02 Pact Xpp Technologies Ag Method and device for processing data
US8156284B2 (en) 2002-08-07 2012-04-10 Martin Vorbach Data processing method and device
US7278055B2 (en) 2002-08-29 2007-10-02 Fortinet, Inc. System and method for virtual router failover in a network routing system
US8819486B2 (en) 2002-08-29 2014-08-26 Google Inc. Fault tolerant routing in a non-hot-standby configuration of a network routing system
US7096383B2 (en) 2002-08-29 2006-08-22 Cosine Communications, Inc. System and method for virtual router failover in a network routing system
US20070162783A1 (en) * 2002-08-29 2007-07-12 Fortinet, Inc. System and method for virtual router failover in a network routing system
US20100011245A1 (en) * 2002-08-29 2010-01-14 Fortinet, Inc. Fault tolerant routing in a non-hot-standby configuration of a network routing system
US8412982B2 (en) 2002-08-29 2013-04-02 Google Inc. Fault tolerant routing in a non-hot-standby configuration of a network routing system
US20040078621A1 (en) * 2002-08-29 2004-04-22 Cosine Communications, Inc. System and method for virtual router failover in a network routing system
US7761743B2 (en) 2002-08-29 2010-07-20 Fortinet, Inc. Fault tolerant routing in a non-hot-standby configuration of a network routing system
US8310274B2 (en) 2002-09-06 2012-11-13 Martin Vorbach Reconfigurable sequencer structure
US7782087B2 (en) 2002-09-06 2010-08-24 Martin Vorbach Reconfigurable sequencer structure
US8803552B2 (en) 2002-09-06 2014-08-12 Pact Xpp Technologies Ag Reconfigurable sequencer structure
US7928763B2 (en) 2002-09-06 2011-04-19 Martin Vorbach Multi-core processing system
US9014186B2 (en) 2002-11-18 2015-04-21 Fortinet, Inc. Hardware-accelerated packet multicasting
US10200275B2 (en) 2002-11-18 2019-02-05 Fortinet, Inc. Hardware-accelerated packet multicasting
US7266120B2 (en) 2002-11-18 2007-09-04 Fortinet, Inc. System and method for hardware accelerated packet multicast in a virtual routing system
US9407449B2 (en) 2002-11-18 2016-08-02 Fortinet, Inc. Hardware-accelerated packet multicasting
US8644311B2 (en) 2002-11-18 2014-02-04 Fortinet, Inc. Hardware-accelerated packet multicasting in a virtual routing system
US20040095934A1 (en) * 2002-11-18 2004-05-20 Cosine Communications, Inc. System and method for hardware accelerated packet multicast in a virtual routing system
US20140330919A1 (en) * 2003-03-05 2014-11-06 Websense, Inc. Asynchronous mechanism and message pool
US20060098022A1 (en) * 2003-06-30 2006-05-11 International Business Machines Corporation System and method for transfer of data between processors using a locked set, head and tail pointers
US20060095672A1 (en) * 2003-06-30 2006-05-04 Microsoft Corporation System and method for parallel execution of data generation tasks
US7355601B2 (en) 2003-06-30 2008-04-08 International Business Machines Corporation System and method for transfer of data between processors using a locked set, head and tail pointers
US9853917B2 (en) 2003-08-27 2017-12-26 Fortinet, Inc. Heterogeneous media packet bridging
US9509638B2 (en) 2003-08-27 2016-11-29 Fortinet, Inc. Heterogeneous media packet bridging
US9331961B2 (en) 2003-08-27 2016-05-03 Fortinet, Inc. Heterogeneous media packet bridging
US8503463B2 (en) 2003-08-27 2013-08-06 Fortinet, Inc. Heterogeneous media packet bridging
US8812820B2 (en) 2003-08-28 2014-08-19 Pact Xpp Technologies Ag Data processing device and method
US20050093357A1 (en) * 2003-10-29 2005-05-05 Mitsumasa Takeda Structure of backrest assembly for standing ride type driver's platform in industrial vehicle
US20050198422A1 (en) * 2003-12-18 2005-09-08 Arm Limited Data communication mechanism
US20060026358A1 (en) * 2004-07-29 2006-02-02 International Business Machines Corporation Method and apparatus of supporting cacheable registers
US7840757B2 (en) 2004-07-29 2010-11-23 International Business Machines Corporation Method and apparatus for providing high speed memory for a processing unit
US9167016B2 (en) 2004-09-24 2015-10-20 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US9166805B1 (en) 2004-09-24 2015-10-20 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US7881244B2 (en) 2004-09-24 2011-02-01 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US8369258B2 (en) 2004-09-24 2013-02-05 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US10038567B2 (en) 2004-09-24 2018-07-31 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US20100142527A1 (en) * 2004-09-24 2010-06-10 Fortinet, Inc. Scalable IP-Services Enabled Multicast Forwarding with Efficient Resource Utilization
US9319303B2 (en) 2004-09-24 2016-04-19 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US8213347B2 (en) 2004-09-24 2012-07-03 Fortinet, Inc. Scalable IP-services enabled multicast forwarding with efficient resource utilization
US20110122872A1 (en) * 2004-09-24 2011-05-26 Fortinet, Inc. Scalable ip-services enabled multicast forwarding with efficient resource utilization
US20090225754A1 (en) * 2004-09-24 2009-09-10 Fortinet, Inc. Scalable ip-services enabled multicast forwarding with efficient resource utilization
US7305524B2 (en) 2004-10-08 2007-12-04 International Business Machines Corporation Snoop filter directory mechanism in coherency shared memory system
US7577794B2 (en) 2004-10-08 2009-08-18 International Business Machines Corporation Low latency coherency protocol for a multi-chip multiprocessor system
US20060080512A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Graphics processor with snoop filter
US20060080398A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Direct access of cache lock set data without backing memory
US20060080511A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Enhanced bus transactions for efficient support of a remote cache directory copy
US7475190B2 (en) 2004-10-08 2009-01-06 International Business Machines Corporation Direct access of cache lock set data without backing memory
US20070294481A1 (en) * 2004-10-08 2007-12-20 Hoover Russell D Snoop filter directory mechanism in coherency shared memory system
US20060080508A1 (en) * 2004-10-08 2006-04-13 International Business Machines Corporation Snoop filter directory mechanism in coherency shared memory system
US8332592B2 (en) 2004-10-08 2012-12-11 International Business Machines Corporation Graphics processor with snoop filter
US20080320553A1 (en) * 2004-11-18 2008-12-25 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US20080317231A1 (en) * 2004-11-18 2008-12-25 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US7843813B2 (en) 2004-11-18 2010-11-30 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US7869361B2 (en) 2004-11-18 2011-01-11 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US7808904B2 (en) 2004-11-18 2010-10-05 Fortinet, Inc. Method and apparatus for managing subscriber profiles
US20080117917A1 (en) * 2004-11-18 2008-05-22 Fortinet, Inc. Method and apparatus for managing subscriber profiles
US7876683B2 (en) 2004-11-18 2011-01-25 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US20070115979A1 (en) * 2004-11-18 2007-05-24 Fortinet, Inc. Method and apparatus for managing subscriber profiles
US20090007228A1 (en) * 2004-11-18 2009-01-01 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US20080317040A1 (en) * 2004-11-18 2008-12-25 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US7961615B2 (en) 2004-11-18 2011-06-14 Fortinet, Inc. Managing hierarchically organized subscriber profiles
US20060140122A1 (en) * 2004-12-28 2006-06-29 International Business Machines Corporation Link retry per virtual channel
US20090077433A1 (en) * 2004-12-28 2009-03-19 Shearer Robert A Self-healing link sequence counts within a circular buffer
US7499452B2 (en) 2004-12-28 2009-03-03 International Business Machines Corporation Self-healing link sequence counts within a circular buffer
US7978705B2 (en) 2004-12-28 2011-07-12 International Business Machines Corporation Self-healing link sequence counts within a circular buffer
US20060140188A1 (en) * 2004-12-28 2006-06-29 International Business Machines Corporation Self-healing link sequence counts within a circular buffer
US7779418B2 (en) * 2004-12-30 2010-08-17 Oracle International Corporation Publisher flow control and bounded guaranteed delivery for message queues
US8397244B2 (en) 2004-12-30 2013-03-12 Oracle International Corporation Publisher flow control and bounded guaranteed delivery for message queues
US20100281491A1 (en) * 2004-12-30 2010-11-04 Kapil Surlaker Publisher flow control and bounded guaranteed delivery for message queues
US20060149787A1 (en) * 2004-12-30 2006-07-06 Kapil Surlaker Publisher flow control and bounded guaranteed delivery for message queues
US20060190659A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corportion Computer system bus bridge
US7757032B2 (en) 2005-02-24 2010-07-13 International Business Machines Corporation Computer system bus bridge
US20060190667A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Pipeline bit handling circuit and method for a bus bridge
US7469312B2 (en) 2005-02-24 2008-12-23 International Business Machines Corporation Computer system bus bridge
US20080307147A1 (en) * 2005-02-24 2008-12-11 International Business Machines Corporation Computer system bus bridge
US20060190655A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Apparatus and method for transaction tag mapping between bus domains
US7330925B2 (en) 2005-02-24 2008-02-12 International Business Machines Corporation Transaction flow control mechanism for a bus bridge
US7194567B2 (en) 2005-02-24 2007-03-20 International Business Machines Corporation Method and system for ordering requests at a bus interface
US20060190660A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Data ordering translation between linear and interleaved domains at a bus interface
US7275125B2 (en) 2005-02-24 2007-09-25 International Business Machines Corporation Pipeline bit handling circuit and method for a bus bridge
US7275124B2 (en) 2005-02-24 2007-09-25 International Business Machines Corporation Method and system for controlling forwarding or terminating of a request at a bus interface based on buffer availability
US20060190668A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Computer system architecture
US20060190662A1 (en) * 2005-02-24 2006-08-24 International Business Machines Corporation Transaction flow control mechanism for a bus bridge
US20060190651A1 (en) * 2005-02-24 2006-08-24 International Business Machiness Corporation Method and system for ordering requests at a bus interface
US7206886B2 (en) 2005-02-24 2007-04-17 International Business Machines Corporation Data ordering translation between linear and interleaved domains at a bus interface
US8196150B2 (en) 2005-10-07 2012-06-05 Oracle International Corporation Event locality using queue services
US20070101341A1 (en) * 2005-10-07 2007-05-03 Oracle International Corporation Event locality using queue services
US20070121659A1 (en) * 2005-11-30 2007-05-31 Broadcom Corporation Ring-based cache coherent bus
US7500031B2 (en) * 2005-11-30 2009-03-03 Broadcom Corporation Ring-based cache coherent bus
US8250503B2 (en) 2006-01-18 2012-08-21 Martin Vorbach Hardware definition method including determining whether to implement a function as hardware or software
US7693257B2 (en) 2006-06-29 2010-04-06 Accuray Incorporated Treatment delivery optimization
US20080002811A1 (en) * 2006-06-29 2008-01-03 Allison John W Treatment delivery optimization
US7716397B2 (en) * 2007-07-03 2010-05-11 Lsi Corporation Methods and systems for interprocessor message exchange between devices using only write bus transactions
US20090013113A1 (en) * 2007-07-03 2009-01-08 Henry Russell J Methods and systems for interprocessor message exchange between devices using only write bus transactions
US20100153965A1 (en) * 2008-12-16 2010-06-17 International Buisness Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on inter-thread communications
US20100153541A1 (en) * 2008-12-16 2010-06-17 International Buisness Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on processor workload
US20100153542A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on broadcast information
US20100153966A1 (en) * 2008-12-16 2010-06-17 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster using local job tables
US9384042B2 (en) 2008-12-16 2016-07-05 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on inter-thread communications
US8122132B2 (en) 2008-12-16 2012-02-21 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on broadcast information
US9396021B2 (en) 2008-12-16 2016-07-19 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster using local job tables
US8239524B2 (en) * 2008-12-16 2012-08-07 International Business Machines Corporation Techniques for dynamically assigning jobs to processors in a cluster based on processor workload
WO2017206101A1 (en) * 2016-06-01 2017-12-07 Beijing Yi Zhang Yun Feng Technology Co., Ltd. Synchronization of messages based on queues
CN109564560A (en) * 2016-06-01 2019-04-02 北京易掌云峰科技有限公司 Message synchronization based on queue
US10747598B2 (en) 2016-06-01 2020-08-18 Beijing Yi Zhang Yun Feng Technology Co., Ltd. Synchronization of messages based on queues

Similar Documents

Publication Publication Date Title
US5841973A (en) Messaging in distributed memory multiprocessing system having shell circuitry for atomic control of message storage queue's tail pointer structure in local memory
US5864738A (en) Massively parallel processing system using two data paths: one connecting router circuit to the interconnect network and the other connecting router circuit to I/O controller
US5581705A (en) Messaging facility with hardware tail pointer and software implemented head pointer message queue for distributed memory massively parallel processing system
US5592625A (en) Apparatus for providing shared virtual memory among interconnected computer nodes with minimal processor involvement
US7493417B2 (en) Method and data processing system for microprocessor communication using a processor interconnect in a multi-processor system
US7734877B2 (en) Method and data processing system for processor-to-processor communication in a clustered multi-processor system
US5835925A (en) Using external registers to extend memory reference capabilities of a microprocessor
US7698373B2 (en) Method, processing unit and data processing system for microprocessor communication in a multi-processor system
JP4660064B2 (en) Performing locks in a distributed processing system
US6009472A (en) Process for transferring data between nodes in a multinodal information system
US7818364B2 (en) Method and data processing system for microprocessor communication in a cluster-based multi-processor system
JP2004118855A (en) Maintenance method for multiprocessor computer system
US5954794A (en) Computer system data I/O by reference among I/O devices and multiple memory units
JP2002373115A (en) Replacement control method for shared cache memory and device therefor
US11157405B2 (en) Programmable cache coherent node controller
US5737638A (en) System for determining plurality of data transformations to be performed upon single set of data during single transfer by examining communication data structure
Ang et al. StarT-Voyager: A flexible platform for exploring scalable SMP issues
US6904465B2 (en) Low latency inter-reference ordering in a multiple processor system employing a multiple-level inter-node switch
US6363458B1 (en) Adaptive granularity method for integration of fine and coarse communication in the distributed shared memory system
US6708258B1 (en) Computer system for eliminating memory read-modify-write operations during packet transfers
Circuitry et al. System Components
JPH0950423A (en) Data transmission method between remote information-processing systems
Lindenstruth et al. 10. SCI Physical Layer API
Exploring Scalable CSAIL
JPH01194055A (en) Parallel computer

Legal Events

Date Code Title Description
AS Assignment

Owner name: CRAY RESEARCH, INC., MINNESOTA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KESSLER, RICHARD E.;OBERLIN, STEVEN M.;SCOTT, STEVEN L.;REEL/FRAME:008016/0793;SIGNING DATES FROM 19960510 TO 19960515

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: SILICON GRAPHICS, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CRAY RESEARCH, L.L.C.;REEL/FRAME:010927/0853

Effective date: 20000524

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: FOOTHILL CAPITAL CORPORATION, CALIFORNIA

Free format text: SECURITY AGREEMENT;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:012428/0236

Effective date: 20011109

FPAY Fee payment

Year of fee payment: 4

REMI Maintenance fee reminder mailed
AS Assignment

Owner name: U.S. BANK NATIONAL ASSOCIATION, AS TRUSTEE, CALIFO

Free format text: SECURITY INTEREST;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:014805/0855

Effective date: 20031223

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: GENERAL ELECTRIC CAPITAL CORPORATION,CALIFORNIA

Free format text: SECURITY INTEREST;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:018545/0777

Effective date: 20061017

Owner name: GENERAL ELECTRIC CAPITAL CORPORATION, CALIFORNIA

Free format text: SECURITY INTEREST;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:018545/0777

Effective date: 20061017

AS Assignment

Owner name: MORGAN STANLEY & CO., INCORPORATED, NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GENERAL ELECTRIC CAPITAL CORPORATION;REEL/FRAME:019995/0895

Effective date: 20070926

Owner name: MORGAN STANLEY & CO., INCORPORATED,NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GENERAL ELECTRIC CAPITAL CORPORATION;REEL/FRAME:019995/0895

Effective date: 20070926

FPAY Fee payment

Year of fee payment: 12

AS Assignment

Owner name: SILICON GRAPHICS INTERNATIONAL, CORP., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SILICON GRAPHICS, INC. ET AL.;SGI INTERNATIONAL, INC.;SIGNING DATES FROM 20090508 TO 20120208;REEL/FRAME:027727/0212

AS Assignment

Owner name: SGI INTERNATIONAL, INC., CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:SILICON GRAPHICS INTERNATIONAL, INC.;REEL/FRAME:032714/0230

Effective date: 20090513

Owner name: SILICON GRAPHICS INTERNATIONAL, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SILICON GRAPHICS, INC.;REEL/FRAME:032714/0032

Effective date: 20090508

Owner name: SILICON GRAPHICS INTERNATIONAL CORP., CALIFORNIA

Free format text: MERGER;ASSIGNOR:SGI INTERNATIONAL, INC.;REEL/FRAME:032714/0427

Effective date: 20120808

AS Assignment

Owner name: CRAY RESEARCH, L.L.C., CALIFORNIA

Free format text: MERGER;ASSIGNOR:CRAY RESEARCH, INC.;REEL/FRAME:034837/0218

Effective date: 19981229

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:SILICON GRAPHICS INTERNATIONAL CORP.;REEL/FRAME:035200/0722

Effective date: 20150127

AS Assignment

Owner name: SILICON GRAPHICS, INC., CALIFORNIA

Free format text: ORDER...AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, ENCUMBRANCES, AND INTERESTS;ASSIGNOR:FOOTHILL CAPITAL CORPORATION;REEL/FRAME:039474/0606

Effective date: 20090430

Owner name: SILICON GRAPHICS, INC., CALIFORNIA

Free format text: ORDER...AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, ENCUMBRANCES, AND INTERESTS;ASSIGNOR:MORGAN STANLEY & CO., INCORPORATED;REEL/FRAME:039482/0015

Effective date: 20090508

AS Assignment

Owner name: SILICON GRAPHICS, INC, CALIFORNIA

Free format text: ORDER. . .AUTHORIZING THE SALE OF ALL OR SUBSTANTIALLY ALL OF THE ASSETS OF THE DEBTORS FREE AND CLEAR OF ALL LIENS, CLAIMS, ENCUMBRANCES, AND INTERESTS;ASSIGNOR:U.S. BANK NATIONAL ASSOCIATION, AS TRUSTEE;REEL/FRAME:039484/0869

Effective date: 20090430

AS Assignment

Owner name: SILICON GRAPHICS INTERNATIONAL CORP., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT;REEL/FRAME:040545/0362

Effective date: 20161101

AS Assignment

Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SILICON GRAPHICS INTERNATIONAL CORP.;REEL/FRAME:044128/0149

Effective date: 20170501