WO2001016760A1

WO2001016760A1 - Switchable shared-memory cluster

Info

Publication number: WO2001016760A1
Application number: PCT/US2000/024039
Authority: WO
Inventors: Lynn Parker West; Karlon K. West
Original assignee: Times N Systems, Inc.
Priority date: 1999-08-31
Filing date: 2000-08-31
Publication date: 2001-03-08
Also published as: WO2001016738A8; WO2001016740A2; WO2001016738A9; WO2001016750A2; WO2001016743A3; WO2001016737A3; AU6949600A; CA2382927A1; EP1214652A2; WO2001016743A8; WO2001016742A2; CA2382929A1; AU7110000A; WO2001016743A2; WO2001016761A2; WO2001016750A3; AU7108300A; AU6949700A; AU7108500A; WO2001016742A3

Abstract

Methods, systems and devices are described for a switchable shared-memory cluster. An apparatus, includes a shared memory unit; a first processing node coupled to said shared memory unit; and a second processing node coupled to said shared memory unit. A memory page can be switched from a shared state to a private state or from said private state to a shared state. The methods, systems and devices provide advantages because the speed and scalability of parallel processor systems is enhanced.

Description

SWITCHABLE SHARED-MEMORY CLUSTER

BACKGROUND OF THE INVENTION 1. Field of the Invention

The invention relates generally to the field of computer systems which have multiple processing nodes and in which each processing node is provided with private, local memory and also in which each processing node has access to a range of memory which is shared with other processing nodes. More particularly, the invention relates to computer science techniques that utilize a switchable shared-memory cluster.

2. Discussion of the Related Art

The clustering of workstations is a well-known art. In the most common cases, the clustering involves workstations that operate almost totally independently, utilizing the network only to share such services as a printer, license-limited applications, or shared files.

In more-closely-coupled environments, some software packages (such as NQS) allow a cluster of workstations to share work. In such cases the work arrives, typically as batch jobs, at an entry point to the cluster where it is queued and dispatched to the workstations on the basis of load.

In both of these cases, and all other known cases of clustering, the operating system and cluster subsystem are built around the concept of message-passing. The term message-passing means that a given workstation operates on some portion of a job until communications (to send or receive data, typically) with another workstation is necessary. Then, the first workstation prepares and communicates with the other workstation.

Another well-known art is that of clustering processors within a machine, usually called a Massively Parallel Processor or MPP, in which the techniques are essentially identical to those of clustered workstations. Usually, the bandwidth and latency of the interconnect network of an MPP are more highly optimized, but the system operation is the same. In the general case, the passing of a message is an extremely expensive operation; expensive in the sense that many CPU cycles in the sender and receiver are consumed by the process of sending, receiving, bracketing, verifying, and routing the message, CPU cycles that are therefore not available for other operations. A highly streamlined message-passing subsystem can typically require 10,000 to 20,000 CPU cycles or more.

There are specific cases wherein the passing of a message requires significantly less overhead. However, none of these specific cases is adaptable to a general-purpose computer system. Message-passing parallel processor systems have been offered commercially for years but have failed to capture significant market share because of poor performance and difficulty of programming for typical parallel applications. Message-passing parallel processor systems do have some advantages. In particular, because they share no resources, message-passing parallel processor systems are easier to provide with high-availability features.

What is needed is a better approach to parallel processor systems.

There are alternatives to the passing of messages for closely-coupled cluster work. One such alternative is the use of shared memory for inter- processor communication. Shared-memory systems, have been much more successful at capturing market share than message-passing systems because of the dramatically superior performance of shared-memory systems, up to about four-processor systems. In Search of Clusters, Gregory F. Pfister 2nd ed. (January 1998) Prentice Hall Computer Books, ISBN: 0138997098 describes a computing system with multiple processing nodes in which each processing node is provided with private, local memory and also has access to a range of memory which is shared with other processing nodes. The disclosure of this publication in its entirety is hereby expressly incorporated herein by reference for the purpose of indicating the background of the invention and illustrating the state of the art.

However, providing high availability for traditional shared-memory systems has proved to be an elusive goal. The nature of these systems, which share all code and all data, including that data which controls the shared operating systems, is incompatible with the separation normally required for high availability. What is needed is an approach to shared-memory systems that improves availability. Although the use of shared memory for inter-processor communication is a well-known art, prior to the teachings of U.S. Ser. No. 09/273,430, filed March 19, 1999, entitled Shared Memory Apparatus and Method for Multiprocessing Systems, the processors shared a single copy of the operating system. The problem with such systems is that they cannot be efficiently scaled beyond four to eight way systems except in unusual circumstances. All known cases of said unusual circumstances are such that the systems are not good price-performance systems for general-purpose computing.

The entire contents of U.S. Patent Applications 09/273,430, filed March 19, 1999 and PCT/US00/01262, filed January 18, 2000 are hereby expressly incorporated by reference herein for all purposes. U.S. Ser. No. 09/273,430, improved upon the concept of shared memory by teaching the concept which will herein be referred to as a tight cluster. The concept of a tight cluster is that of individual computers, each with its own CPU(s), memory, I/O, and operating system, but for which collection of computers there is a portion of memory which is shared by all the computers and via which they can exchange information. U.S. Ser. No. 09/273,430 describes a system in which each processing node is provided with its own private copy of an operating system and in which the connection to shared memory is via a standard bus. The advantage of a tight cluster in comparison to an SMP is "scalability" which means that a much larger number of computers can be attached together via a tight cluster than an SMP with little loss of processing efficiency.

What is needed are improvements to the concept of the tight cluster. What is also needed is an expansion of the concept of the tight cluster. Another well-known art is the use of memory caches to improve performance. Caches provide such a significant performance boost that most modern computers use them. At the very top of the performance (and price) range all of memory is constructed using cache-memory technologies. However, this is such an expensive approach that few manufacturers use it. All manufacturers of personal computers (PCs) and workstations use caches except for the very low end of the PC business where caches are omitted for price reasons and performance is, therefore, poor. Caches, however, present a problem for shared-memory computing systems; the problem of coherence. As a particular processor reads or writes a word of shared memory, that word and usually a number of surrounding words are transferred to that particular processor's cache memory transparently by cache-memory hardware. That word and the surrounding words (if any) are transferred into a portion of the particular processor's cache memory that is called a cache line or cache block.

If the transferred cache line is modified by the particular processor, the representation in the cache memory will become different from the value in shared memory. That cache line within that particular processor's cache memory is, at that point, called a "dirty" line. The particular processor with the dirty line, when accessing that memory address will see the new (modified) value. Other processors, accessing that memory address will see the old (unmodified) value in shared memory. This lack of coherence between such accesses will lead to incorrect results. Modern computers, workstations, and PCs which provide for multiple processors and shared memory, therefore, also provide high-speed, transparent cache coherence hardware to assure that if a line in one cache changes and another processor subsequently accesses a value which is in that address range, the new values will be transferred back to memory or at least to the requesting processor.

Caches can be maintained coherent by software provided that sufficient cache-management instructions are provided by the manufacturer. However, in many cases, an adequate arsenal of such instructions are not provided. Moreover, even in cases where the instruction set is adequate, the software overhead is so great that no examples of are known of commercially successful machines which use software-managed coherence. SUMMARY OF THE INVENTION A goal of the invention is to simultaneously satisfy the above-discussed requirements of improving and expanding the tight cluster concept which, in the case of the prior art, are not satisfied. One embodiment of the invention is based on an apparatus, comprising: a shared memory unit; a first processing node coupled to said shared memory unit; and a second processing node coupled to said shared memory unit, wherein a memory page can be switched from a shared state to a private state or from said private state to a shared state. Another embodiment of the invention is based on an apparatus, comprising: a shared memory unit; a first processing node coupled to said shared memory unit; and a second processing node coupled to said shared memory unit, wherein a memory page can be switched from a shared state to a private state or from said private state to a shared state. Another embodiment of the invention is based on an electronic media, comprising: a computer program adapted to switch a page of memory from a shared state to a private state or from a private state to a shared state. Another embodiment of the invention is based on a computer program comprising computer program means adapted to perform the steps of switching a page of memory from a shared state to a private state or from a private state to a shared state when said computer program is run on a computer. Another embodiment of the invention is based on a system, comprising a multiplicity of workstations and a shared memory unit (SMU) interconnected and arranged such that memory accesses by a given workstation in a first set of address ranges will be to its local, private memory and that memory accesses to a second set of address ranges will be to shared memory, and arranged such that said accesses to shared memory are recorded and signaled by a Cache Emulation Adapter (CEA) capable of recognizing and responding to cache-coherence signals within the workstation. Another embodiment of the invention is based on a parallel computing system which has multiple PRNs each with private memory and which also is provided with some shared memory, means to switch some portion memory on a dynamic basis from shared memory to private memory. These, and other goals and embodiments of the invention will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following description, while indicating preferred embodiments of the invention and numerous specific details thereof, is given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the invention without departing from the spirit thereof, and the invention includes all such modifications.

BRIEF DESCRIPTION OF THE DRAWINGS

A clear conception of the advantages and features constituting the invention, and of the components and operation of model systems provided with the invention, will become more readily apparent by referring to the exemplary, and therefore nonlimiting, embodiments illustrated in the drawings accompanying and forming a part of this specification, wherein like reference characters (if they occur in more than one view) designate the same parts. It should be noted that the features illustrated in the drawings are not necessarily drawn to scale.

FIG. 1 illustrates a block schematic view of a systems structure showing private memories plus shared memory, representing an embodiment of the invention.

FIG. 2 illustrates a block schematic view of the details of a private memory in one processing node, showing regions which can be dynamically allocated to hold shared pages, representing an embodiment of the invention. FIG. 3 illustrates a block schematic view of a shared page being moved to one of the private memories, representing an embodiment of the invention.

FIG. 4 illustrates a block schematic view of a request from an alternative processing node to release a lock on a switched-shared page, representing an embodiment of the invention. FIG. 5 illustrates a block schematic view of a shared page being moved back to a shared-memory node, representing an embodiment of the invention. DESCRIPTION OF PREFERRED EMBODIMENTS The invention and the various features and advantageous details thereof are explained more fully with reference to the nonlimiting embodiments that are illustrated in the accompanying drawings and detailed in the following description of preferred embodiments. Descriptions of well known components and processing techniques are omitted so as not to unnecessarily obscure the invention in detail.

The teachings of U.S. Ser. No. 09/273,430 include a system which is a single entity; one large supercomputer. The invention is also applicable to a cluster of workstations, or even a network.

The invention is applicable to systems of the type of Pfister or the type of U.S. Ser. No. 09/273,430 in which each processing node has its own copy of an operating system. The invention is also applicable to other types of multiple processing node systems. The context of the invention can include a tight cluster as described in

U.S. Ser. No. 09/273,430. A tight cluster is defined as a cluster of workstations or an arrangement within a single, multiple-processor machine in which the processors are connected by a high-speed, low-latency interconnection, and in which some but not all memory is shared among the processors. Within the scope of a given processor, accesses to a first set of ranges of memory addresses will be to local, private memory but accesses to a second set of memory address ranges will be to shared memory. The significant advantage to a tight cluster in comparison to a message-passing cluster is that, assuming the environment has been appropriately established, the exchange of information involves a single STORE instruction by the sending processor and a subsequent single LOAD instruction by the receiving processor.

The establishment of the environment, taught by U.S. Ser. No. 09/273,430 and more fully by companion disclosures (U.S. Provisional Application Ser. No. 60/220,794, filed July 26, 2000; U.S. Provisional Application Ser. No. 60/220,748, filed July 26, 2000; WSGR 15245-711;

WSGR 15245-712; WSGR 15245-713; WSGR 15245-715; WSGR 15245-716; WSGR 15245-717; WSGR 15245-718; and WSGR 15245-719. the entire contents of all which are hereby expressly incorporated herein by reference for all purposes) can be performed in such a way as to require relatively little system overhead, and to be done once for many, many information exchanges. Therefore, a comparison of 10,000 instructions for message-passing to a pair of instructions for tight-clustering, is valid.

The invention can be embodied in the environment described in U.S. Ser. No. 09/273,430 with the additional feature that multiple computers can selectively switch, on a range-by-range basis, certain portions of memory from shared to private status. Thus, the invention can be embodied in a tight cluster system in which several (or many, e.g., 100, 1000, or 10,000 etc.) ranges in the second set of address ranges is to memory that can be dynamically switched between shared and private. When switched to private, a given range will be private with regard to a particular processor and will be inaccessible to other processors. In a system with some local, private memory and shared, globally visible memory, the invention can include dynamically allocating portions of shared memory to individual processors for temporary private use, and for subsequently returning those portions to the common, shared pool at a later time. The invention contemplates such an environment in which each standing computer is provided with very high-speed, low-latency communication means to shared-memory, and which shared-memory interface provides means for switching each of several banks of memory from shared to private status. Each computer in the system can also be provided with a specific inter-connection to the shared memory. In a system with some memory local and private to each of a multiplicity of processing nodes (PRNs) in which each PRN is provided with its own copy of the operating system, and in which some memory is globally visible to all PRNs and shared by all, certain advantages accrue relative to alternative computing systems. The most significant of these advantages is that only data which is shared is placed into shared memory, thus eliminating all operating-system contention and most memory-access contention among the PRN's. There can be, of course, contention between the PRNs for access to shared data which is in shared memory. The invention can further reduce contention in such a system for applications which exhibit certain data-access characteristics. Specifically, the systems of the type of Pfister or U.S. Ser. No. 09/273,430 work best if data which is shared is shared on a fine-grained basis.

That is, if the accesses to that shared data by any particular PRN are few in number prior to accesses to that shared data by other PRNs. For large-grained data sharing (one PRN uses some particular set of data many times before any other PRN uses that data at all) these share-as-needed systems still in general outperform other kinds of parallel systems, but a further improvement is possible. This invention describes such an improvement, called switched- memory systems.

In a switched-memory system, some of memory can be switched to be shared under certain circumstances but private to one particular PRN under other circumstances. In a preferred embodiment, the memory is not physically switched but rather its characteristics are switched. Operating-system services and API's can be used to control access to shared memory.

Referring to FIGS. 1 and 2, a close cluster system is shown. A shared memory 100 is coupled to a first processor 0 via a first private memory 10. The shared memory 100 is coupled to a second processor 1 via a second private memory 1 1. The shared memory 100 is coupled to a third processor 2 via a third private memory 12. The dashed line in FIG. 1 indicates that there may be more processors than are shown. Referring to FIG. 2, the first private memory 10 includes two switchable regions 51 and 61. These regions can be termed pages.

If an application needs access to a block of memory (the word "page" will be used hereafter, and this term means a page in the well-understood virtual-memory operating sense) and if the memory space needed is in shared memory, the application will use an API interface which will here be called tnsl_shared_memory_get(). The API interface tnsl_shared_memory_get() returns to the application a pointer to the memory (if available) and information relative to one or more pages of shared memory. In contrast, if an application needs access to a block of memory and the memory space needed is in private, local memory, the application will use a different API interface which will be here called local_memory_get(). In such a case, the operating system returns a pointer to the memory (if available) and returns other information indicating whether some other process or thread is already utilizing that memory.

Different kinds of locks can be obtained on shared memory, including Shared locks and Exclusive locks. If a PRN has an exclusive lock on a page and some other PRN needs that page, the second PRN will request that the first PRN release its lock. However, if a first PRN has a shared lock on some page and a second PRN requires shared access to that page, the first PRN does not need to release its shared lock on that page. For a third case, however, if a first PRN has a shared lock on some page and a second PRN requires exclusive access to that page, then the second PRN will request that the first PRN release its shared lock on the page.

Referring to FIGS. 3-5, dynamic switching and lock release is shown. FIG. 3 shows a page 300 being switched from a shared state to a private state. The shared memory 100 includes a region 110 that contains a locked pointer to the shared page. FIG. 4 shows a Processor N passing a release request to the private memory 10 through the region 110. FIG. 5 shows the page 300 being switched from a private state to a shared state.

In a preferred embodiment, tnsl_shared_memory_get() is augmented with either a second, similar function which has added call or return parameters, or with augmenting the original function with added call and return parameters. The new call parameters include but are not limited to: (1) exclusive private access requested; (2) shared private access requested; (3) exclusive private access required; and (4) shared private access required. The return parameters include an indicator as to whether the request was successfully fulfilled, whether private access is granted, and in the case of shared private access whether some other PRN is also sharing the page.

In a preferred embodiment, if a PRN acquires private access, the most significant result of the acquisition is that the PRN may then cache the data in its local, level- 1 cache. Since this cache can operate at much higher speed than other caches and main memory, such access can result in very significant performance speedups. This performance improvement will only be observed if the application does in fact access the data a number of times before releasing the page, but if the application accesses the data many times before releasing it, the speedup can be dramatic.

In the case of exclusive private access, the corresponding release function call by the application to the operating system extensions will cause the operating system extension to remove or invalidate the data in the private cache system of that particular PRN and write any modified data back to shared memory. Therefore, for exclusive private access, the system and hardware will assure correctness of operation.

In the case of shared private access, the application can assure correctness of operation if it never issues a Store to any data address in such a page; or, if it does Store into the page the application can assure correctness of operation if any data it does modify during said shared access period cannot affect other applications which may access the page while or after it has been in shared-private state. In general, shared private access is for Read-Only access. Another embodiment of the invention is to reconfigure the memory access switch so that a particular range of addresses may be in shared memory for one memory configuration, but may be addressable by a single PRN for other configurations.

While not being limited to any particular performance indicator or diagnostic identifier, preferred embodiments of the invention can be identified one at a time by testing for the substantially highest performance. The test for the substantially highest performance can be carried out without undue experimentation by the use of a simple and conventional benchmark (speed) experiment.

The term substantially, as used herein, is defined as at least approaching a given state (e.g., preferably within 10% of, more preferably within 1% of, and most preferably within 0.1% of). The term coupled, as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically. The term means, as used herein, is defined as hardware, firmware and or software for achieving a result. The term program or phrase computer program, as used herein, is defined as a sequence of instructions designed for execution on a computer system. A program may include a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, and/or other sequence of instructions designed for execution on a computer system. Practical Applications of the Invention A practical application of the invention that has value within the technological arts is waveform transformation. Further, the invention is useful in conjunction with data input and transformation (such as are used for the purpose of speech recognition), or in conjunction with transforming the appearance of a display (such as are used for the purpose of video games), or the like. There are virtually innumerable uses for the invention, all of which need not be detailed here.

Advantages of the Invention A system, representing an embodiment of the invention, can be cost effective and advantageous for at least the following reasons. The invention improves the speed of parallel computing systems. The invention improves the scalability of parallel computing systems.

All the disclosed embodiments of the invention described herein can be realized and practiced without undue experimentation. Although the best mode of carrying out the invention contemplated by the inventors is disclosed above, practice of the invention is not limited thereto. Accordingly, it will be appreciated by those skilled in the art that the invention may be practiced otherwise than as specifically described herein.

For example, although the switchable shared-memory cluster described herein can be a separate module, it will be manifest that the switchable shared- memory cluster may be integrated into the system with which it is associated. Furthermore, all the disclosed elements and features of each disclosed embodiment can be combined with, or substituted for, the disclosed elements and features of every other disclosed embodiment except where such elements or features are mutually exclusive.

It will be manifest that various additions, modifications and rearrangements of the features of the invention may be made without deviating from the spirit and scope of the underlying inventive concept. It is intended that the scope of the invention as defined by the appended claims and their equivalents cover all such additions, modifications, and rearrangements.

The appended claims are not to be interpreted as including means-plus- function limitations, unless such a limitation is explicitly recited in a given claim using the phrase "means for." Expedient embodiments of the invention are differentiated by the appended subclaims.

Claims

CLAIMS What is claimed is:

1. An apparatus, comprising: a shared memory unit; a first processing node coupled to said shared memory unit; and a second processing node coupled to said shared memory unit, wherein a memory page can be switched from a shared state to a private state or from said private state to a shared state.

2. The apparatus of claim 1, wherein said memory page is dynamically switchable between said shared state and said private state.

3. The apparatus of claim 1, wherein said memory page is switched by changing characteristics of said memory page.

4. The apparatus of claim 1 , wherein said memory page composes one of a plurality of banks of memory, each of said banks separately switchable from said shared state to said private state or from said private state to a shared state.

5. A computer system comprising the apparatus of claim 1.

6. A method, comprising switching a page of memory from a shared state to a private state or from a private state to a shared state.

7. The method of claim 6, wherein said page of memory is switched from the shared state to the private state; and then further comprising switching said page of memory from the private state to a shared state.

8. The method of claim 6, wherein said page of memory is switched from the private state to the shared state; and then further comprising switching said page of memory from the shared state to a private state.

9. The method of claim 6, wherein said page of memory composes one of a plurality of ranges of memory.

10. The method of claim 6, further comprising releasing a lock on said page of memory before switching said page of memory from said shared state to said private state or from said private state to said shared state

11. The method of claim 10, further comprising requesting the release of the lock on said page of memory before releasing the lock.

12. The method of claim 1 1, wherein requesting the release includes providing an augmentation selected from the group consisting of exclusive private access requested; shared private access requested; exclusive private access required; and shared private access required.

13. An electronic media, comprising: a computer program adapted to switch a page of memory from a shared state to a private state or from a private state to a shared state.

14. A computer program comprising computer program means adapted to perform the steps of switching a page of memory from a shared state to a private state or from a private state to a shared state when said computer program is run on a computer.

15. A computer program as claimed in claim 14, embodied on a computer- readable medium.

16. A system, comprising a multiplicity of workstations and a shared memory unit (SMU) interconnected and arranged such that memory accesses by a given workstation in a first set of address ranges will be to its local, private memory and that memory accesses to a second set of address ranges will be to shared memory, and arranged such that said accesses to shared memory are recorded and signaled by a Cache Emulation Adapter (CEA) capable of recognizing and responding to cache-coherence signals within the workstation.

17. The system of claim 16, further comprising means for signaling between the CEA and the SMU to LOAD or STORE shared-memory values from or to the SMU.

18. The system of claim 17, further comprising means for signaling between the CEA units and the SMU sufficient to satisfy cache coherence operations across the system when said cache coherence operations involve shared memory.

19. The system of claim 16, wherein the SMU of said system includes a directory for keeping track of which workstation has which cache line.

20. The system of claim 16, wherein said SMU cache directory includes means for keeping track of the ownership status of each workstation-owned cache line (READ SHARED, READ EXCLUSIVE, WRITE).

21. The system of claim 16, wherein each CEA includes a directory of which workstation has which cache line.

22. The system of claim 16, wherein said CEA cache directory includes means for keeping track of the ownership status of each workstation-owned cache line (READ SHARED, READ EXCLUSIVE, WRITE).

23. The system of claim 16, wherein said CEA includes caching of shared- memory accesses.

24. A parallel computing system which has multiple PRNs each with private memory and which also is provided with some shared memory, means to switch some portion memory on a dynamic basis from shared memory to private memory.

25. A system of claim 24, in which the memory which is switched may be switched to one particular PRN at one time and then may be switched to another

PRN at another time, and to fully shared at yet other times.

26. A system of claim 25, in which memory which is switched to private memory may be cached by one PRN for exclusive (Read and Write) use for a period of time.