US20020162050A1 - System and method for recovering from memory failures in computer systems - Google Patents
System and method for recovering from memory failures in computer systems Download PDFInfo
- Publication number
- US20020162050A1 US20020162050A1 US09/845,469 US84546901A US2002162050A1 US 20020162050 A1 US20020162050 A1 US 20020162050A1 US 84546901 A US84546901 A US 84546901A US 2002162050 A1 US2002162050 A1 US 2002162050A1
- Authority
- US
- United States
- Prior art keywords
- memory access
- error
- logged
- instruction sequence
- access error
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0772—Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/073—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a memory management context, e.g. virtual memory or cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1405—Saving, restoring, recovering or retrying at machine instruction level
- G06F11/141—Saving, restoring, recovering or retrying at machine instruction level for bus or memory accesses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
Definitions
- the present invention relates generally to systems and methods for recoverable programming, and more particularly to a recoverable programming system and method for memory system failures in multi-processor computer systems.
- SW software
- HW hardware
- Tandem indicates that such errors also apply to processor cores or on-chip caches at modern die sizes/voltage levels. They claim that processors, cache, and main memory are all susceptible to high transient error rates.
- a typical processor's silicon can have a soft-error rate of 4000 FYIT, of which approximately 50% will affect processor logic and 50% the large on-chip cache. Due to increasing speeds, denser technology, and lower voltages, such errors are likely to become more probable than other single hardware component failures. With the increasing evolution to larger tightly interconnected commodity machines (such as Sun's Enterprise 10000 machines), the probability of soft-errors and error containment problems increases further.
- Soft-error probability increases not only due to increased system scale, but also due to an increased number of components on the memory access path. Since the machines are tightly coupled, memory path soft-errors introduce error containment problems which without some form of soft-ware error containment can lead to complete loss of availability.
- ECC Error Correction Codes
- ChipKill see, Dell, T. J., “A White Paper on the benefits of Chipkill Correct ECC for PC Server Main Memory”, IBM Microelectronics Division, November 1997) have been used in main memories and interconnects to correct some of these errors (90% for ECC).
- ECC Error Correction Codes
- a 1 Gb memory system based on 64 Mbit DRAMs still has a combined visible error rate of 3435 FIT when using Single Error Correct Double Error Detect (SECDED) ECC.
- SECDED Single Error Correct Double Error Detect
- This is equivalent to around 900 errors in 10000 machines in 3 years.
- current commodity hardware and software provide little to no support for recovery from errors not covered by ECC whether detected or not.
- Such problems have been considered by mainframe technology for years, but in the field of commodity hardware, it is currently not cost effective to provide full redundancy/support in order to mask errors. Therefore, the burden falls to commodity hardware and the software using it to attempt to handle these errors for the highest availability.
- Availability in computer systems is determined by hardware and software reliability. Hardware reliability has traditionally existed only in proprietary servers, with specialized redundantly configured hardware and critical software components, possibly with support for processor pairs (see, Bartlett, J., “A Nonstop Kernel”, Proceedings of the Eighth Symposium on Operating Systems Principles, Asilomar, Ca, pp 22-29, December 1981 ), e.g. IBM S/390 Parallel Sysplex (see, Nick, J. M., et al., “S/390 Cluster Technology: Parallel Sysplex”, IBMSystems Journal, vol 36, no 2., pp 172-201, 1997), and Tandem NonStop Himalaya (see, Compaq, Product description for Tandem Nonstop Kernel 3.0.
- Sysplex supports hot swap execution, redundant shared disk with fault-aware system software for error detection and fail-over restart.
- Tandem supports redundant fail-over lock-stepped processors with a NonStop kernel and middleware, which provide improved integrity through the software stack.
- These systems provide full automatic support to mask the effects of data and resource loss. They rely on reliable memory and fail-over rather than direct memory error recovery.
- Another approach is fault containment and recovery with “node” granularity. In these systems, each node has its own kernel. When one node fails, the others can recover and continue to provide services.
- Systems of this type include the early cluster systems (see, Pfister, G., “In Search of Clusters”, Prentice Hall, 1998), and NUMA architectures, such as Hive (see, Chapin, J., et al., “Hive: Fault Containment for Shared Memory Multiprocessors,” Proc. of the 15 th SOSP, December 1995, pp 12-25, and Teodosiu, D., et al., “Hardware Fault Containment in Scalable Shared Memory Multiprocessors,” Proc. of the 24 th ISCA, pp 73-84, June 1997). Hardware faults are difficult to catch and repeat. Software reliability has been more difficult to achieve in commodity software even with extensive testing and quality assurance.
- the present invention is a system and method for recovering from memory failures in computer systems.
- the method of the present invention includes the steps of: identifying a predetermined instruction sequence; monitoring for memory access errors in response to the request; logging a memory access error in an error logging register; polling the register for any logged memory access error during execution of the instruction sequence; and raising exceptions, if the memory access error is logged.
- the method may include the steps of: checkpointing a predetermined set of system data; recovering from the memory access error using the checkpointed system data, if the memory access error is logged during execution of the instruction sequence; setting data returned in response to the memory access request equal to a set of predefined fake data, if the memory access error is logged during execution of the instruction sequence; skipping the polling and raising steps if the data returned in response to the memory access request is not equivalent to the predefined fake data; masking a machine check abort handle; updating pointers, if the memory access error is logged; and re-executing the memory access request, if software so commands.
- memory access errors are stored in an error logging register, hardware generated machine check abort handles are masked, and a memory controller, which responds to memory access errors is under full control of either O/S or application software so that memory access errors are intercepted and responded to without necessitating a system reboot.
- the present invention is particularly applicable to O/S level code which can not otherwise be restarted in response to memory errors without rebooting.
- the present invention is incorporated within application level code, the present invention also enables the application to recover from memory errors, instead of otherwise being shut down and restarted.
- FIG. 1 is a dataflow diagram of a system for recovering from memory access failures
- FIG. 2 is a flowchart of a first embodiment of a method for recovering from memory access failures
- FIG. 3 is a flowchart of a second embodiment of the method for recovering from memory access failures.
- FIG. 4 is a flowchart of a third embodiment of the method for recovering from memory access failures.
- FIG. 1 is a dataflow diagram of a system 100 for recovering from memory access failures.
- the system 100 includes a memory 102 , a memory controller 104 , error logging registers 106 , and a central processing unit (CPU) 108 , coupled together by a bus 110 .
- the CPU 108 is controlled by software 112 .
- the software 112 is preferably included within the system's 100 operating system, however the software 112 could also be instantiated within an application program as well.
- the software 112 configures the memory controller 104 and has access to the error registers 106 . Operation of the system 100 is discussed with respect to method FIGS. 2, 3, and 4 .
- FIG. 2 is a flowchart of a first embodiment of a method 200 for recovering from memory access failures.
- the method 200 begins with step 202 where the software 112 identifies a predetermined critical computer instruction sequence about to be executed by the CPU 108 , which includes a memory access request.
- the predetermined critical computer instruction sequence can be part of a set of instruction sequences, identified by the software 112 designer, for which error recovery is required. While the critical computer instruction sequence discussed below include a memory access request, those skilled in the art will know that concepts discussed with respect to recovery from a memory access request error can be applied to other recovery critical instruction sequences which would otherwise require rebooting of the system 100 to recover from.
- the present invention is particularly applicable to O/S level code which can not otherwise be restarted in response to memory errors without rebooting.
- the present invention is incorporated within application level code, the present invention enables the application to recover from memory errors, instead of otherwise being shut down and restarted.
- the present invention may also be used on non-critical computer instruction sequences and for non-memory related errors.
- step 204 the software 112 then instructs the memory controller 104 to mask any raised machine check abort (MCA) handle.
- MCA machine check abort
- step 206 the CPU 108 executes the memory access request.
- the memory controller 104 logs any memory access error in the error logging register 106 , in step 208 .
- step 210 the software 112 polls the error logging register 106 for any memory access errors, during execution of the instruction sequence.
- step 212 the software 112 raises exceptions and updates pointers, if a memory access error was logged during execution of the instruction sequence.
- the exceptions perform various diagnostic functions in response to the memory error.
- the housekeeping functions may include system recovery, memory management, and other reset procedures. Pointers are updated when during memory error diagnosis, there are indications that a portion or sector of the memory 102 may be physically damaged or corrupt.
- the software 112 may command the CPU 108 to re-execute the memory access request, in step 214 .
- the software 112 will command the CPU 108 to re-execute the memory access request if the memory access error detected is most likely due to a transitory error condition, which is not likely to occur again.
- the software 112 will not instruct the CPU 108 to re-execute the memory access request.
- the software 112 instructs the memory controller 104 to enable the MCA handle.
- FIG. 3 is a flowchart of a second embodiment of the method for recovering from memory access failures.
- the method 300 begins with step 302 where the software 112 identifies a predetermined critical computer instruction sequence about to be executed by the CPU 108 , which includes a memory access request.
- step 304 the software 112 then checkpoints a predetermined set of system data necessary to recover should the memory access request fail.
- Checkpointing is component of a transactional paradigm in which permanent modifications to system data are not made until all associated operations within the transaction have been successfully committed. Thus if during a transaction, such as the memory access request, an error occurs, the system data stored during the checkpoint can be restored.
- step 306 the software 112 then instructs the memory controller 104 to mask any raised machine check abort (MCA) handle.
- MCA machine check abort
- step 308 the CPU 108 executes the memory access request.
- the memory controller 104 logs any memory access error in the error logging register 106 , in step 310 .
- step 312 the software 112 polls the error logging register 106 for any memory access errors, during execution of the instruction sequence. If a memory access error is logged during execution of the instruction sequence, the software 112 : raises exceptions and updates pointers, in step 314 ; recovers the checkpointed system data, in step 316 ; and restores the system data, in step 318
- the software 112 may command the CPU 108 to re-execute the memory access request, in step 320 .
- the software 112 instructs the memory controller 104 to enable the MCA handle.
- FIG. 4 is a flowchart of a third embodiment of the method for recovering from memory access failures.
- the method 400 begins with step 402 where the software 112 identifies a predetermined critical computer instruction sequence about to be executed by the CPU 108 , which includes a memory access request.
- the software 112 then instructs the memory controller 104 to mask any raised machine check abort (MCA) handle.
- MCA machine check abort
- the CPU 108 executes the memory access request.
- the memory controller 104 logs any memory access error in the error logging register 106 , in step 408 .
- step 410 the memory controller 104 sets data returned in response to the memory access request equal to a set of predefined fake data, if a memory access error is logged during execution of the instruction sequence.
- the software 112 has preprogrammed the memory controller 104 to perform the functionality described in step 410 .
- step 412 the software 112 receives data returned in response to the memory access request.
- step 414 the method 400 skips to step 422 , if the data returned in response to the memory access request is not equivalent to the predefined fake data.
- the software 112 knows that no memory access error has occurred, during execution of the instruction sequence, even though the software 112 has not polled the error logging register. Thus, the polling step can be eliminated, speeding up the memory access request.
- step 416 the software 112 polls the error logging register 106 for any memory access errors, during execution of the instruction sequence.
- the software 112 raises exceptions and updates pointers, if a memory access error was logged during execution of the instruction sequence.
- the software 112 may command the CPU 108 to re-execute the memory access request, in step 420 .
- the software 112 instructs the memory controller 104 to enable any hardware raised MCA handles.
- Another enhancement which may be applied to each of the three embodiments discussed above, is to batch access to memory in large chunks whenever possible. By batch accessing data, memory access errors are logged and polled for the entire batch. This has implication on a granularity of the system 100 operation and is limited by pointer manipulation.
Abstract
Description
- 1. Field of the Invention
- The present invention relates generally to systems and methods for recoverable programming, and more particularly to a recoverable programming system and method for memory system failures in multi-processor computer systems.
- 2. Discussion of Background Art
- Demand for increased performance and high availability of commodity computers is increasing with the ubiquitous use of computers and the Internet services which serve them. While commodity systems are tackling the performance issues, availability has received less attention. It is a common belief that software (SW) errors and administration time are, and will continue to be, the most probable cause of the loss of availability. While such failures are clearly commonplace, especially in desktop environments, it is believed that certain other hardware (HW) errors are also becoming more probable.
- Processors, caches, and memories are becoming larger, faster and more dense, while being increasingly used in ubiquitous and adverse environments such as at high altitudes, in space, and in industrial applications. Articles, such as Ziegler, J. F., et al., “IBM Experiments in Soft Fails in Computer Electronics (1978-1994)”, IBM Journal of R&D, vol 40, no 1, pp 3-18, January 1996, and Ziegler, J. F., “Terrestrial Cosmic Rays”, IBM Journal of R&D, vol 40, no 1, pp 19-40, January 1996, have shown that these changes will lead to increased transient errors in CMOS memory due to the effects of cosmic rays, approximately 6000 FIT (1 FIT equals 1 failure in 10 9 h) for one 4 Mbit DRAM.
- Tandem (see, Compaq Corporation, “Data Integrity for Compaq NonStop Himalaya Servers”, White Paper, 1999) indicates that such errors also apply to processor cores or on-chip caches at modern die sizes/voltage levels. They claim that processors, cache, and main memory are all susceptible to high transient error rates. A typical processor's silicon can have a soft-error rate of 4000 FYIT, of which approximately 50% will affect processor logic and 50% the large on-chip cache. Due to increasing speeds, denser technology, and lower voltages, such errors are likely to become more probable than other single hardware component failures. With the increasing evolution to larger tightly interconnected commodity machines (such as Sun's Enterprise 10000 machines), the probability of soft-errors and error containment problems increases further. Soft-error probability increases not only due to increased system scale, but also due to an increased number of components on the memory access path. Since the machines are tightly coupled, memory path soft-errors introduce error containment problems which without some form of soft-ware error containment can lead to complete loss of availability.
- Techniques such as Error Correction Codes (ECC) and ChipKill (see, Dell, T. J., “A White Paper on the benefits of Chipkill Correct ECC for PC Server Main Memory”, IBM Microelectronics Division, November 1997) have been used in main memories and interconnects to correct some of these errors (90% for ECC). Unfortunately such techniques, only help reduce visible error rates for semiconductor elements that can be covered by such codes (large storage elements). With raw error rates increasing with technological progress and more complicated interconnected memory subsystems, ECC is unable to address all the soft-error problems. For example, a 1 Gb memory system based on 64 Mbit DRAMs still has a combined visible error rate of 3435 FIT when using Single Error Correct Double Error Detect (SECDED) ECC. This is equivalent to around 900 errors in 10000 machines in 3 years. Unfortunately, current commodity hardware and software provide little to no support for recovery from errors not covered by ECC whether detected or not. Such problems have been considered by mainframe technology for years, but in the field of commodity hardware, it is currently not cost effective to provide full redundancy/support in order to mask errors. Therefore, the burden falls to commodity hardware and the software using it to attempt to handle these errors for the highest availability.
- Most contemporary commodity computer systems, while providing good performance, pay little attention to availability issues resulting from such errors. For example, the IA-32 architecture supports only ECC on main memory rather than across the system, requiring system reboot on errors not covered by this ECC. Consequently, commodity software such as the OS, middleware and applications have been unable to deal with the problem. Future commodity processor architectures may provide support to detect and notify the system of such probable errors. For instance, upcoming IA-64 processors, while not recoverable in the general case, do offer some support with certain limitations.
- Availability in computer systems is determined by hardware and software reliability. Hardware reliability has traditionally existed only in proprietary servers, with specialized redundantly configured hardware and critical software components, possibly with support for processor pairs (see, Bartlett, J., “A Nonstop Kernel”, Proceedings of the Eighth Symposium on Operating Systems Principles, Asilomar, Ca, pp 22-29, December1981), e.g. IBM S/390 Parallel Sysplex (see, Nick, J. M., et al., “S/390 Cluster Technology: Parallel Sysplex”, IBMSystems Journal, vol 36, no 2., pp 172-201, 1997), and Tandem NonStop Himalaya (see, Compaq, Product description for Tandem Nonstop Kernel 3.0. Download February 2000, http://www.tandem.com). Sysplex supports hot swap execution, redundant shared disk with fault-aware system software for error detection and fail-over restart. Tandem supports redundant fail-over lock-stepped processors with a NonStop kernel and middleware, which provide improved integrity through the software stack. These systems provide full automatic support to mask the effects of data and resource loss. They rely on reliable memory and fail-over rather than direct memory error recovery. Another approach is fault containment and recovery with “node” granularity. In these systems, each node has its own kernel. When one node fails, the others can recover and continue to provide services. Systems of this type include the early cluster systems (see, Pfister, G., “In Search of Clusters”, Prentice Hall, 1998), and NUMA architectures, such as Hive (see, Chapin, J., et al., “Hive: Fault Containment for Shared Memory Multiprocessors,” Proc. of the 15th SOSP, December 1995, pp 12-25, and Teodosiu, D., et al., “Hardware Fault Containment in Scalable Shared Memory Multiprocessors,” Proc. of the 24th ISCA, pp 73-84, June 1997). Hardware faults are difficult to catch and repeat. Software reliability has been more difficult to achieve in commodity software even with extensive testing and quality assurance. Commodity software fault recovery has not evolved very far. Most operating systems support some form of memory protection between units of execution to detect and prevent wild read/writes. But most commodity operating systems have not tackled problems of memory errors themselves or taken up software reliability research in general. Examples include Windows 2000 and Linux. They typically rely on failover solutions, such as Wolfpack by Microsoft. A lot of work has been undertaken in the fault-tolerant community regarding the problems of reliability and its recovery in software (see, Brown, N. S. and Pradhan, D. K. “Processor and Memory-Based Checkpoint And Rollback Recovery”, IEEE Computer, pp 22-31, February 1993; Gray, J., and Reuter, A., “Transaction Processing: Concepts and Techniques,” Morgan Kaufmann, 1993; and Kermarrec, A M., et al., “A Recoverable Distributed Shared Memory Integrating Coherence and Recoverability”, Proc. of the 25th FTCS, pp 289-298, June 1995). These include techniques such as checkpointing and backward error recovery. A lot of this work has been conducted in the context of distributed systems rather than in single systems. There are also techniques for efficient recoverable software components, e.g. RIO file cache (see, Chen, P. M., et al., “The Rio File Cache: Surviving Operating System Crashes”, Proc. of the 7th ASPLOS, pp 74-83, October 1996), and Recoverable Virtual Memory (RVM) (see, Satyanarayanan, et al. “Lightweight Recoverable Virtual Memory”. Proc. SOSP, pp 146-160, December 1993).
- Rio takes an interesting software-based approach to fault containment aimed at a fault-tolerant file cache, but with general uses. By instrumenting access to shared data structures with memory protection operations, wild access to the shared data structures becomes improbable.
- Other methods for handling memory errors include a try-except block solution. In general, the try-except mechanism itself is not sufficient to handle memory failures. The saved state needed for memory failures is more extensive (as an example, for IA-64 architecture) than what can be obtained by try-except. Thus saving state is an expensive operation in terms of system overhead.
- Since current responses to memory failures are costly to invoke and execute, do not guarantee recovery under all cases for next generation processors, such as IA64, and are impossible to recover at all for current generations of commodity processors, such as the IA32 family, what is needed is a system and method for recoverable programming that overcomes the problems of the prior art.
- The present invention is a system and method for recovering from memory failures in computer systems. The method of the present invention includes the steps of: identifying a predetermined instruction sequence; monitoring for memory access errors in response to the request; logging a memory access error in an error logging register; polling the register for any logged memory access error during execution of the instruction sequence; and raising exceptions, if the memory access error is logged.
- In other aspects of the invention, the method may include the steps of: checkpointing a predetermined set of system data; recovering from the memory access error using the checkpointed system data, if the memory access error is logged during execution of the instruction sequence; setting data returned in response to the memory access request equal to a set of predefined fake data, if the memory access error is logged during execution of the instruction sequence; skipping the polling and raising steps if the data returned in response to the memory access request is not equivalent to the predefined fake data; masking a machine check abort handle; updating pointers, if the memory access error is logged; and re-executing the memory access request, if software so commands.
- Within the system of the present invention, memory access errors are stored in an error logging register, hardware generated machine check abort handles are masked, and a memory controller, which responds to memory access errors is under full control of either O/S or application software so that memory access errors are intercepted and responded to without necessitating a system reboot.
- The system and method of the present invention are particularly advantageous over the prior art because memory failures in commodity multiprocessor computers can now be responded to and remedied without rebooting the computer. The present invention will succeed in responding to memory errors much more effectively than standard machine check abort handles.
- The present invention is particularly applicable to O/S level code which can not otherwise be restarted in response to memory errors without rebooting. When the present invention is incorporated within application level code, the present invention also enables the application to recover from memory errors, instead of otherwise being shut down and restarted.
- These and other aspects of the invention will be recognized by those skilled in the art upon review of the detailed description, drawings, and claims set forth below.
- FIG. 1 is a dataflow diagram of a system for recovering from memory access failures;
- FIG. 2 is a flowchart of a first embodiment of a method for recovering from memory access failures;
- FIG. 3 is a flowchart of a second embodiment of the method for recovering from memory access failures; and
- FIG. 4 is a flowchart of a third embodiment of the method for recovering from memory access failures.
- FIG. 1 is a dataflow diagram of a
system 100 for recovering from memory access failures. Thesystem 100 includes amemory 102, amemory controller 104, error logging registers 106, and a central processing unit (CPU) 108, coupled together by abus 110. TheCPU 108 is controlled bysoftware 112. Thesoftware 112 is preferably included within the system's 100 operating system, however thesoftware 112 could also be instantiated within an application program as well. Thesoftware 112 configures thememory controller 104 and has access to the error registers 106. Operation of thesystem 100 is discussed with respect to method FIGS. 2, 3, and 4. - FIG. 2 is a flowchart of a first embodiment of a
method 200 for recovering from memory access failures. Themethod 200 begins withstep 202 where thesoftware 112 identifies a predetermined critical computer instruction sequence about to be executed by theCPU 108, which includes a memory access request. The predetermined critical computer instruction sequence can be part of a set of instruction sequences, identified by thesoftware 112 designer, for which error recovery is required. While the critical computer instruction sequence discussed below include a memory access request, those skilled in the art will know that concepts discussed with respect to recovery from a memory access request error can be applied to other recovery critical instruction sequences which would otherwise require rebooting of thesystem 100 to recover from. Thus, the present invention is particularly applicable to O/S level code which can not otherwise be restarted in response to memory errors without rebooting. When the present invention is incorporated within application level code, the present invention enables the application to recover from memory errors, instead of otherwise being shut down and restarted. The present invention may also be used on non-critical computer instruction sequences and for non-memory related errors. - In
step 204, thesoftware 112 then instructs thememory controller 104 to mask any raised machine check abort (MCA) handle. Next instep 206, theCPU 108 executes the memory access request. Thememory controller 104 logs any memory access error in theerror logging register 106, instep 208. Next, instep 210, thesoftware 112 polls theerror logging register 106 for any memory access errors, during execution of the instruction sequence. - In
step 212, thesoftware 112 raises exceptions and updates pointers, if a memory access error was logged during execution of the instruction sequence. The exceptions perform various diagnostic functions in response to the memory error. The housekeeping functions may include system recovery, memory management, and other reset procedures. Pointers are updated when during memory error diagnosis, there are indications that a portion or sector of thememory 102 may be physically damaged or corrupt. - Depending upon the memory access error which occurred, the
software 112 may command theCPU 108 to re-execute the memory access request, instep 214. Thesoftware 112 will command theCPU 108 to re-execute the memory access request if the memory access error detected is most likely due to a transitory error condition, which is not likely to occur again. On the other hand, if the memory access error suggest that thememory 102 itself is physically damaged, thesoftware 112 will not instruct theCPU 108 to re-execute the memory access request. Instep 216, thesoftware 112 instructs thememory controller 104 to enable the MCA handle. - FIG. 3 is a flowchart of a second embodiment of the method for recovering from memory access failures. The
method 300 begins withstep 302 where thesoftware 112 identifies a predetermined critical computer instruction sequence about to be executed by theCPU 108, which includes a memory access request. - In
step 304, thesoftware 112 then checkpoints a predetermined set of system data necessary to recover should the memory access request fail. Checkpointing is component of a transactional paradigm in which permanent modifications to system data are not made until all associated operations within the transaction have been successfully committed. Thus if during a transaction, such as the memory access request, an error occurs, the system data stored during the checkpoint can be restored. - In
step 306, thesoftware 112 then instructs thememory controller 104 to mask any raised machine check abort (MCA) handle. Instep 308, theCPU 108 executes the memory access request. Thememory controller 104 logs any memory access error in theerror logging register 106, instep 310. Next, instep 312, thesoftware 112 polls theerror logging register 106 for any memory access errors, during execution of the instruction sequence. If a memory access error is logged during execution of the instruction sequence, the software 112: raises exceptions and updates pointers, instep 314; recovers the checkpointed system data, instep 316; and restores the system data, instep 318 - As discussed above, with reference to FIG. 2, depending upon the memory access error which occurred, the
software 112 may command theCPU 108 to re-execute the memory access request, instep 320. Instep 322, thesoftware 112 instructs thememory controller 104 to enable the MCA handle. - FIG. 4 is a flowchart of a third embodiment of the method for recovering from memory access failures. The method400 begins with
step 402 where thesoftware 112 identifies a predetermined critical computer instruction sequence about to be executed by theCPU 108, which includes a memory access request. Instep 404, thesoftware 112 then instructs thememory controller 104 to mask any raised machine check abort (MCA) handle. Instep 406, theCPU 108 executes the memory access request. Thememory controller 104 logs any memory access error in theerror logging register 106, instep 408. - In
step 410, thememory controller 104 sets data returned in response to the memory access request equal to a set of predefined fake data, if a memory access error is logged during execution of the instruction sequence. Thesoftware 112 has preprogrammed thememory controller 104 to perform the functionality described instep 410. By setting the returned data to the predefined fake data in when a memory access error occurs, corrupted data is not returned to the software, which might otherwise necessitate a system reboot. - In
step 412, thesoftware 112 receives data returned in response to the memory access request. Instep 414, the method 400 skips to step 422, if the data returned in response to the memory access request is not equivalent to the predefined fake data. When the data returned is not equal to the fake data, thesoftware 112 knows that no memory access error has occurred, during execution of the instruction sequence, even though thesoftware 112 has not polled the error logging register. Thus, the polling step can be eliminated, speeding up the memory access request. - In
step 416, thesoftware 112 polls theerror logging register 106 for any memory access errors, during execution of the instruction sequence. Instep 418, thesoftware 112 raises exceptions and updates pointers, if a memory access error was logged during execution of the instruction sequence. - As discussed above, with reference to FIG. 2, depending upon the memory access error which occurred, the
software 112 may command theCPU 108 to re-execute the memory access request, instep 420. Instep 422, thesoftware 112 instructs thememory controller 104 to enable any hardware raised MCA handles. - Another enhancement which may be applied to each of the three embodiments discussed above, is to batch access to memory in large chunks whenever possible. By batch accessing data, memory access errors are logged and polled for the entire batch. This has implication on a granularity of the
system 100 operation and is limited by pointer manipulation. - While one or more embodiments of the present invention have been described, those skilled in the art will recognize that various modifications may be made. Variations upon and modifications to these embodiments are provided by the present invention, which is limited only by the following claims.
Claims (19)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/845,469 US6851074B2 (en) | 2001-04-30 | 2001-04-30 | System and method for recovering from memory failures in computer systems |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/845,469 US6851074B2 (en) | 2001-04-30 | 2001-04-30 | System and method for recovering from memory failures in computer systems |
Publications (2)
Publication Number | Publication Date |
---|---|
US20020162050A1 true US20020162050A1 (en) | 2002-10-31 |
US6851074B2 US6851074B2 (en) | 2005-02-01 |
Family
ID=25295305
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/845,469 Expired - Lifetime US6851074B2 (en) | 2001-04-30 | 2001-04-30 | System and method for recovering from memory failures in computer systems |
Country Status (1)
Country | Link |
---|---|
US (1) | US6851074B2 (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070006015A1 (en) * | 2005-06-29 | 2007-01-04 | Rao Sudhir G | Fault-tolerance and fault-containment models for zoning clustered application silos into continuous availability and high availability zones in clustered systems during recovery and maintenance |
US20080126780A1 (en) * | 2006-09-20 | 2008-05-29 | Anurupa Rajkumari | Containing machine check events in a virtual partition |
FR3040523A1 (en) * | 2015-08-28 | 2017-03-03 | Continental Automotive France | METHOD OF DETECTING AN UNCOMPRIGIBLE ERROR IN A NON-VOLATILE MEMORY OF A MICROCONTROLLER |
US10346237B1 (en) * | 2015-08-28 | 2019-07-09 | EMC IP Holding Company LLC | System and method to predict reliability of backup software |
CN110286962A (en) * | 2019-06-28 | 2019-09-27 | 深圳市元征科技股份有限公司 | A kind of encryption chip fault recovery method, system and electronic equipment and storage medium |
US20220206899A1 (en) * | 2020-12-24 | 2022-06-30 | Advanced Micro Devices, Inc. | Method and apparatus to support instruction replay for executing idempotent code in dependent processing in memory devices |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8403478B2 (en) * | 2001-11-02 | 2013-03-26 | High Performance Optics, Inc. | Ophthalmic lens to preserve macular integrity |
US8500274B2 (en) | 2000-11-03 | 2013-08-06 | High Performance Optics, Inc. | Dual-filter ophthalmic lens to reduce risk of macular degeneration |
US7000100B2 (en) * | 2001-05-31 | 2006-02-14 | Hewlett-Packard Development Company, L.P. | Application-level software watchdog timer |
JP2004062309A (en) * | 2002-07-25 | 2004-02-26 | Fujitsu Ltd | Method of processing illegal instruction and processor |
US7526598B2 (en) * | 2003-03-03 | 2009-04-28 | Sandisk Il, Ltd. | Efficient flash memory device driver |
US7149869B2 (en) * | 2003-05-30 | 2006-12-12 | Sun Microsystems, Inc. | Method and apparatus for generating generic descrambled data patterns for testing ECC protected memory |
US7360114B2 (en) * | 2003-06-17 | 2008-04-15 | International Business Machines Corporation | Logging of exception data |
US7343521B2 (en) * | 2004-05-28 | 2008-03-11 | International Business Machines Corporation | Method and apparatus to preserve trace data |
US9377569B2 (en) * | 2006-03-20 | 2016-06-28 | High Performance Optics, Inc. | Photochromic ophthalmic systems that selectively filter specific blue light wavelengths |
US8360574B2 (en) * | 2006-03-20 | 2013-01-29 | High Performance Optics, Inc. | High performance selective light wavelength filtering providing improved contrast sensitivity |
US8113651B2 (en) * | 2006-03-20 | 2012-02-14 | High Performance Optics, Inc. | High performance corneal inlay |
US8882267B2 (en) | 2006-03-20 | 2014-11-11 | High Performance Optics, Inc. | High energy visible light filter systems with yellowness index values |
US20120075577A1 (en) | 2006-03-20 | 2012-03-29 | Ishak Andrew W | High performance selective light wavelength filtering providing improved contrast sensitivity |
US8291379B2 (en) * | 2006-12-13 | 2012-10-16 | International Business Machines Corporation | Runtime analysis of a computer program to identify improper memory accesses that cause further problems |
US20080151181A1 (en) * | 2006-12-20 | 2008-06-26 | Bausch & Lomb Incorporated | Coatings and Solutions for Contact Lenses |
US9495278B2 (en) * | 2006-12-27 | 2016-11-15 | International Business Machines Corporation | Dynamic discovery of data segments within instrumented code |
JP4485591B2 (en) * | 2007-02-09 | 2010-06-23 | 富士通株式会社 | Degeneration method and information processing apparatus |
US7702971B2 (en) * | 2007-06-06 | 2010-04-20 | Dell Products L.P. | System and method for predictive failure detection |
US7818633B2 (en) * | 2007-06-29 | 2010-10-19 | International Business Machines Corporation | Method and apparatus for identification of program check errors indicating code with high potential for storage overlay |
US20120185741A1 (en) * | 2011-01-14 | 2012-07-19 | Sergey Sergeevich Grekhov | Apparatus and method for detecting a memory access error |
US9830224B2 (en) * | 2013-03-15 | 2017-11-28 | Nvidia Corporation | Selective fault stalling for a GPU memory pipeline in a unified virtual memory system |
US9798163B2 (en) | 2013-05-05 | 2017-10-24 | High Performance Optics, Inc. | Selective wavelength filtering with reduced overall light transmission |
US9683102B2 (en) | 2014-05-05 | 2017-06-20 | Frontier Scientific, Inc. | Photo-stable and thermally-stable dye compounds for selective blue light filtered optic |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4174537A (en) * | 1977-04-04 | 1979-11-13 | Burroughs Corporation | Time-shared, multi-phase memory accessing system having automatically updatable error logging means |
US4371949A (en) * | 1977-05-31 | 1983-02-01 | Burroughs Corporation | Time-shared, multi-phase memory accessing system having automatically updatable error logging means |
US4811347A (en) * | 1986-01-30 | 1989-03-07 | U.S. Philips Corporation | Apparatus and method for monitoring memory accesses and detecting memory errors |
US5261084A (en) * | 1988-05-06 | 1993-11-09 | Nec Corporation | Error judgment method |
US5428766A (en) * | 1992-12-01 | 1995-06-27 | Digital Equipment Corporation | Error detection scheme in a multiprocessor environment |
US5644709A (en) * | 1994-04-21 | 1997-07-01 | Wisconsin Alumni Research Foundation | Method for detecting computer memory access errors |
US5867642A (en) * | 1995-08-10 | 1999-02-02 | Dell Usa, L.P. | System and method to coherently and dynamically remap an at-risk memory area by simultaneously writing two memory areas |
US5953530A (en) * | 1995-02-07 | 1999-09-14 | Sun Microsystems, Inc. | Method and apparatus for run-time memory access checking and memory leak detection of a multi-threaded program |
US6505305B1 (en) * | 1998-07-16 | 2003-01-07 | Compaq Information Technologies Group, L.P. | Fail-over of multiple memory blocks in multiple memory modules in computer system |
-
2001
- 2001-04-30 US US09/845,469 patent/US6851074B2/en not_active Expired - Lifetime
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4174537A (en) * | 1977-04-04 | 1979-11-13 | Burroughs Corporation | Time-shared, multi-phase memory accessing system having automatically updatable error logging means |
US4371949A (en) * | 1977-05-31 | 1983-02-01 | Burroughs Corporation | Time-shared, multi-phase memory accessing system having automatically updatable error logging means |
US4811347A (en) * | 1986-01-30 | 1989-03-07 | U.S. Philips Corporation | Apparatus and method for monitoring memory accesses and detecting memory errors |
US5261084A (en) * | 1988-05-06 | 1993-11-09 | Nec Corporation | Error judgment method |
US5428766A (en) * | 1992-12-01 | 1995-06-27 | Digital Equipment Corporation | Error detection scheme in a multiprocessor environment |
US5644709A (en) * | 1994-04-21 | 1997-07-01 | Wisconsin Alumni Research Foundation | Method for detecting computer memory access errors |
US5953530A (en) * | 1995-02-07 | 1999-09-14 | Sun Microsystems, Inc. | Method and apparatus for run-time memory access checking and memory leak detection of a multi-threaded program |
US5867642A (en) * | 1995-08-10 | 1999-02-02 | Dell Usa, L.P. | System and method to coherently and dynamically remap an at-risk memory area by simultaneously writing two memory areas |
US6505305B1 (en) * | 1998-07-16 | 2003-01-07 | Compaq Information Technologies Group, L.P. | Fail-over of multiple memory blocks in multiple memory modules in computer system |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070006015A1 (en) * | 2005-06-29 | 2007-01-04 | Rao Sudhir G | Fault-tolerance and fault-containment models for zoning clustered application silos into continuous availability and high availability zones in clustered systems during recovery and maintenance |
US8195976B2 (en) | 2005-06-29 | 2012-06-05 | International Business Machines Corporation | Fault-tolerance and fault-containment models for zoning clustered application silos into continuous availability and high availability zones in clustered systems during recovery and maintenance |
US8286026B2 (en) | 2005-06-29 | 2012-10-09 | International Business Machines Corporation | Fault-tolerance and fault-containment models for zoning clustered application silos into continuous availability and high availability zones in clustered systems during recovery and maintenance |
US20080126780A1 (en) * | 2006-09-20 | 2008-05-29 | Anurupa Rajkumari | Containing machine check events in a virtual partition |
US7657776B2 (en) | 2006-09-20 | 2010-02-02 | Hewlett-Packard Development Company, L.P. | Containing machine check events in a virtual partition |
WO2017036584A1 (en) * | 2015-08-28 | 2017-03-09 | Continental Automotive France | Method for detecting an uncorrectable error in a non-volatile memory of a microcontroller |
FR3040523A1 (en) * | 2015-08-28 | 2017-03-03 | Continental Automotive France | METHOD OF DETECTING AN UNCOMPRIGIBLE ERROR IN A NON-VOLATILE MEMORY OF A MICROCONTROLLER |
CN107924352A (en) * | 2015-08-28 | 2018-04-17 | 法国大陆汽车公司 | Not amendable wrong detection method in the nonvolatile memory of microcontroller |
US10346237B1 (en) * | 2015-08-28 | 2019-07-09 | EMC IP Holding Company LLC | System and method to predict reliability of backup software |
US10628269B2 (en) * | 2015-08-28 | 2020-04-21 | Continental Automotive France | Method for detecting an uncorrectable error in a non-volatile memory of a microcontroller |
CN110286962A (en) * | 2019-06-28 | 2019-09-27 | 深圳市元征科技股份有限公司 | A kind of encryption chip fault recovery method, system and electronic equipment and storage medium |
US20220206899A1 (en) * | 2020-12-24 | 2022-06-30 | Advanced Micro Devices, Inc. | Method and apparatus to support instruction replay for executing idempotent code in dependent processing in memory devices |
US11656945B2 (en) * | 2020-12-24 | 2023-05-23 | Advanced Micro Devices, Inc. | Method and apparatus to support instruction replay for executing idempotent code in dependent processing in memory devices |
Also Published As
Publication number | Publication date |
---|---|
US6851074B2 (en) | 2005-02-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6851074B2 (en) | System and method for recovering from memory failures in computer systems | |
US8635492B2 (en) | State recovery and lockstep execution restart in a system with multiprocessor pairing | |
Bartlett et al. | Commercial fault tolerance: A tale of two systems | |
CN100489801C (en) | Firmware mechanism for correcting soft errors | |
Spainhower et al. | IBM S/390 parallel enterprise server G5 fault tolerance: A historical perspective | |
US6622260B1 (en) | System abstraction layer, processor abstraction layer, and operating system error handling | |
US7627781B2 (en) | System and method for establishing a spare processor for recovering from loss of lockstep in a boot processor | |
US5317752A (en) | Fault-tolerant computer system with auto-restart after power-fall | |
US5815651A (en) | Method and apparatus for CPU failure recovery in symmetric multi-processing systems | |
JP4603185B2 (en) | Computer and its error recovery method | |
Namjoo et al. | Watchdog processors and capability checking | |
US20100318746A1 (en) | Memory change track logging | |
KR20010005956A (en) | Fault tolerant computer system | |
US20100313061A1 (en) | Method and system for reliable exception handling in a computer system | |
KR100304319B1 (en) | Apparatus and method for implementing time-lag duplexing techniques | |
US7366948B2 (en) | System and method for maintaining in a multi-processor system a spare processor that is in lockstep for use in recovering from loss of lockstep for another processor | |
Milojicic et al. | Increasing relevance of memory hardware errors: a case for recoverable programming models | |
EP0683456B1 (en) | Fault-tolerant computer system with online reintegration and shutdown/restart | |
US7502958B2 (en) | System and method for providing firmware recoverable lockstep protection | |
US6799285B2 (en) | Self-checking multi-threaded processor | |
Le et al. | Applying microreboot to system software | |
Smith et al. | Surviving peripheral failures in embedded systems | |
US7624302B2 (en) | System and method for switching the role of boot processor to a spare processor responsive to detection of loss of lockstep in a boot processor | |
US7356733B2 (en) | System and method for system firmware causing an operating system to idle a processor | |
JP2004252525A (en) | Emulator and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: HEWLETT-PACKARD COMPANY, COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MILOJICIC, DEJAN S.;WYLEGALA, THOMAS;PONG, FONG;AND OTHERS;REEL/FRAME:012127/0847;SIGNING DATES FROM 20010430 TO 20010525 |
|
AS | Assignment |
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY L.P., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:014061/0492 Effective date: 20030926 Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY L.P.,TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:014061/0492 Effective date: 20030926 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
AS | Assignment |
Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.;REEL/FRAME:037079/0001 Effective date: 20151027 |
|
REMI | Maintenance fee reminder mailed | ||
FPAY | Fee payment |
Year of fee payment: 12 |
|
SULP | Surcharge for late payment |
Year of fee payment: 11 |
|
AS | Assignment |
Owner name: HEWLETT-PACKARD COMPANY, COLORADO Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE SEVENTH ASSIGNOR'S NAME FROM MUNOZ ALBERT J. PREVIOUSLY RECORDED ON REEL 012127 FRAME 0847. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:MILOJICIC, DEJAN S.;WYLEGALA, THOMAS;PONG, FONG;AND OTHERS;SIGNING DATES FROM 20010430 TO 20010525;REEL/FRAME:052619/0285 |
|
AS | Assignment |
Owner name: OT PATENT ESCROW, LLC, ILLINOIS Free format text: PATENT ASSIGNMENT, SECURITY INTEREST, AND LIEN AGREEMENT;ASSIGNORS:HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP;HEWLETT PACKARD ENTERPRISE COMPANY;REEL/FRAME:055269/0001 Effective date: 20210115 |
|
AS | Assignment |
Owner name: VALTRUS INNOVATIONS LIMITED, IRELAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:OT PATENT ESCROW, LLC;REEL/FRAME:056157/0492 Effective date: 20210503 |