WO2001090865A1

WO2001090865A1 - Time synchronization within a distributed processing system

Info

Publication number: WO2001090865A1
Application number: PCT/US2001/015945
Authority: WO
Inventors: Brian C. Deck; Joseph D. Kidder; Roland T. Provecher; Peter B. Everdell
Original assignee: Equipe Communications Corporation
Priority date: 2000-05-20
Filing date: 2001-05-16
Publication date: 2001-11-29
Also published as: AU2001261711A1

Abstract

The present invention provides time synchronization across a distributed processing system (10) by sending a synchronization set up command over a first bus (72) and sending a synchronization execution command over a second bus (80). A distributed system reference clock (56) is used to insure that central (14) and distributed (38) controllers each synchronize to the same clock signal. Synchronization is set up through a software controlled communication bus (32) and actual synchronization is executed through a hardware controlled communication bus (33). Use of a hardware controlled communication bus (33) provides immediate synchronization of a high resolution real time clock and permits delayed synchronization of a lower resolution real time clock.

Description

TIME SYNCHRONIZATION WITHIN A DISTRIBUTED PROCESSING

SYSTEM

This application is a continuation-in-part of U.S. Serial Number 09/574,341, filed May 20, 2000, entitled "Policy Based Provisioning of Network Device Resources" and U.S. Serial Number 09/574,343, filed May 20, 2000, and entitled "Functional Separation of Internal and External Controls in Network Devices", both of which are still pending.

Background

A distributed processing system is a collection of independent computers that appear to the user of the system as a single computer. Generally, a distributed processing system includes a main or central control processing module, card or printed circuit board (herein after module) having a central control processing subsystem ("central controller") that actively controls system operation by performing a major portion of the control functions (e.g., booting and system management) for the system, hi addition, a distributed processing system includes multiple distributed processing modules that each include a control processor subsystem ("distributed controllers"). Each control processor subsystem, including the central controller and distributed controllers, executes software to control their local module and to communicate with the other control processor subsystems. Each control processor subsystem operates in an autonomous fashion but the software presents the distributed processing system to the user as a single computer.

Although the central controller and the distributed controllers operate independently, there are many instances when it is important that they are synchronized to the same clock signal. For example, when faults, errors or other types of events occur (hereinafter collectively referred to as events), generally a record in a log is created including a time-stamp. The time-stamp helps relate one event to other events, which may help in determining a cause, a solution and/or next steps to take as a response to the event. In a distributed processing system, the time-stamps may be used to help determine if events on multiple modules are related. Unfortunately, it is difficult to synchronize control subsystems on different modules. In addition, the resolution of time-stamps is typically only one second, corresponding to the low resolution provided by publicly available real-time clock (RTC) chips. Since a great many events may occur within the same second, it may be impossible to determine the order in which logged events occurred. Typically, synchronization of a distributed processing system is initiated by sending a synchronization command across a software controlled communication bus to each controller (central and distributed). Unfortunately, one or more controllers may be busy handling other commands or interrupts and, thus, the execution of the synchronization command by each controller may vary. Such variation may lead to inaccurate or insufficient synchronization across the distributed processing system, which may cause faults and/or a system crash.

Summary

The present invention provides a method of synchronizing a distributed processing computer system including the steps of sending a synchronization set up command over a first bus to one or more distributed processor modules and sending a synchronization execution command over a second bus to the distributed processor modules. Sending the synchronization execution command may include detecting a system reference clock pulse and sending the synchronization execution command over the second bus to the distributed processor modules in response to the detection of the system reference clock pulse. The method may also include the steps of clearing high resolution real time clocks on the distributed processor modules and a central processor module in response to the synchronization execution command and sending an interrupt signal to local controller processing subsystems on the distributed processing modules and the control processor module in response to the synchronization execution command. The method may further include writing a synchronization time into low resolution real time clocks on the distributed processor modules and the central processing module in response to the interrupt. The high resolution real time clocks may include counters, and clearing high resolution real time clocks may include resetting the high resolution clocks. The low resolution real time clocks may include real time clock integrated circuits, and writing the synchronization time into low resolution real time clocks may include resetting the low resolution real time clocks. Writing the synchronization time into low resolution real time clocks includes reading a synchronization command from a control register within the hardware controller on the distributed processor modules and the central processor module and writing the synchronization time provided with the synchronization set up command into the low resolution real time clocks. A central processor may send the synchronization set up command, and the distributed processing computer system comprises a network device. In another aspect, the present invention provides a method of distributing a real time clock in a distributed processing computer system including the steps of providing a system reference clock to one or more distributed processing modules and a central processing module, broadcasting a synchronization execution command to the distributed processing modules, clearing a real time counter in each of the distributed processing modules and in the central processing module in response to the broadcast synchronization execution command and using the system reference clock to increment each of the real time counters. The distributed processing computer system may include a network device.

In yet another aspect, the present invention provides a distributed processing computer system including a central controller, one or more distributed controllers, a first communication bus coupled to the central controller, wherein the central controller is capable of transmitting synchronization set up commands over the first communication bus to the distributed controllers and a second communication bus couple to the central controller, wherein the central controller is capable of transmitting synchronization execution commands over the second communication bus to the distributed controllers. The distributed processing computer system may also include a central hardware controller coupled to the central controller for transmitting synchronization execution commands over the second communication bus. The first communication bus may be an Ethernet bus, and the second communication bus may be an Inter-IC communications bus. The distributed processing computer system may be a network device.

In a further aspect, the present invention provides a computer system including a real time clock having both a low resolution real time clock and a high resolution real time clock. The low resolution real time clock may be a real time clock integrated circuit component. The computer system may include a system reference clock, and the high resolution real time clock may be a time counter incremented by the system reference clock.

Brief Description of the Drawings Fig. 1 is a block diagram of a computer system with a distributed processing system;

Fig. 2 is a block diagram of a computer system with a hardware controlled communication bus;

Fig. 3 is a block diagram of a distributed hardware controller and a distributed controller;

Fig. 4 is a block diagram of a distributed processing computer system having central and distributed timing subsystems; and

Figs. 5 and 6 are flow diagrams depicting a method of synchronizing a distributed processing computer system.

Detailed Description The present invention provides time synchronization across a distributed processing system by sending a synchronization set up command over a first bus and sending a synchronization execution command over a second bus. A distributed system reference clock is also used to insure that central and distributed controllers each synchronize to the same clock signal. Synchronization is set up through a software controlled commumcation bus and actual synchronization is executed through a hardware controlled communication bus. Use of a hardware controlled communication bus provides immediate synchronization of a high resolution real time clock and permits delayed synchronization of a lower resolution real time clock. Consequently, within a predetermined amount of time, the central and distributed controllers are accurately synchromzed. In addition, the increased resolution of the real time clock allows for more granular time stamps and more accurate time synchronization across the distributed processing system. Distributed Processing System

Referring to Fig. 1, computer system 10 includes a central controller module 12 with a central control processor subsystem ("central controller") 14 that executes an instance of kernel software 20 including master control programs and server programs to actively control system operation by performing a major portion of the control functions (e.g., booting and system management) for the system. In addition, computer system 10 includes multiple distributed controller modules 16a-16n. Each distributed controller module includes a distributed control processor subsystem ("distributed controller") 18a- 18n, which runs an instance of kernel software 22a-22n including slave and client programs as well as module specific software applications. Each control processor subsystem 14, 18a-18n operates in an autonomous fashion but the software presents computer system 10 to the user as a single computer.

Each control processor subsystem includes a processor integrated circuit (chip) 24, 26a-26n, for example, a Motorola 8260 or an Intel Pentium processor. The control processor subsystem also includes a memory subsystem 28, 30a-30n including a combination of non- volatile or persistent (e.g., PROM and flash memory) and volatile (e.g., SRAM and DRAM) memory components. Computer system 10 also includes a software controlled internal communication bus 32 coupled to each processor 24, 26a- 26n. In one embodiment, the software controlled communication bus is a switched Fast Ethernet providing 100Mb of dedicated bandwidth to each processor allowing the distributed processors to exchange control information at high frequencies. A backup or redundant Ethernet switch may also be connected to each board such that if the primary Ethernet switch fails, the boards can fail-over to the backup Ethernet switch.

Computer system 10 may be a network device, for example, a switch, router or hybrid switch-router, in which case, computer system 10 may also include a data path 34. In this example, Ethernet 32 provides an out-of-band control path, meaning that control information passes over Ethernet 32 but the network data being switched by computer system 10 passes to and from external network connections 31a-31x over data path 34. External network control data is passed from the distributed controller modules to the central controller module over Ethernet 32. Hardware Controlled Communications Bus

Computer system 10 also includes a hardware controlled communications bus 33. Hardware controlled communications bus 33 is provided for low level system service operations, including, for example, the detection of newly installed (or removed) hardware, reset and interrupt control and real time clock (RTC) synclironization across the system. In one embodiment, this is an Inter-IC communications (I²C) bus. See http://www.semiconductors.philips.com for an I C specification which is incorporated herein by reference.

Referring to Fig. 2, hardware controlled communications bus 33 is coupled with a central hardware controller 36 on the central controller module 12 and with a distributed hardware controller 38a-38n on each of the distributed controller modules 16a-16n. The hardware controllers 36, 38a-38n are connected to the central controller 14 and distributed controllers 18a-18n, respectively as well as various other hardware. In one embodiment, the hardware controllers are field programmable gate arrays (FPGAs), available, for example, from Xilinx Corporation of San Jose, CA.

Alternatively, the hardware controllers may be discrete logic or application specific integrated circuits (ASICs). Although one hardware controlled communications bus 33 is shown, it should be understood, that in large systems, bus 33 may include multiple busses. In addition, a backup or redundant hardware communications bus may also be connected to each board such that if the primary hardware commumcations bus fails, the boards can fail-over to the backup hardware communications bus.

Referring also to Fig. 3, each distributed hardware controller 38a-38n includes two registers, a control register 40 and a status register 42. In one embodiment these registers are eight bits wide corresponding to eight bit I²C commands and allowing for up to 256 different commands and 256 different statuses.

Each distributed controller 18a-18n writes data representing the status of its module into its status register 42, and central controller 14 uses communications bus 33 to read the data from each of the status registers 42. If the distributed controller does not write status into status register 42, then a default value of, for example, 00000001, will be present and when read by the central controller may be used as an indication that the distributed controller did not boot properly. If the distributed controller does boot properly, it may enter a value of, for example, 00000002, in the status register and when read by the central controller may be used as an indication that the distributed controller booted properly but has not yet passed its diagnostic tests. If the distributed controller does pass its diagnostic tests, it may enter a value of, for example, 00000003, in the status register and. when read by the central controller will indicate that the distributed controller is operating properly. Various other statuses may be written into status register 42 to notify the central controller of the distributed module's status.

Central controller 14 (Figs. 1 and 2) uses communications bus 33 to write data representing encoded commands into control register 40. Control hardware 44 within distributed hardware controllers 38a-38n detects newly written commands, decodes the commands and takes actions in accordance with the decoded commands. For example, a reset command may be decoded and the distributed hardware controller 38a-38n may cause a reset line 46 to send a reset signal to multiple hardware components on the distributed controller module. As another example, a synchronization command may be decoded and acted on as described below.

Distributed High Resolution Real Time Clock

A high resolution clock signal may be used within computer system 10 to improve synchronization across all modules and to provide for finer granularity time- stamps on logged events. Referring to Fig. 4, the central controller module 12 includes a real time clock

(RTC) chip 49 and each distributed controller module 16a-16n includes an RTC chip 50a-50n. RTC chips 49 and 50a-50n are connected to the central controller 14 and distributed controllers 18a-18n, respectively, and the central and distributed controllers may read a time from their local RTC chip, including the year, month, day, hour, minute and second. Typically, the lowest resolution provided by publicly available RTC chips is seconds. In addition, the distributed controllers may also write a time into the RTC chip to reset or synchronize the RTC chip with other RTC chips within the system.

The central and distributed controllers may use the time read from their local RTC chip as a time-stamp for logged events. Many events, however, may happen in the same second and when logged each event may be assigned the same time-stamp. Having the same time-stamp for multiple events increases the difficultly of later determining the correct sequence of the logged events. Similarly, if the RTC chips used by the central and distributed controllers are not synchronized, then time stamps alone may not accurately reflect the sequence of events on multiple modules.

To provide a higher resolution time stamp, the central hardware controller includes a time counter 51 and the distributed hardware controllers include time counters 52a-52n. In one embodiment, the hardware controllers (central and distributed) increment the time counters on microsecond intervals. When an event occurs, the controller (e.g., central controller 14 or distributed controllers 18a-18n) that detects the event and/or the controller responsible for logging the event reads the time from both the local RTC chip 49 or 50a-50n and the time counter 51 or 52a-52n. Consequently, the time-stamp resolution is increased from seconds to microseconds providing significantly better granularity.

In one embodiment, the time counter is 32 bits allowing the hardware controllers to count microseconds, seconds, minutes and hours, up to one hundred and forty-nine hours. The controller (e.g., central controller 14 and distributed controllers 18a-18n) may use the seconds, minutes and hours from either the time counter or the RTC chip. The added hardware required to provide a real time clock with higher resolution consumes valuable system resources, however, higher resolution provides significant advantages such as more accurate synchronization and finer granular time-stamps. To allow the hardware controllers to increment the time counters every 125 microseconds, an 8KHz clock signal 53 and 54a-54n is delivered to each hardware controller. The hardware controllers may then increment the time counters upon the detection of each clock pulse. The 8KHz clock signal may be provided in various ways. For example, an oscillator chip may be present on each board and used to provide the 8KHz clock signal to each chip either directly from the oscillator chip if it is 8KHz or through other discrete logic that may multiply or divide the signal from the oscillator chip to provide 8KHz. To aid in synchronization across the distributed processing system, preferably system reference clock signals are sent instead from a central timing subsystem 56 on the central controller module to a distributed timing subsystem 57 and 58a-58n on each module. The central timing subsystem may send 8KHz signals directly to each of the distributed timing subsystems or, preferably, combination clock signals 59 and 60a-60n are sent from central timing subsystem 56 to each distributed timing subsystems 57 and 58a-58n. Often, many different clock signals are required by each module, and hardware within the distributed timing subsystems derive the necessary clock signals from the combination clock signal. In one embodiment, central timing subsystem 56 provides a hard-wired clock signal 59, 60a-60n to each distributed timing subsystem 58a-58n which is a combination of a 19.44MHz clock signal and an 8KHz clock signal. Clock signals 59 and 60a-60n are separate hard-wired signals due to load restrictions and the importance of clean, strong clock signals. Alternatively, one clock signal may be shared by two or more of the distributed timing subsystems. The distributed timing subsystem then extracts the 8KHz clock signal from the combination clock signal and provides the 8KHz clock signal 53, 54a- 54n to central hardware controller 36 and distributed hardware controllers 38a-38b, respectively. The hardware controllers use the 8KHz clock signals to synchronously increment their time counters.

In one embodiment, the distributed processing system is a network device connected to external optical fibers carrying signals following the synchronous optical network (SONET) protocol, and the 8KHz clock signal also provides timing for SONET framing. Multiples of 19.44MHz provide base frequencies for various SONET data transfer rates. For example, four times 19.44MHz is 77.76MHz which is the base frequency for an OC1 SONET stream, two times 77.76MHz provides the base frequency for an OC3 SONET stream and eight times 77.76MHz provides the base frequency for an OC12 SONET stream.

Synchronization Across a Distributed Processing System

Referring to Fig. 5, to ensure that central controller 14 and distributed controllers 18a-18b operate synchronously, periodically — for example, once a day — central controller 14 initiates (block 70) a synchronization cycle. The synchronization cycle includes synchronization set up and synchronization execution. For synchronization set up the central controller sends (block 72) a synchronization command across software controlled commumcation bus 32 (e.g., Ethernet bus) to each distributed controller. The synchronization command tells each distributed controller to prepare for a synchronization execution and provides a synchronization time to be used at the next synchronization execution. For synchronization execution, central controller 14 sets up (block 74) central hardware controller 36 to perform a broadcast write over hardware controlled communications bus 33 (e.g., I²C bus, Fig. 4) to write a synchronization command into control register 40 (Fig. 3) within each hardware controller 38a-38n. Referring again to Fig. 4, after the next positive 8KHz clock pulse is received on system reference clock line 53, central hardware controller 36 then broadcast writes a synchronization command over the hardware controlled communication bus to the control register within each distributed hardware controller 38a-38n. Both the central and distributed hardware controllers then clear their time counters 51 and 52a-52n prior to the next 8KHz clock pulse and send an interrupt over lines 75 and 77a-77n to their local controller (e.g., central controller 12 or distributed controllers 18a-18n). The 8KHz system reference clock on line 53 may be treated by central hardware controller 36 as an interrupt and masked until central controller 14 sets up central hardware controller 36 for the broadcast write. Referring to Fig. 6, when a distributed controller receives (block 76) a synchronization set up command and synchronization time over software controlled communication bus 32, the distributed controller prepares (block 78) to write the synchronization time into its local RTC chip 50a-50n. Similarly, after sending the synchronization set up command, the central controller also prepares (block 78) to write the synchronization time into its local RTC chip 49. When the distributed hardware controllers receive (block 80) a broadcast synchronization command over hardware controlled bus 33, they clear (block 82) their time counter 52a-52n and send (block 84) an interrupt to their local distributed controller 18a-18n. Similarly, when the central hardware controller sends the broadcast synchronization command over the hardware communication bus, the central hardware controller clears (block 82) its time counter 51 and sends (block 84) an interrupt to central controller 14.

When central controller 14 and distributed controllers 18a-18n receive an interrupt from their local hardware controllers 36, 38a-38n, respectively, they read the control register within their local hardware controller. The synchronization command read from the control register causes the central and distributed controllers to write the synchronization time previously received during the synchronization set up command sent over the software controlled communication bus 32 into their local RTC chip 49 and 50a-50n. Once the synchronization time is written into the RTC chips, then all modules within the computer system are synchronized.

The central and distributed controllers may be busy handling other commands and interrupts when they receive the interrupt from the hardware controller, thus, each may be delayed a different amount of time with respect to the others with regard to synchronizing their local RTC chips. These delays are not long, however, and in any event, the high resolution real time clock stored in the time counters are synchronized immediately upon receipt of the broadcast synchronization command. Broadcasting the synchronization command on the rising edge of the 8KHz system reference clock insures that each of the hardware controllers will have enough time to clear their time counter before the next 8KHz clock pulse, thus, synchronizing each time counter to the same 8KHz clock pulse. Since the time counters store the high resolution real time clocks (e.g., microseconds, seconds, minutes and hours), each of the modules (central and distributed) are immediately synchronized at the highest resolution. As mentioned previously, the RTC chips generally hold the year, month, day, hour, and second. The central and distributed controllers may use the year, month and day from the RTC chip and the hour, minute, second, millisecond and microsecond from the time counter. In this case, the controllers would have a full day or 24 hours to synchronize their RTC chips. Instead, the central and distributed controllers may use the year, month, day and hour from the RTC chip and the minute, second, millisecond and microsecond from the time counter. In this case, the controllers would have 1 hour or 60 minutes to synchronize their RTC chips. Similarly, the central and distributed controllers may use the year, month, day, hour and minute from the RTC chip and the second, millisecond and microsecond from the time counter. In this case, the controllers would have 1 minute or 60 seconds to synchronize their RTC chips. Instead, the central and distributed controllers may use the year, month, day, hour, minute and second from the RTC chip and the millisecond and microsecond from the time counter. In this case, the controllers would have 1 second to synchronize their RTC chips. In most systems, one second is a sufficient amount of time within which to have each controller synchronize their RTC chip. Clearly, more or less than 32 bits may be provided in the time counters, and RTC chips with lower resolution may also be used. As a result, the real time clock - a combination of the value read from the RTC chip and the time counter within the hardware controller ~ may be simultaneously used by hardware as a run-time clock and used by software to time stamp events. When an event occurs, the central or distributed controller that detected the event or the controller responsible for logging the event reads the time from both the RTC chip and the time counter - in less than 125 microseconds— and stores the time as the time stamp for the logged event. With a high resolution time stamp of 125 microseconds, the order in which events occurred will be clear. In addition, the high resolution time synchronization across the distributed processing computer system insures that the time stamps logged with each event are accurate across the entire system.

Alternatively, instead of using the combination of the RTC chip and time counter to provide a real time clock, the time counter may be used in combination with software managed extensions. In other words, the year, month, day, hour and second may be stored and updated by software based on the time counter. When the time counter wraps, software increments the higher order values. The RTC chip is used at powerup and/or on synchronization to initialize the higher order values stored in software.

Instead of sending the synchronization set up command across software controlled communication bus 32 (e.g., Ethernet), it may be sent across hardware controlled communication bus 33. However, the synchronization set up command and synchronization time may take more time and effort to establish through the hardware controlled communication bus (e.g., I²C bus) since the I²C bus is a two serial line bus and commands are only eight bits.

It will be understood that variations and modifications of the above described methods and apparatuses will be apparent to those of ordinary skill in the art and may be made without departing from the inventive concepts described herein. Accordingly, the embodiments described herein are to be viewed merely as illustrative, and not limiting, and the inventions are to be limited solely by the scope and spirit of the appended claims.

Claims

We claim:

1. A method of synchronizing a distributed processing computer system, comprising: sending a synchronization set up command over a first bus to a plurality of distributed processor modules; and sending a synchronization execution command over a second bus to the plurality of distributed processor modules.

2. The method of claim 1, wherein sending the synchronization execution command includes: detecting a system reference clock pulse; and sending the synchronization execution command over the second bus to the plurality of distributed processor modules in response to the detection of the system reference clock pulse.

3. The method of claim 1, further comprising: clearing high resolution real time clocks on the plurality of distributed processor modules in response to the synchronization execution command; clearing a high resolution real time clock on a central processor module in response to the synchronization execution command; and sending an interrupt signal to local controller processing subsystems on the distributed processing modules and the control processor module in response to the synchronization execution command.

4. The method of claim 3, wherein the high resolution real time clocks comprise counters.

5. The method of claim 3, wherein the clearing high resolution real time clocks comprises: resetting the high resolution real time clocks.

6. The method of claim 3, further comprising: writing a synchronization time into low resolution real time clocks on the distributed processor modules and the central processing module in response to the interrupt.

7. The method of claim 6, wherein the low resolution real time clocks comprise real time clock integrated circuits.

8. The method of claim 6, wherein writing the synchronization time into low resolution real time clocks comprises: resetting the low resolution real time clocks.

9. The method of claim 6, wherein writing the synchronization time into low resolution real time clocks includes: reading a synchronization command from a control register within the hardware controller on the distributed processor modules; and writing the synchronization time provided with the synchronization set up command into the low resolution real time clocks.

10. The method of claim 6, wherein writing the synchronization time into low resolution real time clocks includes: reading a synchronization command from a control register within the hardware controller on the central processor module; and writing the synchronization time provided with the synchronization set up command into the low resolution real time clocks.

11. The method of claim 1 , wherein a central processor sends the synchronization set up command.

12. The method of claim 1, wherein the distributed processing computer system comprises a network device.

13. A method of distributing a real time clock in a distributed processing computer system, comprising: providing a system reference clock to a plurality of distributed processing modules and a central processing module; broadcasting a synchronization execution command to the plurality of distributed processing modules; clearing a real time counter in each of the plurality of distributed processing modules and in the central processing module in response to the broadcast synchronization execution command; and using the system reference clock to increment each of the real time counters.

14. The method of claim 13, wherein the distributed processing computer system comprises a network device.

15. A distributed processing computer system comprising: a central controller; a plurality of distributed controllers; a first communication bus coupled to the central controller, wherein the central controller is capable of transmitting synchronization set up commands over the first communication bus to the plurality of distributed controllers; and a second communication bus couple to the central confroller, wherein the central controller is capable of transmitting synchronization execution commands over the second communication bus to the plurality of distributed controllers.

16. The distributed processing computer system of claim 15, further comprising: a central hardware controller coupled to the central controller for transmitting synchronization execution commands over the second communication bus.

17. The distributed processing computer system of claim 16, wherein the first commumcation bus comprises an Ethernet bus.

18. The distributed processing computer system of claim 16, wherein the second communication bus comprise an Inter-IC communications bus. The distributed processing computer system of claim 16, wherein the distributed processing computer system comprises a network device.

19. A computer system comprising: a real time clock including: a low resolution real time clock; and a high resolution real time clock.

20. The computer system of claim 20, wherein the low resolution real time clock comprises a real time clock integrated circuit component

21. The computer system of claim 20, further comprising: a system reference clock, and wherein the high resolution real time clock comprises a time counter incremented by the system reference clock.