US20070266060A1 - Remote online information back-up system - Google Patents
Remote online information back-up system Download PDFInfo
- Publication number
- US20070266060A1 US20070266060A1 US11/620,461 US62046107A US2007266060A1 US 20070266060 A1 US20070266060 A1 US 20070266060A1 US 62046107 A US62046107 A US 62046107A US 2007266060 A1 US2007266060 A1 US 2007266060A1
- Authority
- US
- United States
- Prior art keywords
- disk
- driver
- device driver
- computing system
- storage medium
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000004891 communication Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000000034 method Methods 0.000 description 19
- 238000001152 differential interference contrast microscopy Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 238000013459 approach Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000010367 cloning Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 229920000638 styrene acrylonitrile Polymers 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2071—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring using a plurality of controllers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/953—Organization of data
- Y10S707/959—Network
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99951—File or database maintenance
- Y10S707/99952—Coherency, e.g. same view to multiple users
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99951—File or database maintenance
- Y10S707/99952—Coherency, e.g. same view to multiple users
- Y10S707/99955—Archiving or backup
Definitions
- the invention relates to the field of data back-up systems, and in particular to a real-time online device drivers for a data back-up system.
- a company's information assets (data) are critical to the operations of the company. Continuous availability of the data is a necessary. Therefore, backup systems are required to ensure continuous availability of the data in the event of system failure in the primary storage system. The cost in personnel and equipment of recreating lost data can run into hundreds of thousands dollars.
- Local hardware replication techniques e.g., mirrored disks
- mirrored disks To ensure continuous operation even in the presence of catastrophic failures, a backup copy of the primary data is maintained up-to-date at an off-site location.
- data may be lost (i.e., the data updated since the last backup operation).
- a problem with conventional remote backup techniques is that they occur at the application program level.
- real-time online remote backup is relatively expensive and inefficient.
- a storage area network is a dedicated storage network in which systems and intelligent subsystems (e.g., primary and secondary) communicate with each other to control and manage the movement and storage of data from a central point.
- the foundation of a SAN is the hardware on which it is built. The high cost of hardware/software installation and maintenance makes SANs prohibitively expensive for all but the largest businesses.
- a private backup network is a network designed exclusively for backup traffic. Data management software is required to operate this network. It consequently increases system resource contention at the application level. The backup is not real-time, thus exposing the business to a risk of data loss.
- This configuration eliminates all backup traffic from the public network at the cost of installing and maintaining a separate network. Use of PBNs in business is limited due to the high cost.
- a third known backup technique is database (DB) built-in backup.
- DB database
- export/import utilities and offline backup routines are disruptive, since they lock database and associated structures, making the data inaccessible to all users.
- processing must cease in order to create the backup, this method of course does not provide real-time capabilities.
- remote backup strategies which add additional overhead to DB performance. While not achieving real-time capabilities the installation of any of these backup scheme is a time consuming and difficult task for the database administrator.
- a remote online information back-up comprises a local computing system and a remote computing system.
- the local computing system includes a local disk and a local device driver.
- the local device driver is responsive to requests from a local application executing on the local computing system, for selectively processing calls to the local disk or to a remote disk for backup of data resident on the local disk.
- the remote computing system includes the remote disk and a remote device driver.
- the remote device driver is responsive to calls from either the local device driver or calls from a remote application executing on the remote computing system, wherein calls from the local device driver are processed to perform backup operations to the remote disk of data resident on the local computing system.
- the backup technique of the present invention is implemented at the device driver level, and it achieves the backup by “cloning” every change of the local system to a network connected remote system.
- FIG. 1 is a block diagram illustration of a remote online information backup system
- FIG. 2 is a block diagram illustration of communication between the local system and the remote system of the information backup system of FIG. 1 ;
- FIG. 3 is a plot of a single-write performance comparison
- FIG. 4 is a plot of a burst-write performance comparison
- FIG. 5 is a plot of a burst-write performance comparison.
- FIG. 1 is a block diagram illustration of a remote online information backup system 10 .
- the system 10 includes a local system 12 and a remote backup system 14 , which may be connected via a communications medium 16 such as for example, a LAN, WAN, Internet, etc.
- the local system 12 includes disk memory 18 (e.g., a plurality of disks arranged in a RAID architecture) that interface with a disk driver 20 .
- the disk driver 20 interfaces with a disk-caching-disk (DCD) device driver 22 , which communicates with a realtime online remote information backup (ROBIB) disk driver 22 .
- DCD disk-caching-disk
- ROBIB realtime online remote information backup
- the ROBIB device driver 24 is transparent to a user 26 (i.e., a file system and application programs) such as a Database Management System (DBMS).
- DBMS Database Management System
- the ROBIB device driver 24 requires no changes to the existing operating system or physical data layout. As a result, it can be used as a “drop-in” filter between the traditional disk device driver and the file system.
- the RORIB device driver 24 acts as bridge between the file system and the lower level device driver, such as a NIC driver 28 and the disk driver 26 . Between the RORIB driver and raw disk driver 20 is the DCD driver 22 , which increases small write performance. Such a multi-layer device driver approach reduces implementation overhead and increases portability.
- the remote system 14 includes a NIC driver 50 that receives requests and data sent from the local system 12 via the network 16 .
- the NIC driver passes the requests and data to a RORIB server device driver 52 that writes the data to back up system disk 54 via a remote system.
- DCD device driver 56 and disk driver 58 .
- the RORIB server device driver 52 is also responsible to requests from a server application 60 .
- the RORIB client device driver 24 To back-up data to be written to the local disk 18 , the RORIB client device driver 24 also writes the local data to the local NIC 28 , which outputs the data to the network 16 . The RORIB server device driver 52 then receives the local data communicated over the network 16 and writes the data to the backup disk 54 .
- the RORIB driver 24 receives various commands from the file system 26 for access to the local disk 18 . Accesses to the local disk 18 require a determination of whether the local disk 18 is active or inactive to perform the commands requested by the file system 26 . Therefore, the RORIB driver 24 determines whether the local disk 18 is active or inactive. The RORIB driver 24 sends commands to the DCD driver 24 to request the status of the local disk. The DCD driver 24 then notifies the local disk driver 20 to check whether the local disk 18 is active. The local disk driver 20 notifies the DCD driver 22 of its determination and the DCD driver 22 provides that determination to the RORIB driver 24 .
- the RORIB driver 24 issues a call to the NIC driver 28 to setup a connection with the remote backup system 14 in order to access the backup data.
- the NIC driver 28 provides an interface between the communication network 16 and a computing system, including the remote backup system 28 . Also, the NIC driver provides the necessary protocol, such as TCP/IP, for sending and receiving messages from the network 16 .
- the remote backup system 14 proceeds to provide a backup technique that is transparent to the file system 26 , applications, and user of the local system 12 .
- the remote NIC driver 50 receives and sends messages over the network 16 .
- the remote NIC driver 50 receives a call from local NIC driver 28 , the local disk 18 not being active, the remote NIC driver 50 processes the message and formats the message in a format appropriate for the RORIB 52 to process.
- the RORIB 52 sends a call the DCD driver 56 which issues a call to the disk driver 58 when needed to process command calls from the RORIB 52 .
- the file system 60 may also issue calls to the RORIB 52 to process commands internal to the remote backup system 14 .
- FIG. 2 is a block diagram illustration of communication between the local system and the remote system 14 of the information backup system of FIG. 1 .
- the driver 20 passes a file handle by a connected TCP socket (not shown) to a kernel driver 30 . Because of this the kernel driver 30 does not have to expend resources to connect.
- the protocol is as follows: if the driver is asked to read from a block device, it sends a “request” packet. When the operation is complete, the server 16 responds with a “reply” packet. In other words, a client requests a local resource and thus generates a response from a daemon 24 (a memory-resident program) on the server via the driver.
- the client/server handshake begins when the server negotiates a connection with the client. If initialization is successful, connection occurs before timeout. The connection will be maintained, checked, renegotiated and recovered automatically by the server side daemon. Recovery is transparent to the user as long as there is currently one working connection.
- the daemon is responsible for monitoring and maintaining all ports and sub-processes, which occurs transparent to the users.
- a complementary daemon also operates on the client. This daemon handles kernel requests made for data on local disk. The data are then passed across the network to the requesting server.
- the RORIB device driver combines the functions of Internet protocol (IP) and storage drivers so that selected storage systems connected through the network (e.g., the Internet) appear to be a uniform storage system to a user.
- IP Internet protocol
- storage drivers so that selected storage systems connected through the network (e.g., the Internet) appear to be a uniform storage system to a user.
- the process can be implemented either using software at device driver level, or using hardware at a controller level.
- a test demo has been implemented using software at device driver level under Linux OS.
- the backup technique of the present invention is implemented at the driver level, and as a result requires no change to the operating system (OS) and software applications. Thus reduces the overhead, it makes use of applicant's previous invention referred to as DCD (disk caching disk) to make realtime online back a reality.
- DCD disk caching disk
- a multi-layered device driver approach was used for the device driver to communicate with the disk hardware directly.
- Implementation was achieved by adding the RORIB driver on top of the traditional device drivers (device drivers are specific to the hardware).
- the RORIB driver calls the lower level disk driver and the NIC driver, through the standard device driver interface, to perform actual I/O operations.
- This approach has three major advantages over current structures. First, it greatly simplifies the implementation efforts since the RORIB driver avoids the complex task of direct communication with the hardware. Second, the same RORIB driver works with all kinds of disks and NICs in the system because all the low-level disk device drivers use a standard interface. Third, it is easy to transfer the current implementation to other Linux systems.
- the data transmitted over the network may be encrypted using SSL (secure socket layer) to provide data security.
- SSL secure socket layer
- Linux is used as the operating system. It is possible to write and incorporate specific device drivers.
- the device drivers for a class of block devices provide class specific interfaces to that class of device.
- Linux supports several different block device drivers.
- One such device is a network block device (NBD) driver that makes a remote resource look like a local device via TCP layer.
- NBD exists as kernel module, which means it can be loaded or unloaded at any time.
- a prototype RORIB device driver was designed and tested individually (loop back) before integrating it into a network.
- PostgreSQL was used as the DBMS.
- the system configuration was two PCs connected over network.
- Java Servtet runs on the web browser to measure performance. Two sets of experiments were performed. The first experiment compared single-time read and write performance of the proposed backup strategy with the previous discussed backup methods. The second compared burst read and write performance with different time intervals to simulate a real world environment with multi-client connections, again compared to current strategies.
- the results of the first experiment were as follows.
- the read speed of the proposed backup strategy is identical to existing procedures on a single database. This is expected, as backup does not occur until the database is updated.
- FIG. 3 shows the write performance comparison for a single write.
- a plot along line 32 (Strategy 1 ) corresponds to a write once to a single database (no backup).
- Strategy 2 along line 34 was to do internet-based real-time online backup using the prototype device driver.
- Strategy 3 along line 36 is to write to two data files on different local disks of the same database on a workstation. This approach simulates an application level local backup solution.
- Strategy 4 along line 38 was to write to databases on two workstations over the Internet. It simulates application level remote backup solution.
- the second experiment is a burst-write performance under different workloads and different time intervals, as shown in FIG. 4 .
- the burst read performance is similar to single read performance.
- Each point plotted in FIG. 4 represents the response time for a 20-database-insertion (20 threads), simulating twenty (20) clients utilizing the database server at different time intervals.
- there are fluctuations in response time which is caused by contention.
- Strategy 1 along line 40 (no backup) and strategy 2 along line 42 (RORIB) still have similar response time for database queries, because strategy 2 along line 42 avoids application level overhead when doing Real-time Online Remote Info Backup.
- Strategy 3 along line 44 exhibits greater response time than strategy 4 along line 46 (application level remote backup) under heavier workload, as shown in FIG. 5 .
- local machine resource contention accounts for more time degradation than network traffic contention.
- network traffic contention becomes more distinguishable than local resource contention, as shown in FIG. 5 .
- the backup technique of the present invention is implemented at device driver level. Thus, it is transparent to the user, the file system, and the application programs such as DBMS. It requires no changes to the existing operating system or the physical file structure. As a result, it can be used as a ‘drop-in’ filter between the traditional disk device driver and the file system in an existing system to obtain immediate functionality.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
- Debugging And Monitoring (AREA)
- Alarm Systems (AREA)
Abstract
A remote online information back-up comprises a local computing system and a remote computing system. The local computing system includes a local disk and a local device driver. The local device driver is responsive to requests from a local application executing on the local computing system, for selectively processing calls to the local disk or to a remote disk for backup of data resident on the local disk. The remote computing system includes the remote disk and a remote device driver. The remote device driver is responsive to calls from either the local device driver or calls from a remote application executing on the remote computing system, wherein calls from the local device driver are processed to perform backup operations to the remote disk of data resident on the local computing system.
Description
- This application claims priority from provisional application Ser. No. 60/276,286 filed Mar. 15, 2001; and from provisional application Ser. No. 60/287,946 filed May 1, 2001; and from provisional application Ser. No. 312,471 filed Aug. 15, 2001. Each of these applications are hereby incorporated by reference.
- The invention relates to the field of data back-up systems, and in particular to a real-time online device drivers for a data back-up system.
- A company's information assets (data) are critical to the operations of the company. Continuous availability of the data is a necessary. Therefore, backup systems are required to ensure continuous availability of the data in the event of system failure in the primary storage system. The cost in personnel and equipment of recreating lost data can run into hundreds of thousands dollars.
- Local hardware replication techniques (e.g., mirrored disks) increase the fault tolerance of a system by keeping a backup copy readily available. To ensure continuous operation even in the presence of catastrophic failures, a backup copy of the primary data is maintained up-to-date at an off-site location. When backup occurs at periodic intervals rather than in real-time, data may be lost (i.e., the data updated since the last backup operation). A problem with conventional remote backup techniques is that they occur at the application program level. In addition, real-time online remote backup is relatively expensive and inefficient.
- A storage area network (SAN) is a dedicated storage network in which systems and intelligent subsystems (e.g., primary and secondary) communicate with each other to control and manage the movement and storage of data from a central point. The foundation of a SAN is the hardware on which it is built. The high cost of hardware/software installation and maintenance makes SANs prohibitively expensive for all but the largest businesses.
- A private backup network (PBN) is a network designed exclusively for backup traffic. Data management software is required to operate this network. It consequently increases system resource contention at the application level. The backup is not real-time, thus exposing the business to a risk of data loss. This configuration eliminates all backup traffic from the public network at the cost of installing and maintaining a separate network. Use of PBNs in business is limited due to the high cost.
- A third known backup technique is database (DB) built-in backup. The increasing business reliance on databases has created greater demand and interest in backup procedure. Most commercial databases have built-in backup functionality. However, export/import utilities and offline backup routines are disruptive, since they lock database and associated structures, making the data inaccessible to all users. Because processing must cease in order to create the backup, this method of course does not provide real-time capabilities. The same is true for remote backup strategies, which add additional overhead to DB performance. While not achieving real-time capabilities the installation of any of these backup scheme is a time consuming and difficult task for the database administrator. (DBA)
- Therefore, there is a need for a remote online information back-up system.
- Briefly, according to an aspect of the invention, a remote online information back-up comprises a local computing system and a remote computing system. The local computing system includes a local disk and a local device driver. The local device driver is responsive to requests from a local application executing on the local computing system, for selectively processing calls to the local disk or to a remote disk for backup of data resident on the local disk. The remote computing system includes the remote disk and a remote device driver. The remote device driver is responsive to calls from either the local device driver or calls from a remote application executing on the remote computing system, wherein calls from the local device driver are processed to perform backup operations to the remote disk of data resident on the local computing system.
- The backup technique of the present invention is implemented at the device driver level, and it achieves the backup by “cloning” every change of the local system to a network connected remote system.
- The advantages of this system compared to the current prior art methods are it is easy to install, the backup is done at device driver level, it does not require any system changes, and it is easy to implement and maintain. Because the backup is done at the device driver level, it is completely transparent to both the operating system and application programs on the local system, as a result, after installation no further user intervention is needed, it is economical and efficient, it can do the entire system-wide real-time, online, remote backup for the minimal cost of a device driver.
- These and other objects, features and advantages of the present invention will become apparent in light of the following detailed description of preferred embodiments thereof, as illustrated in the accompanying drawings.
-
FIG. 1 is a block diagram illustration of a remote online information backup system; -
FIG. 2 is a block diagram illustration of communication between the local system and the remote system of the information backup system ofFIG. 1 ; -
FIG. 3 is a plot of a single-write performance comparison; -
FIG. 4 is a plot of a burst-write performance comparison; and -
FIG. 5 is a plot of a burst-write performance comparison. -
FIG. 1 is a block diagram illustration of a remote onlineinformation backup system 10. Thesystem 10 includes alocal system 12 and aremote backup system 14, which may be connected via acommunications medium 16 such as for example, a LAN, WAN, Internet, etc. Thelocal system 12 includes disk memory 18 (e.g., a plurality of disks arranged in a RAID architecture) that interface with adisk driver 20. Thedisk driver 20 interfaces with a disk-caching-disk (DCD)device driver 22, which communicates with a realtime online remote information backup (ROBIB)disk driver 22. The DCD architecture is disclosed in U.S. Pat. No. 5,754,888 entitled “System for Destaging Data During Idle Time By Transferring to Destage Buffer, Marking Segment Blank, Reordering Data in a Buffer, and Transferring to Beginning of Segment” and U.S. Pat. No. 6,243,795 entitled “Redundant, Asymmetrically Parallel Disk Cache for a Data Storage System”, both hereby incorporated by reference. - The ROBIB
device driver 24 is transparent to a user 26 (i.e., a file system and application programs) such as a Database Management System (DBMS). The ROBIBdevice driver 24 requires no changes to the existing operating system or physical data layout. As a result, it can be used as a “drop-in” filter between the traditional disk device driver and the file system. - The RORIB
device driver 24 acts as bridge between the file system and the lower level device driver, such as aNIC driver 28 and thedisk driver 26. Between the RORIB driver andraw disk driver 20 is theDCD driver 22, which increases small write performance. Such a multi-layer device driver approach reduces implementation overhead and increases portability. - The
remote system 14 includes aNIC driver 50 that receives requests and data sent from thelocal system 12 via thenetwork 16. The NIC driver passes the requests and data to a RORIBserver device driver 52 that writes the data to back upsystem disk 54 via a remote system.DCD device driver 56 anddisk driver 58. The RORIBserver device driver 52 is also responsible to requests from aserver application 60. - To back-up data to be written to the
local disk 18, the RORIBclient device driver 24 also writes the local data to thelocal NIC 28, which outputs the data to thenetwork 16. The RORIBserver device driver 52 then receives the local data communicated over thenetwork 16 and writes the data to thebackup disk 54. - The
RORIB driver 24 receives various commands from thefile system 26 for access to thelocal disk 18. Accesses to thelocal disk 18 require a determination of whether thelocal disk 18 is active or inactive to perform the commands requested by thefile system 26. Therefore, theRORIB driver 24 determines whether thelocal disk 18 is active or inactive. TheRORIB driver 24 sends commands to theDCD driver 24 to request the status of the local disk. TheDCD driver 24 then notifies thelocal disk driver 20 to check whether thelocal disk 18 is active. Thelocal disk driver 20 notifies theDCD driver 22 of its determination and theDCD driver 22 provides that determination to theRORIB driver 24. If it is determined that thelocal disk 18 is inactive, then theRORIB driver 24 issues a call to theNIC driver 28 to setup a connection with theremote backup system 14 in order to access the backup data. TheNIC driver 28 provides an interface between thecommunication network 16 and a computing system, including theremote backup system 28. Also, the NIC driver provides the necessary protocol, such as TCP/IP, for sending and receiving messages from thenetwork 16. - Once a connection is established, the
remote backup system 14 proceeds to provide a backup technique that is transparent to thefile system 26, applications, and user of thelocal system 12. Theremote NIC driver 50 receives and sends messages over thenetwork 16. When theremote NIC driver 50 receives a call fromlocal NIC driver 28, thelocal disk 18 not being active, theremote NIC driver 50 processes the message and formats the message in a format appropriate for theRORIB 52 to process. TheRORIB 52 sends a call theDCD driver 56 which issues a call to thedisk driver 58 when needed to process command calls from theRORIB 52. Also, thefile system 60 may also issue calls to theRORIB 52 to process commands internal to theremote backup system 14. -
FIG. 2 is a block diagram illustration of communication between the local system and theremote system 14 of the information backup system ofFIG. 1 . Thedriver 20 passes a file handle by a connected TCP socket (not shown) to akernel driver 30. Because of this thekernel driver 30 does not have to expend resources to connect. The protocol is as follows: if the driver is asked to read from a block device, it sends a “request” packet. When the operation is complete, theserver 16 responds with a “reply” packet. In other words, a client requests a local resource and thus generates a response from a daemon 24 (a memory-resident program) on the server via the driver. After setup, the client/server handshake begins when the server negotiates a connection with the client. If initialization is successful, connection occurs before timeout. The connection will be maintained, checked, renegotiated and recovered automatically by the server side daemon. Recovery is transparent to the user as long as there is currently one working connection. - The daemon is responsible for monitoring and maintaining all ports and sub-processes, which occurs transparent to the users. A complementary daemon also operates on the client. This daemon handles kernel requests made for data on local disk. The data are then passed across the network to the requesting server.
- The RORIB device driver combines the functions of Internet protocol (IP) and storage drivers so that selected storage systems connected through the network (e.g., the Internet) appear to be a uniform storage system to a user. As a result, not only can the
RORIB 16 provide realtime data backup, but also distributed data services to a company or an organization with decentralized offices. - The process can be implemented either using software at device driver level, or using hardware at a controller level. A test demo has been implemented using software at device driver level under Linux OS.
- The backup technique of the present invention is implemented at the driver level, and as a result requires no change to the operating system (OS) and software applications. Thus reduces the overhead, it makes use of applicant's previous invention referred to as DCD (disk caching disk) to make realtime online back a reality.
- For the device driver to communicate with the disk hardware directly, a multi-layered device driver approach was used. Implementation was achieved by adding the RORIB driver on top of the traditional device drivers (device drivers are specific to the hardware). The RORIB driver calls the lower level disk driver and the NIC driver, through the standard device driver interface, to perform actual I/O operations. This approach has three major advantages over current structures. First, it greatly simplifies the implementation efforts since the RORIB driver avoids the complex task of direct communication with the hardware. Second, the same RORIB driver works with all kinds of disks and NICs in the system because all the low-level disk device drivers use a standard interface. Third, it is easy to transfer the current implementation to other Linux systems.
- The data transmitted over the network (e.g., the Internet) between the local and remote systems may be encrypted using SSL (secure socket layer) to provide data security.
- Linux is used as the operating system. It is possible to write and incorporate specific device drivers. The device drivers for a class of block devices provide class specific interfaces to that class of device.
- Linux supports several different block device drivers. One such device is a network block device (NBD) driver that makes a remote resource look like a local device via TCP layer. NBD exists as kernel module, which means it can be loaded or unloaded at any time.
- A prototype RORIB device driver was designed and tested individually (loop back) before integrating it into a network. PostgreSQL was used as the DBMS. The system configuration was two PCs connected over network. Java Servtet runs on the web browser to measure performance. Two sets of experiments were performed. The first experiment compared single-time read and write performance of the proposed backup strategy with the previous discussed backup methods. The second compared burst read and write performance with different time intervals to simulate a real world environment with multi-client connections, again compared to current strategies.
- The results of the first experiment were as follows. The read speed of the proposed backup strategy is identical to existing procedures on a single database. This is expected, as backup does not occur until the database is updated.
- Current backup procedures are wholly a software process of the DBMS. Backup is done at the application level.
FIG. 3 shows the write performance comparison for a single write. A plot along line 32 (Strategy1) corresponds to a write once to a single database (no backup). Strategy2 alongline 34 was to do internet-based real-time online backup using the prototype device driver. Strategy3 alongline 36 is to write to two data files on different local disks of the same database on a workstation. This approach simulates an application level local backup solution. Strategy4 alongline 38 was to write to databases on two workstations over the Internet. It simulates application level remote backup solution. This experiment shows the backup strategy, based on the proposed driver (strategy2) has faster writing speed than the other backup strategies (strategy3 and strategy4). This is expected as strategy2 eliminates application level overhead. Different computer hardware configuration and network infrastructure setup will affect the backup performance. - The second experiment is a burst-write performance under different workloads and different time intervals, as shown in
FIG. 4 . Note that the burst read performance is similar to single read performance. Each point plotted inFIG. 4 represents the response time for a 20-database-insertion (20 threads), simulating twenty (20) clients utilizing the database server at different time intervals. As shown inFIG. 4 , there are fluctuations in response time, which is caused by contention.Strategy 1 along line 40 (no backup) and strategy 2 along line 42 (RORIB) still have similar response time for database queries, because strategy 2 alongline 42 avoids application level overhead when doing Real-time Online Remote Info Backup. Strategy 3 along line 44 (application level local backup) exhibits greater response time than strategy 4 along line 46 (application level remote backup) under heavier workload, as shown inFIG. 5 . At this point, local machine resource contention accounts for more time degradation than network traffic contention. However when there is less local contention, network traffic contention becomes more distinguishable than local resource contention, as shown inFIG. 5 . - The backup technique of the present invention is implemented at device driver level. Thus, it is transparent to the user, the file system, and the application programs such as DBMS. It requires no changes to the existing operating system or the physical file structure. As a result, it can be used as a ‘drop-in’ filter between the traditional disk device driver and the file system in an existing system to obtain immediate functionality.
- A prototype of the described RORIB device driver was tested. Results of the proposed driver show significant performance improvements regarding current backup strategies. This performance improvement is achieved without a significant cost increase as would occur using a PBN or a SAN, thus presenting an extremely economically valuable solution to the present alternatives.
- The improvement occurs because additional processing requirement is off loaded from application level to device driver level. By utilizing the existing processing capability of hardware technology this can lead to additional file management capabilities residing in hardware devices rather than the current slower software approach in use today.
- Although the present invention has been shown and described with respect to several preferred embodiments thereof, various changes, omissions and additions to the form and detail thereof, may be made therein, without departing from the spirit and scope of the invention.
Claims (31)
1-22. (canceled)
23. A computer readable storage medium on which a plurality of software components are stored which include instructions to be executed on an associated host system to backup data, the software components comprising:
a first device driver configured to be disposed between a first disk driver and a first system interface of a first host system for selectively processing calls to a first disk or a second disk for backup of data resident on the first disk, wherein when executed on the first host system, the first device driver is configured to perform operations comprising;
monitoring availability of the first disk through interrogation of the first disk driver independent of the system interface, wherein if the first disk is deemed inaccessible, the first device driver routes read/write operations of the first computing system to a second host system.
24. The storage medium of claim 23 , wherein the operations of the first device driver further comprise communicating with a network interface card driver on a first host system to create a connection with the second host system.
25. The storage medium of claim 24 , wherein the network interface card driver on the first host system communicates with the second host system via the Internet.
26. The storage medium of claim 23 , wherein the operations of the first device driver further comprise communicating with a first disk cache disk driver associated with the first host system to perform caching.
27. The storage medium of claim 23 , wherein the first device driver when installed on the first host system is configured as a code layer disposed between the system interface and the first disk driver.
28. The storage medium of claim 23 , wherein the operations of the first device driver further comprise communicating with a second device driver disposed in the second host system.
29. The storage medium of claim 28 , wherein the second device driver communicates with the first device driver through a network interface card driver disposed in the second host system.
30. A computer readable storage medium on which a plurality of software components are stored which include instructions to be executed on an associated computer system for backing up data between a first computer system and a second computer system, the software components comprising:
a first device driver configured to be disposed between a first disk driver and a first system interface of a first computing system for selectively processing calls to a first disk or a second disk for backup of data resident on the first disk, wherein when executed on the first computer system, the first device driver is configured to perform operations comprising:
monitoring availability of the first disk through interrogation of the first disk driver independent of the system interface, wherein if the first disk is deemed inaccessible by the first device driver, the first device driver routes read/write operations of the first computing system to a second computing system; and
a second device driver disposed between a second disk driver and a second system interface of a second computing system for selectively processing calls to a second disk or the first disk, wherein when executed on the second computer system, the second device driver is configured to perform operations comprising:
monitoring availability of the second disk through interrogation of the second disk driver independent of the second system interface, wherein if the second disk is deemed inaccessible by the second device driver, the second device driver routes read/write operations of the second computing system to the first computing system.
31. The storage medium of claim 30 , wherein the operations of the first device driver comprise communicating with a network interface card driver on the first computing system to create a connection with the second computing system.
32. The storage medium of claim 31 , wherein the network interface card driver on the first computing system communicates with the second computing system via an Internet connection, a WAN connection, or a LAN connection.
33. The storage medium of claim 30 , wherein the operations of the first device driver comprise communicating with a first disk cache disk driver associated with the first computer system to perform caching.
34. The storage medium of claim 30 , wherein when installed on the first computer system, the first device driver is configured as a code layer disposed between the system interface and the first disk driver.
35. The storage medium of claim 30 , wherein when installed on the second computer system, the second device driver is configured as a code layer disposed between the second system interface and the second disk driver.
36. The storage medium of claim 35 , wherein the second device driver communicates with the first device driver through a network interface card driver disposed in the second computing system.
37. A computer readable storage medium on which a plurality of software components are stored which include instructions to be executed on an associated computer system for routing data between a first computer system and a second computer system, the software components comprising:
a first device driver configured to be disposed between a first system interface and a first disk driver of a first computing system, wherein the first device driver when executed on the first computer systems is configured to perform operations comprising:
monitoring at least some data calls intended to be processed by the first disk driver and interrogating the first disk driver to determine the accessibility of read/write access to a first disk controlled by the first disk driver to service the at least some data calls, wherein when the device driver determines from the interrogation of the disk driver that the first disk is unable to service one or more read/write calls, the device driver routes the read/write calls to a second computing system.
38. The storage medium of claim 37 , wherein the operations of the first device driver comprise communicating with a network interface card driver on the first computing system to create a connection with the second computing system.
39. The storage medium of claim 38 , wherein the network interface card driver on the first computing system communicates with the second computing system via the Internet.
40. The storage medium of claim 37 , wherein the operations of the first device driver comprise communicating with a first disk cache disk driver associated with the first computer system to perform caching.
41. The storage medium of claim 37 , wherein the first device driver communicates with a second device driver of the second computing system through a network connection.
42. The storage medium of claim 41 , further comprising a NIC driver configured to establish a communication connection between the first device driver and the second device driver.
43. A system for reading and writing data between a first computer system and a second computer system, the system comprising:
a processor:
a computer readable storage medium coupled to the processor, wherein the computer readable storage medium includes:
code for a first device driver disposed between a first disk driver and a first system interface for selectively processing calls to a first disk or a second disk for backup of data resident on the first disk; and
code for monitoring availability of the first disk through interrogation of the first disk driver independent of the system interface, wherein if the first disk is deemed inaccessible, the first device driver routes read/write operations of the first computing system to a second computing system.
44. The system of claim 43 , wherein the first device driver comprises code for communicating with a network interface card driver on a first computing system to create a connection with the second computing system.
45. The system of claim 43 , wherein the first device driver communicates with a first disk cache disk driver to perform caching.
46. The system of claim 43 , wherein the first device driver is configured as a code layer disposed between the system interface and the first disk driver.
47. The system of claim 43 , wherein the first device driver communicates with a second device driver disposed in the second computing system.
48. A system for routing data between a first computer system and a second computer system, the system comprising:
a processor:
a computer readable storage medium coupled to the processor, wherein the computer readable storage medium includes:
code for a first device driver disposed between a first system interface and a first disk driver of a first computing system, wherein the first device driver monitors at least some data calls intended to be processed by the first disk driver and wherein the first device driver interrogates the first disk driver to determine the accessibility of read/write access to a first disk controlled by the first disk driver, wherein when the device driver determines from the interrogation of the disk driver that the first disk is unable to service read/write calls, the device driver routes the read/write calls to a second computing system.
49. The system of claim 48 , wherein the first device driver comprises code for communicating with a network interface card driver on the first computing system to create a connection with the second computing system.
50. The system of claim 48 , wherein the first device driver communicates with a first disk cache disk driver to perform caching.
51. The system of claim 48 , wherein the first device driver communicates with a second device driver of the second computing system through a network connection.
52. The system of claim 51 , further comprising a NIC driver configured to establish a communication connection between the first device driver and the second device driver.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/620,461 US20070266060A1 (en) | 2001-03-15 | 2007-01-05 | Remote online information back-up system |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US27628601P | 2001-03-15 | 2001-03-15 | |
US28794601P | 2001-05-01 | 2001-05-01 | |
US31247101P | 2001-08-15 | 2001-08-15 | |
PCT/US2002/007773 WO2002075582A1 (en) | 2001-03-15 | 2002-03-15 | Remote online information back-up system |
US10/659,127 US7177887B2 (en) | 2001-03-15 | 2003-09-10 | Remote online information back-up system |
US11/620,461 US20070266060A1 (en) | 2001-03-15 | 2007-01-05 | Remote online information back-up system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/659,127 Continuation US7177887B2 (en) | 2001-03-15 | 2003-09-10 | Remote online information back-up system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20070266060A1 true US20070266060A1 (en) | 2007-11-15 |
Family
ID=27402810
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/659,127 Expired - Lifetime US7177887B2 (en) | 2001-03-15 | 2003-09-10 | Remote online information back-up system |
US11/620,461 Abandoned US20070266060A1 (en) | 2001-03-15 | 2007-01-05 | Remote online information back-up system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/659,127 Expired - Lifetime US7177887B2 (en) | 2001-03-15 | 2003-09-10 | Remote online information back-up system |
Country Status (5)
Country | Link |
---|---|
US (2) | US7177887B2 (en) |
EP (1) | EP1388085B1 (en) |
AT (1) | ATE347140T1 (en) |
DE (1) | DE60216443T2 (en) |
WO (1) | WO2002075582A1 (en) |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1390854A4 (en) * | 2001-05-01 | 2006-02-22 | Rhode Island Education | Distributed raid and location independence caching system |
US7493626B2 (en) * | 2003-04-02 | 2009-02-17 | Apple Inc. | Method and apparatus for communicating between device drivers in a computer system |
JP4124348B2 (en) * | 2003-06-27 | 2008-07-23 | 株式会社日立製作所 | Storage system |
US7219201B2 (en) * | 2003-09-17 | 2007-05-15 | Hitachi, Ltd. | Remote storage disk control device and method for controlling the same |
JP4412989B2 (en) | 2003-12-15 | 2010-02-10 | 株式会社日立製作所 | Data processing system having a plurality of storage systems |
JP4477370B2 (en) * | 2004-01-30 | 2010-06-09 | 株式会社日立製作所 | Data processing system |
US7370163B2 (en) * | 2004-05-03 | 2008-05-06 | Gemini Storage | Adaptive cache engine for storage area network including systems and methods related thereto |
JP4519563B2 (en) | 2004-08-04 | 2010-08-04 | 株式会社日立製作所 | Storage system and data processing system |
US7818608B2 (en) * | 2005-02-18 | 2010-10-19 | Microsoft Corporation | System and method for using a file system to automatically backup a file as a generational file |
US7702781B2 (en) * | 2006-03-03 | 2010-04-20 | Teoco Corporation | System and method of storing data files at a remote storage facility |
US7624134B2 (en) * | 2006-06-12 | 2009-11-24 | International Business Machines Corporation | Enabling access to remote storage for use with a backup program |
US20070299891A1 (en) * | 2006-06-26 | 2007-12-27 | Bellsouth Intellectual Property Corporation | Data back-up utility |
US7882073B1 (en) * | 2007-09-28 | 2011-02-01 | Symantec Operating Corporation | Backup agent for dynamically decomposing a database backup request to constituent files to facilitate backup restartability |
US20090158284A1 (en) * | 2007-12-18 | 2009-06-18 | Inventec Corporation | System and method of processing sender requests for remote replication |
US8271443B1 (en) * | 2009-06-29 | 2012-09-18 | Symantec Operating Corporation | Backup system including a privately accessible primary backup server and a publicly accessible alternate backup server |
KR20110019891A (en) * | 2009-08-21 | 2011-03-02 | 삼성전자주식회사 | Remote date back-up method and remote data back-up system |
US9497266B2 (en) | 2013-02-22 | 2016-11-15 | International Business Machines Corporation | Disk mirroring for personal storage |
TWI579689B (en) | 2015-09-25 | 2017-04-21 | 瑞昱半導體股份有限公司 | Data backup system |
CN106569908B (en) * | 2015-10-08 | 2020-05-15 | 合肥沛睿微电子股份有限公司 | Data backup system |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5754888A (en) * | 1996-01-18 | 1998-05-19 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | System for destaging data during idle time by transferring to destage buffer, marking segment blank , reodering data in buffer, and transferring to beginning of segment |
US5764903A (en) * | 1994-09-26 | 1998-06-09 | Acer America Corporation | High availability network disk mirroring system |
US5771354A (en) * | 1993-11-04 | 1998-06-23 | Crawford; Christopher M. | Internet online backup system provides remote storage for customers using IDs and passwords which were interactively established when signing up for backup services |
US5857208A (en) * | 1996-05-31 | 1999-01-05 | Emc Corporation | Method and apparatus for performing point in time backup operation in a computer system |
US5923876A (en) * | 1995-08-24 | 1999-07-13 | Compaq Computer Corp. | Disk fault prediction system |
US5974563A (en) * | 1995-10-16 | 1999-10-26 | Network Specialists, Inc. | Real time backup system |
US6092066A (en) * | 1996-05-31 | 2000-07-18 | Emc Corporation | Method and apparatus for independent operation of a remote data facility |
US6243795B1 (en) * | 1998-08-04 | 2001-06-05 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | Redundant, asymmetrically parallel disk cache for a data storage system |
US6324654B1 (en) * | 1998-03-30 | 2001-11-27 | Legato Systems, Inc. | Computer network remote data mirroring system |
-
2002
- 2002-03-15 DE DE60216443T patent/DE60216443T2/en not_active Expired - Lifetime
- 2002-03-15 EP EP02753627A patent/EP1388085B1/en not_active Expired - Lifetime
- 2002-03-15 WO PCT/US2002/007773 patent/WO2002075582A1/en active IP Right Grant
- 2002-03-15 AT AT02753627T patent/ATE347140T1/en not_active IP Right Cessation
-
2003
- 2003-09-10 US US10/659,127 patent/US7177887B2/en not_active Expired - Lifetime
-
2007
- 2007-01-05 US US11/620,461 patent/US20070266060A1/en not_active Abandoned
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5771354A (en) * | 1993-11-04 | 1998-06-23 | Crawford; Christopher M. | Internet online backup system provides remote storage for customers using IDs and passwords which were interactively established when signing up for backup services |
US5764903A (en) * | 1994-09-26 | 1998-06-09 | Acer America Corporation | High availability network disk mirroring system |
US5923876A (en) * | 1995-08-24 | 1999-07-13 | Compaq Computer Corp. | Disk fault prediction system |
US5974563A (en) * | 1995-10-16 | 1999-10-26 | Network Specialists, Inc. | Real time backup system |
US5754888A (en) * | 1996-01-18 | 1998-05-19 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | System for destaging data during idle time by transferring to destage buffer, marking segment blank , reodering data in buffer, and transferring to beginning of segment |
US5857208A (en) * | 1996-05-31 | 1999-01-05 | Emc Corporation | Method and apparatus for performing point in time backup operation in a computer system |
US6092066A (en) * | 1996-05-31 | 2000-07-18 | Emc Corporation | Method and apparatus for independent operation of a remote data facility |
US6324654B1 (en) * | 1998-03-30 | 2001-11-27 | Legato Systems, Inc. | Computer network remote data mirroring system |
US6243795B1 (en) * | 1998-08-04 | 2001-06-05 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | Redundant, asymmetrically parallel disk cache for a data storage system |
Also Published As
Publication number | Publication date |
---|---|
EP1388085A4 (en) | 2005-09-21 |
DE60216443D1 (en) | 2007-01-11 |
US20040117344A1 (en) | 2004-06-17 |
EP1388085A1 (en) | 2004-02-11 |
DE60216443T2 (en) | 2007-09-27 |
EP1388085B1 (en) | 2006-11-29 |
WO2002075582A1 (en) | 2002-09-26 |
ATE347140T1 (en) | 2006-12-15 |
US7177887B2 (en) | 2007-02-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20070266060A1 (en) | Remote online information back-up system | |
US7203807B2 (en) | Storage system making possible data synchronization confirmation at time of asynchronous remote copy | |
US7028078B1 (en) | System and method for performing virtual device I/O operations | |
US9020895B1 (en) | Disaster recovery for virtual machines across primary and secondary sites | |
US7464223B2 (en) | Storage system including storage adapters, a monitoring computer and external storage | |
US7809907B2 (en) | System and method for backup by splitting a copy pair and storing a snapshot | |
US20040078633A1 (en) | Distributing manager failure-induced workload through the use of a manager-naming scheme | |
US8316110B1 (en) | System and method for clustering standalone server applications and extending cluster functionality | |
JP2003076592A (en) | Data storage system | |
US20040225697A1 (en) | Storage operation management program and method and a storage management computer | |
US7734878B1 (en) | System and method for performing virtual device I/O operations | |
US7240153B2 (en) | Storage system and control method thereof for uniformly managing the operation authority of a disk array system | |
JP2005267327A (en) | Storage system | |
WO2003044697A1 (en) | Data replication system and method | |
JP2004252938A (en) | Wide area storage localization system | |
CN113849136B (en) | Automatic FC block storage processing method and system based on domestic platform | |
JP2006092535A (en) | Internal mirroring operation in storage network | |
Khattar et al. | Introduction to Storage Area Network, SAN | |
US20080183961A1 (en) | Distributed raid and location independent caching system | |
US20020194268A1 (en) | Distribute object mechanism | |
US20040158687A1 (en) | Distributed raid and location independence caching system | |
US8234465B1 (en) | Disaster recovery using mirrored network attached storage | |
KR20110090705A (en) | High-availability system structured by virtual device mapping | |
Shu et al. | An implementation of storage-based synchronous remote mirroring for SANs | |
Orimo et al. | Distributed file management techniques based on an autonomous decentralized system concept |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BOARD OF GOVERNORS FOR HIGHER EDUCATION, STATE OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YANG, QING;LI, JIAN;SIGNING DATES FROM 20040109 TO 20060725;REEL/FRAME:025853/0040 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION |