CN102855299A - Method for realizing iterative migration of distributed database without interrupting service - Google Patents
Method for realizing iterative migration of distributed database without interrupting service Download PDFInfo
- Publication number
- CN102855299A CN102855299A CN2012102921793A CN201210292179A CN102855299A CN 102855299 A CN102855299 A CN 102855299A CN 2012102921793 A CN2012102921793 A CN 2012102921793A CN 201210292179 A CN201210292179 A CN 201210292179A CN 102855299 A CN102855299 A CN 102855299A
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- application
- iteration
- guarantee
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention aims at providing a method for realizing capacity expansion and data migration of a distributed database without interrupting application service in a production environment. By dividing data needed to be migrated, mass data is divided into a plurality of few data sets for iterative migration, so as to reduce influences on the normal operation of the application service during migration. At the same time, by understanding a relationship between associated data, the related data is divided into the same iterative process to guarantee data consistency and integrity.
Description
Technical field
The present invention relates to cluster distributed calculating field.
Background technology
In today of rapid development of information technology, no matter be internet or the data volume of enterprises, all be in the development and increase of index speed.It is unprecedented important that the problem of data dilatation has seemed.The data dilatation relates to the redistributing of available data, thereby need to carry out a large amount of migrations to available data, and under production environment, how data are moved in a large number and do not have influence on the normal use of application service, be key subjects that face.
In order under production environment, distributed data base to be moved, invented the method for iteration migration data, the method is divided into some low volume data collection by the data that needs are moved and carries out the iteration migration, except current this low volume data that is moving, remainder data can carry out normal running.Pass through simultaneously correlation analysis, relevant data are distributed in the same iterative process move, thereby guarantee data's consistency and integrality.
Summary of the invention
The purpose of this invention is to provide a kind of dilatation and Data Migration are carried out in application service to distributed data base method of under production environment, not interrupting, by dividing by migration data, mass data is divided into some low volume datas carries out the iteration migration, thus the impact that the normal operation that the reduction transition process is served application brings.Simultaneously by understanding the relation between the associated data, related data is divided into guarantees data consistency and integrality in the same iterative process.
For achieving the above object, the present invention has set a kind of iteration migration strategy.
1. computation migration amount, and carry out data and divide is assigned to the data of need migration in each iterative process fifty-fifty.
2. data are carried out the incidence relation analysis, guarantee that related data is divided in the same iterative process.
3. each iterative process is carried out affairs control, guarantee data consistency and integrality.
4. notice is used when each iterative process begins, and this partial data moves, and can not increase, deletes, change operation to it again, and the notice application migration finishes when finishing, and can increase, delete, change to it operation.
Description of drawings
Accompanying drawing is workflow schematic diagram of the present invention, in this example, is 4 memory nodes with original 2 memory node dilatations.By alternative manner with a part of data in original 2 nodes in the situation that continual service migrate in 2 newly-increased memory nodes.
Embodiment
Below in conjunction with accompanying drawing technical solutions according to the invention are done and to be described in further detail.
Accompanying drawing has been showed the process how to be 4 nodes with 2 memory node dilatations under production environment, and how with the mutual process of application program.
The step of iteration migration is as follows:
1) calculate the data volume that needs migration, and the target memory node that will migrate to.
2) data are divided into some little iteration migration tasks, this process comprises data dependence to be judged, related data will be divided in the same task.
3) utilize multithreading to start simultaneously a plurality of iteration tasks.
When 4) each task begins this part migration of notification application is begun, simultaneously locking data.
5) carry out Data Migration, each iteration task is processed in same affairs.
This part migration of notification application finished when 6) each task immigration was finished, simultaneously unlock data.
7) whether the main thread inquiry also has iteration migration task, as also having, continues migration, otherwise withdraws from, and all migration is finished.
Claims (6)
1. under the continual service condition, the method for distributed data base iteration migration is characterized in that:
1) under production environment, do not interrupt application service and carry out distributed data base dilatation and Data Migration;
2) mass data will be divided into some low volume datas and carry out iteration migration;
3) related data will be divided into same iteration task, guarantee data consistency and integrality;
4) transition process and mutual the unimpeded of application that guarantee of application;
5) a plurality of distributed data piecemeals, parallel migration guarantees that simultaneously service application is unaffected.
2. as not interrupting the method for application service iteration migration data under the right 1 described distributed database environment, it is characterized in that: in the situation that production environment is not interrupted application service, partial data is migrated to new node, to reach the universal method of data dilatation.
3. will be divided into some low volume datas such as right 1 described mass data and carry out the iteration migration, it is characterized in that: the low volume data that is only moving can be influenced, and application program can not increase, deletes, change operation to it, and remainder data is with unaffected.
4. will be divided into same iteration task such as right 1 described related data, and guarantee data consistency and integrality, it is characterized in that: related data will be moved in a collection of task, and data consistency and integrality will be protected.
5. such as mutual the unimpeded of application that guarantee of right 1 described transition process and application, it is characterized in that: when each iteration task begins to move and after the migration end, application program can be notified timely, cause meaningless lock to wait for to avoid attempting to operate the data of moving, thereby guarantee the unimpeded of application program.
6. such as right 1 described transition process, it is high concurrent that transition process keeps, and distributed migration does not affect the use of application itself simultaneously.It is characterized in that: majority can not affect mutually by parallel migration according to distributing simultaneously, and meanwhile, whole features of application itself are all unaffected, keep availability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012102921793A CN102855299A (en) | 2012-08-16 | 2012-08-16 | Method for realizing iterative migration of distributed database without interrupting service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012102921793A CN102855299A (en) | 2012-08-16 | 2012-08-16 | Method for realizing iterative migration of distributed database without interrupting service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102855299A true CN102855299A (en) | 2013-01-02 |
Family
ID=47401887
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012102921793A Pending CN102855299A (en) | 2012-08-16 | 2012-08-16 | Method for realizing iterative migration of distributed database without interrupting service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102855299A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106909597A (en) * | 2016-06-30 | 2017-06-30 | 阿里巴巴集团控股有限公司 | A kind of database migration method and apparatus |
| CN107015972A (en) * | 2016-01-27 | 2017-08-04 | 五八同城信息技术有限公司 | A kind of computer room business migration methods, devices and systems |
| WO2017143957A1 (en) * | 2016-02-26 | 2017-08-31 | 华为技术有限公司 | Data redistribution method and device |
| CN107870960A (en) * | 2017-06-25 | 2018-04-03 | 平安科技（深圳）有限公司 | Data move system and method |
| CN108282378A (en) * | 2017-01-05 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitoring traffic in network |
| WO2018177107A1 (en) * | 2017-03-29 | 2018-10-04 | 腾讯科技（深圳）有限公司 | Data migration method, migration server, and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070033236A1 (en) * | 2005-08-04 | 2007-02-08 | Fujitsu Limited | Database restructuring apparatus, and computer-readable recording medium recording database restructuring program |
| CN101364186A (en) * | 2008-09-27 | 2009-02-11 | 腾讯科技（深圳）有限公司 | Data migration method, system, data migration server and data interface server |
| CN101997911A (en) * | 2010-10-21 | 2011-03-30 | 中兴通讯股份有限公司 | Data migration method and system |
| CN102073462A (en) * | 2010-11-29 | 2011-05-25 | 华为技术有限公司 | Virtual storage migration method and system and virtual machine monitor |
| CN102244685A (en) * | 2011-08-11 | 2011-11-16 | 中国科学院软件研究所 | Distributed type dynamic cache expanding method and system supporting load balancing |
-
2012
- 2012-08-16 CN CN2012102921793A patent/CN102855299A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070033236A1 (en) * | 2005-08-04 | 2007-02-08 | Fujitsu Limited | Database restructuring apparatus, and computer-readable recording medium recording database restructuring program |
| CN101364186A (en) * | 2008-09-27 | 2009-02-11 | 腾讯科技（深圳）有限公司 | Data migration method, system, data migration server and data interface server |
| CN101997911A (en) * | 2010-10-21 | 2011-03-30 | 中兴通讯股份有限公司 | Data migration method and system |
| CN102073462A (en) * | 2010-11-29 | 2011-05-25 | 华为技术有限公司 | Virtual storage migration method and system and virtual machine monitor |
| CN102244685A (en) * | 2011-08-11 | 2011-11-16 | 中国科学院软件研究所 | Distributed type dynamic cache expanding method and system supporting load balancing |
Non-Patent Citations (1)
| Title |
|---|
| SUDIPTO DAS ET AL.: "Live Database Migration for Elasticity in a Multitenant Database for Cloud Plateforms", 《UCSB COMPUTER SCIENCE TECHNICAL REPORT》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107015972A (en) * | 2016-01-27 | 2017-08-04 | 五八同城信息技术有限公司 | A kind of computer room business migration methods, devices and systems |
| CN107015972B (en) * | 2016-01-27 | 2020-02-18 | 五八同城信息技术有限公司 | Method, device and system for migrating machine room services |
| WO2017143957A1 (en) * | 2016-02-26 | 2017-08-31 | 华为技术有限公司 | Data redistribution method and device |
| CN106909597A (en) * | 2016-06-30 | 2017-06-30 | 阿里巴巴集团控股有限公司 | A kind of database migration method and apparatus |
| CN106909597B (en) * | 2016-06-30 | 2020-07-03 | 阿里巴巴集团控股有限公司 | Database migration method and device |
| CN108282378A (en) * | 2017-01-05 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitoring traffic in network |
| CN108282378B (en) * | 2017-01-05 | 2021-11-09 | 阿里巴巴集团控股有限公司 | Method and device for monitoring network flow |
| WO2018177107A1 (en) * | 2017-03-29 | 2018-10-04 | 腾讯科技（深圳）有限公司 | Data migration method, migration server, and storage medium |
| CN108664496A (en) * | 2017-03-29 | 2018-10-16 | 腾讯科技（深圳）有限公司 | Data migration method and device |
| CN107870960A (en) * | 2017-06-25 | 2018-04-03 | 平安科技（深圳）有限公司 | Data move system and method |
| WO2019001272A1 (en) * | 2017-06-25 | 2019-01-03 | 平安科技（深圳）有限公司 | Data migration system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102855299A (en) | Method for realizing iterative migration of distributed database without interrupting service | |
| US10402316B2 (en) | Tracing garbage collector for search trees under multi-version concurrency control | |
| JP5732536B2 (en) | System, method and non-transitory computer-readable storage medium for scalable reference management in a deduplication-based storage system | |
| US10133770B2 (en) | Copying garbage collector for B+ trees under multi-version concurrency control | |
| US9372867B2 (en) | Similarity analysis method, apparatus, and system | |
| CN105871603B (en) | A kind of the real time streaming data processing fail recovery and method of data grids based on memory | |
| US20180004786A1 (en) | Incremental bloom filter rebuild for b+ trees under multi-version concurrency control | |
| CN104461744A (en) | Resource allocation method and device | |
| CN109643310B (en) | System and method for redistribution of data in a database | |
| CN106126601A (en) | A kind of social security distributed preprocess method of big data and system | |
| CN105320773A (en) | Distributed duplicated data deleting system and method based on Hadoop platform | |
| JP6361223B2 (en) | Transaction system | |
| US9984139B1 (en) | Publish session framework for datastore operation records | |
| CN110874271B (en) | Method and system for rapidly calculating mass building pattern spot characteristics | |
| CN106406762A (en) | A repeated data deleting method and device | |
| CN106156245B (en) | Line element merging method and device in electronic map | |
| US20170177473A1 (en) | Garbage collection scope detection for distributed storage | |
| US10599472B2 (en) | Information processing apparatus, stage-out processing method and recording medium recording job management program | |
| CN106528297A (en) | System for managing tasks | |
| CN104102557A (en) | Cloud computing platform data backup method based on clustering | |
| CN109240941B (en) | Garbage recovery method of storage system and related device | |
| CN102779134A (en) | Lucene-based distributed search method | |
| US20160253386A1 (en) | Grid topology change in a distributed data grid when iterating on the contents of the data grid | |
| GB2585543A (en) | Data migration in a hierarchical storage management system | |
| KR101792189B1 (en) | Apparatus and method for processing big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20130102 |