CN103533004A - Data transmission method and system based on stage compression - Google Patents
Data transmission method and system based on stage compression Download PDFInfo
- Publication number
- CN103533004A CN103533004A CN201210233707.8A CN201210233707A CN103533004A CN 103533004 A CN103533004 A CN 103533004A CN 201210233707 A CN201210233707 A CN 201210233707A CN 103533004 A CN103533004 A CN 103533004A
- Authority
- CN
- China
- Prior art keywords
- compression
- data
- server
- algorithm
- compression algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a data transmission method and system based on stage compression. The method includes: a source server compresses source data to be transmitted through adoption of a first compression algorithm and then transmits the source data to be transmitted to an access server; and the source server unzips the compression data from the source server and compresses the unzipped data through a second compression algorithm and then sends the data to a client, wherein the first compression algorithm and the second algorithm are different compression algorithms. Through the data transmission method and system based on the stage compression, the compression ratio of the data can be optimized and bandwidth cost can be reduced.
Description
Technical field
The present invention relates to the data transmission technology of the Internet, relate in particular to a kind of data transmission method and system based on implements spatial scalable compression.
Background technology
Gzip compress technique, the earliest for the compressing file of unix system, has become the very general a kind of data compression format of use on the Internet now.HTML (Hypertext Markup Language) (HTTP, Hypertext Transport Protocol) the Gzip compression on is a kind of technology that is used for improving network (WEB) application program capacity, the WEB website of large flow usually allows user experience speed faster by Gzip compress technique, this generally refers to a function of installing in http server, when having user to access the website in this server, this function in server is transferred to client after just web page contents being utilized to Gzip compression algorithm, by browser or other HTTP client utilitys, will after compressed file decompress(ion), be shown, generally plain text content is compressible to 40% of life size, like this, just can reduce the output bandwidth of http server, improve message transmission rate.
But there is following problem in Gzip compress technique: because Gzip compression is based on Single document, therefore, compression efficiency can be lower, the limited space of data compression, corresponding, the bandwidth of saving is also limited.At present, Gzip is more common compression method, because all client browsers can carry out decompress(ion) to it substantially.
SDCH (Shared Dictionary Compression over HTTP) compression, compresses identical content in each HTTP page by dictionary compression algorithm, reduces the transmission of identical content.As: in a website, be all common head and afterbody, even some sidebar are also common.If adopt Gzip compress technique, these common information will be reloaded when each page open so, but use SDCH compress mode, and these common contents are only just passable with transmission primaries; Client, after packed data, obtains corresponding dictionary according to wherein subsidiary header information, and utilizes field to carry out decompress(ion) to data.As can be seen here, SDCH compress mode has overcome the 1st problem that Gzip compress mode exists, and SDCH compression ratio, higher than Gzip, makes the output bandwidth of http server less.
For example, but SDCH compress mode also exists some problems: 1, SDCH compression also needs the cooperation of client, is unfavorable for like this improvement of SDCH compression algorithm; 2, SDCH compressed file is used vcdiff form, can not use Hofman tree coding, and it is optimum that compression ratio still can not reach.
Summary of the invention
In view of this, main purpose of the present invention is to provide a kind of data transmission method and system based on implements spatial scalable compression, with the compression ratio of optimization data, reduces bandwidth cost.
For achieving the above object, technical scheme of the present invention is achieved in that
The invention provides a kind of data transmission method based on implements spatial scalable compression, the method comprises:
Source server is transferred to access server after source data waiting for transmission is adopted to the first compression algorithm compression;
Access server carries out decompress(ion) to the packed data from described source server, and sends to client after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
The method also comprises: the data that described client sends access server are carried out decompress(ion).
Described source server comprises the server that is positioned at the DC of data center, and described access server comprises the server that is positioned at the OC of operation center.
Described source server comprises the server that data source is provided, and described access server comprises the nearest available server of distance client in network topology.
The compression efficiency of described the first compression algorithm is higher than described the second compression algorithm.
Described the first compression algorithm comprises the compression algorithm based on sharing dictionary, and described the second compression algorithm comprises Gzip algorithm.
Described access server is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
The present invention also provides a kind of data transmission system based on implements spatial scalable compression, and this system comprises:
Source server is transferred to access server after source data waiting for transmission being adopted to the first compression algorithm compression;
Access server, carries out decompress(ion) for the packed data to from described source server, and sends to client after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
Described client is carried out decompress(ion) for the data that access server is sent.
Described source server comprises the server that is positioned at the DC of data center, and described access server comprises the server that is positioned at the OC of operation center.
Described source server comprises the server that data source is provided, and described access server comprises the nearest available server of distance client in network topology.
The present invention also provides a kind of data transmission system based on implements spatial scalable compression, and this system comprises: the first compression module, the second compression module and client modules, wherein,
Described the first compression module is transferred to described the second compression module after source data waiting for transmission being adopted to the first compression algorithm compression;
Described the second compression module, carries out decompress(ion) for the packed data to from described the first compression module, and sends to client modules after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
Described client modules carries out decompress(ion) for the data that the second compression module is sent.
The compression efficiency of described the first compression algorithm is higher than described the second compression algorithm.
Described the first compression algorithm comprises the compression algorithm based on sharing dictionary, and described the second compression algorithm comprises Gzip algorithm.
Described the second compression module is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
A kind of data transmission method and system based on implements spatial scalable compression provided by the present invention, adopted two stages of compression, first order compression occurs between source server and access server, by source server, compressed, access server decompress(ion), because internet content provider can be controlled source server and access server, so just can between the two, configure the compression algorithm (for example compression algorithm based on sharing dictionary) of the high compression ratio that need to work in coordination, and between access server and client, adopt the client algorithm of decompress(ion) (for example Gzip) voluntarily; Adopt two-stage compression like this, the compression ratio of energy optimization data, improves compression efficiency greatly, and then reaches the object that reduces overall bandwidth cost.
Accompanying drawing explanation
Fig. 1 is the flow chart of a kind of data transmission method based on implements spatial scalable compression of the embodiment of the present invention;
Fig. 2 is the flow chart of the another kind of the embodiment of the present invention data transmission method based on implements spatial scalable compression;
Fig. 3 is the schematic diagram that flows to that in the embodiment of the present invention, HTTP data compression is transmitted;
Fig. 4 is the structural representation of a kind of data compression transmission system of the embodiment of the present invention;
Fig. 5 is the structural representation of the another kind of data compression transmission system of the embodiment of the present invention.
Embodiment
Below in conjunction with the drawings and specific embodiments, the technical solution of the present invention is further elaborated.
Embodiment 1: a kind of data compression transmission method that the embodiment of the present invention provides, as shown in Figure 1, mainly comprises:
In the embodiment of the present invention, source server is to point to the server that user provides Internet service.So-called compression refers to, in compunication, uses the method for bit TYP information still less.
Preferably, the source server in the embodiment of the present invention can be the server that is positioned at data center (DC, Date Center), and described access server is for being positioned at the server of operation center (OC, Operation Center).
Wherein, the first compression algorithm and the second compression algorithm are different compression algorithms, and the compression efficiency of the first compression algorithm is higher than the second compression algorithm.
Preferably, the first compression algorithm comprises the compression algorithm based on sharing dictionary, and the second compression algorithm comprises Gzip algorithm.Wherein, the first compression algorithm is as femtozip or vcdiff algorithm etc.
In addition, the access server in the embodiment of the present invention is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
In practical application, DC is in big city, and bandwidth is more expensive; OC is in small city, and bandwidth is relatively cheap.So-called bandwidth, refers in the unit interval compunication flow that server is exported (or input), generally refers to output bandwidth in practical application.Because the compression efficiency of the first compression algorithm is higher than the second compression algorithm, therefore can significantly reduce the outlet bandwidth of DC side, can increase the outlet bandwidth of OC side simultaneously, utilize the bandwidth price difference of DC and OC, the portions of bandwidth of DC is transferred to OC, thereby reach the object that reduces overall bandwidth cost.
Embodiment 2: a kind of data compression transmission method that the embodiment of the present invention provides, as shown in Figure 1, mainly comprises:
In the embodiment of the present invention, source server is to point to the server that user provides Internet service.So-called compression refers to, in compunication, uses the method for bit TYP information still less.
Described source server is to provide the server of data source, and access server is the nearest available server of distance client in network topology.
Wherein, the first compression algorithm and the second compression algorithm are different compression algorithms, and the compression efficiency of the first compression algorithm is higher than the second compression algorithm.
Preferably, the first compression algorithm comprises the compression algorithm based on sharing dictionary, and the second compression algorithm comprises Gzip algorithm.Wherein, the first compression algorithm is as femtozip or vcdiff algorithm etc.
In addition, the access server in the embodiment of the present invention is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
Embodiment 3: a kind of data compression transmission method that the embodiment of the present invention provides, as shown in Figure 2, mainly comprises:
After client decompress(ion), what obtain is the source data of uncompressed; Client is carried out page demonstration or other relevant treatment according to the data after decompress(ion).
The source data of embodiment of the present invention indication can be HTTP data, the schematic diagram that flows to below in conjunction with the data compression of HTTP shown in Fig. 3 transmission, illustrate, the required bandwidth of HTTP data of supposing uncompressed is M, the compression ratio of Gzip (i.e. the second compression algorithm) is Rg, the compression ratio of the first compression algorithm that the embodiment of the present invention adopts is Rb, and the bandwidth cost of DC is Pd, and the bandwidth cost of OC is Po.
In DC side, with the bandwidth cost of Gzip compression algorithm compression, be so, only: MRgPd;
With the bandwidth cost that the compression method of the embodiment of the present invention compresses, be: MRgPd+MRgPo.
When (Rg-Rb)/Rg > Po/Pd, adopt the compression method of the embodiment of the present invention can save bandwidth.For example, suppose that Rg is 40%, Rb is 10%, Po/Pd is 30%, in DC side, with the bandwidth cost of Gzip compression algorithm compression, be only MPd0.4, with the bandwidth cost that the compression method of the embodiment of the present invention compresses, be: MPd0.1+MPd0.40.3, the bandwidth cost of the embodiment of the present invention only drops to 55% of the bandwidth cost of Gzip compression algorithm compression.
Embodiment 4: the corresponding above-mentioned data transmission method based on implements spatial scalable compression, and the embodiment of the present invention provides a kind of data transmission system based on implements spatial scalable compression, and as shown in Figure 4, this system comprises: source server 10 and access server 20.
Wherein, source server 10 is transferred to access server after source data waiting for transmission being adopted to the first compression algorithm compression;
Wherein, the first compression algorithm and the second compression algorithm are different compression algorithms.
Preferably, source server 10 is for being positioned at the server of DC, and access server 20 is for being positioned at the server of OC.
Preferably, source server 10 is for the server of data source is provided, and access server 20 is the nearest available server of distance client in network topology.
Wherein, the compression efficiency of the first compression algorithm is higher than the second compression algorithm.Preferably, the first compression algorithm comprises the compression algorithm based on sharing dictionary, and the second compression algorithm comprises Gzip algorithm.The first compression algorithm can be femtozip or vcdiff algorithm.
Embodiment 5: the embodiment of the present invention also provides another data transmission system based on implements spatial scalable compression, and as shown in Figure 5, this system comprises: the first compression module 11, the second compression module 21 and client modules.
Wherein, the first compression module 11 is transferred to the second compression module 21 after source data waiting for transmission being adopted to the first compression algorithm compression;
The second compression module 21, carries out decompress(ion) for the packed data to from the first compression module 11, and sends to client modules after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, the first compression algorithm and described the second compression algorithm are different compression algorithms.
Wherein, the compression efficiency of the first compression algorithm is higher than the second compression algorithm.Preferably, the first compression algorithm comprises the compression algorithm based on sharing dictionary, and the second compression algorithm comprises Gzip algorithm.The first compression algorithm can be femtozip or vcdiff algorithm.
In sum, the embodiment of the present invention is based on two stages of compression, first order compression occurs between source server and access server, by source server, compressed, access server decompress(ion), because internet content provider can be controlled source server and access server, so just can between the two, configure the compression algorithm (for example compression algorithm based on sharing dictionary) of the high compression ratio that need to work in coordination, and between access server and client, adopt the client algorithm of decompress(ion) (for example Gzip) voluntarily.The embodiment of the present invention is compared the data transfer mode that only adopts Gzip compression, compression ratio that can optimization data; Compare the data transfer mode that only adopts SDCH compression, it coordinates without client; Adopt two-stage compression like this, can greatly improve compression efficiency, and then reach the object that reduces overall bandwidth cost.
The above, be only preferred embodiment of the present invention, is not intended to limit protection scope of the present invention.
Claims (16)
1. the data transmission method based on implements spatial scalable compression, is characterized in that, the method comprises:
Source server is transferred to access server after source data waiting for transmission is adopted to the first compression algorithm compression;
Access server carries out decompress(ion) to the packed data from described source server, and sends to client after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
2. method according to claim 1, is characterized in that, the method also comprises: the data that described client sends access server are carried out decompress(ion).
3. method according to claim 1, is characterized in that, described source server comprises the server that is positioned at the DC of data center, and described access server comprises the server that is positioned at the OC of operation center.
4. method according to claim 1, is characterized in that, described source server comprises the server that data source is provided, and described access server comprises the nearest available server of distance client in network topology.
5. according to the method described in claim 1 to 4 any one, it is characterized in that, the compression efficiency of described the first compression algorithm is higher than described the second compression algorithm.
6. method according to claim 5, is characterized in that, described the first compression algorithm comprises the compression algorithm based on sharing dictionary, and described the second compression algorithm comprises Gzip algorithm.
7. according to the method described in claim 1 to 4 any one, it is characterized in that, described access server is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
8. the data transmission system based on implements spatial scalable compression, is characterized in that, this system comprises:
Source server is transferred to access server after source data waiting for transmission being adopted to the first compression algorithm compression;
Access server, carries out decompress(ion) for the packed data to from described source server, and sends to client after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
9. system according to claim 8, is characterized in that, described client is carried out decompress(ion) for the data that access server is sent.
10. system according to claim 8, is characterized in that, described source server comprises the server that is positioned at the DC of data center, and described access server comprises the server that is positioned at the OC of operation center.
11. systems according to claim 8, is characterized in that, described source server comprises the server that data source is provided, and described access server comprises the nearest available server of distance client in network topology.
12. 1 kinds of data transmission systems based on implements spatial scalable compression, is characterized in that, this system comprises: the first compression module, the second compression module and client modules, wherein,
Described the first compression module is transferred to described the second compression module after source data waiting for transmission being adopted to the first compression algorithm compression;
Described the second compression module, carries out decompress(ion) for the packed data to from described the first compression module, and sends to client modules after the data acquisition after decompress(ion) is compressed by the second compression algorithm;
Wherein, described the first compression algorithm and described the second compression algorithm are different compression algorithms.
13. systems according to claim 12, is characterized in that, described client modules carries out decompress(ion) for the data that the second compression module is sent.
14. according to the system described in claim 12 or 13, it is characterized in that, the compression efficiency of described the first compression algorithm is higher than described the second compression algorithm.
15. systems according to claim 14, is characterized in that, described the first compression algorithm comprises the compression algorithm based on sharing dictionary, and described the second compression algorithm comprises Gzip algorithm.
16. according to the system described in claim 12 or 13, it is characterized in that, described the second compression module is configured to be suitable for to adopting the data of the first compression method compression to carry out decompress(ion).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210233707.8A CN103533004A (en) | 2012-07-06 | 2012-07-06 | Data transmission method and system based on stage compression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210233707.8A CN103533004A (en) | 2012-07-06 | 2012-07-06 | Data transmission method and system based on stage compression |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103533004A true CN103533004A (en) | 2014-01-22 |
Family
ID=49934679
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210233707.8A Pending CN103533004A (en) | 2012-07-06 | 2012-07-06 | Data transmission method and system based on stage compression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103533004A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760245A (en) * | 2016-02-03 | 2016-07-13 | 华为技术有限公司 | Data storage method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6728785B1 (en) * | 2000-06-23 | 2004-04-27 | Cloudshield Technologies, Inc. | System and method for dynamic compression of data |
| CN1809162A (en) * | 2006-01-26 | 2006-07-26 | 圆刚科技股份有限公司 | Multiple data stream output from multimedia data stream processing |
| US7761793B1 (en) * | 2005-07-27 | 2010-07-20 | Rockwell Collins, Inc. | SATCOM data compression system and method |
| CN102045772A (en) * | 2009-10-21 | 2011-05-04 | 华为技术有限公司 | Data transmission method and device |
-
2012
- 2012-07-06 CN CN201210233707.8A patent/CN103533004A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6728785B1 (en) * | 2000-06-23 | 2004-04-27 | Cloudshield Technologies, Inc. | System and method for dynamic compression of data |
| US7761793B1 (en) * | 2005-07-27 | 2010-07-20 | Rockwell Collins, Inc. | SATCOM data compression system and method |
| CN1809162A (en) * | 2006-01-26 | 2006-07-26 | 圆刚科技股份有限公司 | Multiple data stream output from multimedia data stream processing |
| CN102045772A (en) * | 2009-10-21 | 2011-05-04 | 华为技术有限公司 | Data transmission method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760245A (en) * | 2016-02-03 | 2016-07-13 | 华为技术有限公司 | Data storage method and device |
| CN105760245B (en) * | 2016-02-03 | 2019-03-26 | 华为技术有限公司 | A kind of method and device of storing data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102571966A (en) | Network transmission method for large extensible markup language (XML) document | |
| US8032609B2 (en) | Dynamic endpoint aggregator for web services | |
| JP2009518755A5 (en) | ||
| CN102970287A (en) | Processing method of hyper text transport protocol (HTTP) request data | |
| CN105407180A (en) | Server message pushing method and device | |
| CN102891796B (en) | Mining intelligent transmission gateway | |
| CN101902485B (en) | Rewriting method of reversal Web agent link | |
| WO2008137117A3 (en) | Method and system for extending the capabilities of embedded devices through network clients | |
| CN102801799A (en) | Real-time monitoring system based on B/S architecture | |
| WO2003083597A3 (en) | Collapsed distributed cooperative memory for interactive and scalable media-on-demand systems | |
| CN102752388A (en) | Browser-based interactive system, browser-based interactive method, browser and cloud server | |
| US9866356B2 (en) | Data distribution method and device | |
| CN101500002A (en) | Fusion publishing method and apparatus oriented to Web content | |
| TW200511088A (en) | Method and system for input an output of data with non-standard I/O devices for web applications | |
| CN102724143A (en) | Method for sending internet of things data and internet of things gateway | |
| CN103368872A (en) | Data packet forwarding system and method | |
| CN103533004A (en) | Data transmission method and system based on stage compression | |
| CN102387136A (en) | System and method for converting distributed PPT (Power Point) files into swf (small web format) files | |
| CN204925773U (en) | Cluster communication monitored control system of switch board | |
| CN102170467A (en) | Method for interacting messages of network page end and desktop application program | |
| CN103685326A (en) | WEB concurrent data transmission method and WEB application system | |
| CN103595812A (en) | Method and device for sending Web service message | |
| CN103873564A (en) | Method for realizing SignalR duplex communication under balanced load condition of multiple servers | |
| CN103051679B (en) | Data transmission method and interface equipment, cloud optimal control equipment | |
| EP1865424A1 (en) | Dynamic aggregation of web services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140122 |
|
| RJ01 | Rejection of invention patent application after publication |