CN103561280B - A kind of based on the degree of depth with the remote plotting method of stream of action information - Google Patents

Abstract

A kind of based on the degree of depth with the remote plotting method of stream of action information, the steps include: (1) server dependence buffering carries out Boundary Extraction;(2) according to step (1) calculated boundary information, extract border stream of action information and boundary depth information, carry out data compression process and be sent to client；(3) draw low-resolution image, use H.264 standard to be sent to client；(4) client accepts the information that server end is sent to, and first by expansion, border stream of action information and boundary depth information is penetrated algorithm and recovers, and obtains complete stream of action information and depth information；(5) data obtained according to step (4), use space-time upwards sampling algorithm to obtain full resolution pricture.

Description

A kind of based on the degree of depth with the remote plotting method of stream of action information

Technical field

The invention belongs to computer virtual reality technology field, a kind of method concretely relating to remote plotting, clothes Business device end draws low-resolution image, calculates the degree of depth and stream of action information, and client goes out High-Resolution Map according to these Information recovering Picture, completes remote plotting.

Background technology

In the last few years, general along with mobile terminals such as the quickening of computer hardware update speed and vehicle-mounted computers And, apply to the role that the mutual server/customer end platform performer in our life of 3-D view is the most important. This platform renders work and gives server huge for amount of calculation and complete, then the frame sequence rendered with the side of video Formula is transferred to client, and the function that client has only to have web page browsing and screen broadcasting is the most permissible.This server/ Client graphical platform has lot of advantages, except reducing the requirement to client configuration, it is also possible to substantial amounts of data It is saved in server, simplifies data maintenance work, improve data security, can also protect in cooperative working environment simultaneously The concordance of card data.But there is the two Major Difficulties in this system: one is server load, large scene or complexity Rendering of model always needs server to carry out substantial amounts of calculating, on the one hand adds response time, on the other hand limits clothes Business device provides the ability of service for multiple stage client；Two is bandwidth, bandwidth higher required for transmission high-definition picture, as passed Defeated HD 1080p video needs the bandwidth of at least 5Mbps to obtain preferably to transmit quality.In order to solve these problems, a lot Research is all to start with from both direction:

(1) server end rendering data amount is reduced.Document 1 Dabrius Burschka, Gregory D Hager, Zachary Dodds etc.Recent Method for Image-Based Modeling and Rendering, IEEE Virtual Reality 2003 March 22-26,2003, Los Angeles, CA, 299. provide at rendering server and Between client, distribution renders load and the tailor-made algorithm of data transmission effectively.Document 2 Cohen-or D, Mann Y, Fleishman S.Deep compression for streaming texture intensive animations.In Proceedings of SIGGRAPH (1999), pp.261-268. utilize stream residual values (high-quality service device end frame and low-quality Difference between amount client frame) improve the rendering capability of client, this method needs powerful client, and cannot Server end is drawn again.Document 3 Sitthi Amorn P, Lawrence J, Yang L, Sander P V, Nehab D.An Improved Shading Cache for Modern GPUs.In Proc.APGV (2008), pp.193-197. are logical Crossing analysis time data re-projection expense on GPU and devise a kind of algorithm, the shading information of huge profit present frame improves The speed that following frame sequence coloring calculates.Document 4 Yang L, Sander P V, Lawrence J.Geometry-Aware Framebuffer Level of Detail.Comp.Graph.Forum (Proc.Of EGSR) 27,4 (2008), 1183- 1188. propose a kind of LOD algorithm controls to carry out the expense of processes pixel.HEMS proposes space-time upwards sampling algorithm, knot The conjunction time reduces the expense of processes pixel to up-sampling and space to the algorithm of up-sampling.Document 5 Herzog R, Eisemann E, Myszkowski K, Seidel H P.Spatio-temporal Upsampling on The GPU.In Proc.of I3D (2010) ACM, pp.91-98. combine with video stream compression rendering, and do not need in compression process Part to be removed carries out rendering work, reduces the load of server end.

(2) acceleration is rendered.Increase the data processing amount in the unit interval, realized by software and hardware accelerated method, as GPU adds Speed technology, storage Access Optimization technology, parallel rendering technology etc., wherein parallel rendering technology is the important set of software acceleration method Become part.Judgment mode, document 6 Molnar S., Cox M., Ellsworth D., et al.A is belonged to according to pel Sorting Classification of Parallel Rendering[J].IEEE Computer Graphics and Applications, 1994,14 (4): 23-32. are divided into three types parallel rendering: Sort-first, Sort-middle and Sort-last.The Sort-first system correspondence position when the geometric manipulations stage determines figure source images at screen, sees document 7 Miyachi H., Shigeta H., Kiyokawa K., et al.Parallelization of Particle Based Volume Rendering on Tiled Display Wall[C].Network-Based Information Systems (NBiS), Takayama, Gifu, Japan, 2010. and document 8 Humphreys G., Eldridge M..WireGL:A Scalable Graphics System for Cluster [C] .ACM SIGGRAPH, Los Angeles, California, 2001., advantage is to have the advantages that streamline is separate and the traffic is few, but owing to the reasons such as pel skewness are easy Cause laod unbalance；Sort-middle system pel ownership judge occur geometric manipulations and pixelation processing stage it Between, such as Chromium framework, see document 8 Humphreys G., Houston M., Ng.R., et al.Chromium:a Stream-processing Framework for Interactive Rendering on Clusters [C] .ACM, San Diego, 2008., its advantage is advantageous for modular implementation, but owing to volume of transmitted data increases with processor geometry of numbers progression Long, limit the extensibility of system；The pel ownership of Sort-last system occurs in pixelation processing procedure, as Parallel-Mesa system, the system expandability is good, preferably achieves load balance simultaneously, but needs substantial amounts of pixel transmission, Occupy massive band width so that image aggregate velocity becomes system bottleneck, the problem that simultaneously also there is laod unbalance.Due to this Three kinds of systems are respectively arranged with pluses and minuses, and increasing system begins to use the scheme wherein combined both certain, such as Chromium frame Frame and Parallel-SG.

Current remote plotting technical research is less able to take into account server end load, client configuration and bandwidth etc. many Demand in terms of Zhong, otherwise in order to reduce server end load and add client burden thus cause client configuration needs Higher, or add server end load to reduce client configuration, or can cause that bandwidth demand is the most high asks Topic.

Summary of the invention

The technology of the present invention solves problem: overcome some limitation of prior art, it is provided that a kind of based on the degree of depth and action The remote plotting method of stream information, effectively reduces server end load, client configuration requirement and bandwidth demand.

The technical solution of the present invention: remote plotting based on the degree of depth Yu stream of action information, first has at server end Carrying out the degree of depth and stream of action information retrieval, boundary information extracts and low-resolution image is drawn, and then carries out data compression, finally Client obtains high-definition picture by these information, it is characterised in that step is as follows:

A kind of remote plotting method based on the degree of depth Yu stream of action information, first has to carry out the degree of depth and action at server end Stream information extracts, and boundary information extracts and low-resolution image is drawn, and then carries out data compression, and last client passes through these Information obtains high-definition picture, it is characterised in that step is as follows:

(1) server dependence buffering carries out Boundary Extraction；

(2) according to step (1) calculated boundary information, extract border stream of action information and boundary depth information, enter Row data compression process is also sent to client；

(3) draw low-resolution image, use H.264 standard to be sent to client；

(4) client accepts the information that server end is sent to, first to border stream of action information and boundary depth information Penetrate algorithm by expansion to recover, obtain complete stream of action information and depth information；

(5) data obtained according to step (4), use space-time upwards sampling algorithm to obtain full resolution pricture.

Present invention advantage compared with prior art is:

(1) remote plotting method designed by the present invention significantly reduces the load of server end, in remote plotting, The groundwork of server end has been wasted in and has rendered part, and this method only draws low resolution image, thus server from numerous Rendering in work of weight frees.

(2) present invention reduces the configuration requirement to client, use the mode of blit picture to carry out remote plotting and make visitor Family end is made without models treated, is more made without drawing, client configuration requirement.

(3) present invention reduces the remote plotting requirement to bandwidth, though the transmission of full resolution pricture to be carried out, because clothes Business device end only transmits low resolution image, and high-resolution image is to generate in client, the lowest to bandwidth requirement.

Accompanying drawing explanation

Fig. 1 is the flow chart of the inventive method；

Fig. 2 is data compression algorithm schematic diagram of the present invention；

Fig. 3 is broadcast algorithm schematic diagram of the present invention；

Fig. 4 is the high-definition picture that server end is drawn；

Fig. 5 is the low-resolution image that server end is drawn；

Fig. 6 is the high-definition picture that client is recovered.

Detailed description of the invention

As it is shown in figure 1, the present invention includes server end and client two parts content, specifically comprise the following steps that

1, server end treats the extraction of boundary information of drawing frames image, the steps include: to use Laplace operator, according to Data in attribute buffering carry out Boundary Extraction.Laplace operator is a kind of Second Order Differential Operator, is defined as:

${\▿}^2=\frac{{\∂}^{2}f}{{\∂x}^2}+\frac{{\∂}^{2}f}{{\∂y}^2}---\left(1\right)$

For image, Laplace operator can be reduced to:

G (i, j)=4f (i, j)-f (i+1, j)-f (i-1, j)-f (i, j+1)-f (i, j-1) (2)

Wherein (i j) represents that (i, j) Laplce's Judge Operator of place's pixel, (i j) represents pixel to f to band drawing image to g (i, depth value j).The border extracted is the width of 2 pixels.For the less region of change that prevents loss, carry on border After taking end, every 32*32 pixel is added a Pixel Information again in data boundary.

2, extract boundary depth information and border stream of action information and carry out data compression, the steps include: attribute buffering In have depth information, surface normal, texture information etc., according to data boundary the degree of depth on wherein border and stream of action information Extract, i.e. boundary depth information and border stream of action information；Then these data are encoded, as in figure 2 it is shown, to two Whether kind of one pixel of element is that border sample (whether i.e. one pixel is on border), the value of border sample (are i.e. located The attribute of pixel on border, includes depth value and action flow valuve here) encode respectively.First with the form of bianry image Whether being that border sample encodes to a pixel, 1,0 expression is not if representing that this pixel is border sample.For border sample Value, by by row scanning in the way of store.

3, low-resolution image is drawn and transmission, the steps include:, according to data boundary, to adjust attribute buffering, only retains The value on border, then renders according to attribute buffering, obtains low-resolution image, then uses H.264 standard to carry out figure As coding and transmission.

4, client uses broadcast algorithm to regain one's integrity the degree of depth and dynamic according to boundary depth information and border stream of action information Making stream information, its step is as follows: as it is shown on figure 3, its core is a kind of " push-and-pull mechanism ", first passes through reduction image resolution ratio and comes Filling cavity, concrete operations are by continuous down sample and obtain pyramid shape image:

$D(x,y)=\left\{\begin{array}{cc}I(2x,2y),& w(2x,2y)>0\\ \frac{{\mathrm{\Σ}}_{x^{\′},y^{\′}\∈N_{2x,2y}}w(x^{\′},y^{\′})I(x^{\′},y^{\′})}{{\mathrm{\Σ}}_{x^{\′},y^{\text{'}}\∈N_{2x,2y}}w(x^{\′},y^{\′})},& w(2x,2y)>0\end{array}\right.---\left(3\right)$

Wherein, N_{2x, 2y}={ (x-1, y-1), (x-1, y+1), (x+1, y+1), (x+1, y-1) }, represents local neighbours, I table Showing image based on border, w is weight, and 1.0 represent boundary sample, and 0.0 represents empty pixel.Increased by each rank of this operation Adding a pixel wide of boundary sample, cavity is filled up rapidly, chooses what a pixel obtained even for every 32*32 pixel Boundary sample, the at most down sample through 5 grades of pyramids can fill up all cavities；Then the cavity propagation filled up Return high-definition picture.By top-down mode, the sample of rough rank is " pulled " back to oneself corresponding exact position. Owing to remaining the value (pixel enclosed) of even number position in the stage pushed away, so in the stage drawn only from even bit value, At each layer of pyramid, residual pixel uses the method for local diffusion replicate, use first diagonal in this process The mode of rear coordinate axes can obtain preferable effect.

5, client is according to the degree of depth and stream of action information and the full resolution pricture of former frame, uses space-time to calculate to up-sampling Method recovers high-definition picture, the steps include: that the previous frame image first passing through stream of action information with being stored in client pushes away Break and each location of pixels of present frame.Then use the bidirectional weighting scheme that Herzog proposes to previous frame image and and present frame Pixel adjacent on space-time in low resolution image gives weight respectively, obtains present frame H by weighted sum_tIn the value of each pixel, Finally give full resolution pricture.Concrete formula is as follows:

$h\left(i\right)=\frac{1}{\mathrm{\Σ}{w}_s}\underset{j\∈N\left\{i\right\}}{\mathrm{\Σ}}{w}_s(i,j)l\left(j\right)---\left(4\right)$

Wherein, h (i) represents the end value of pixel at i, and N{i} represents the neighbor set of i, and l (j) represents in low resolution From the value of the nearest pixel of j element, w on rate image l_s(i, j) represents spacetime geometry weight, is specifically defined as:

$w_s(i,j)=n(\max {(\mathrm{0,1}-(\stackrel{\→}{n_i}\·\stackrel{\→}{n_j}))}^2,{\mathrm{\σ}}_n^2)\·d({|z_i-z_j|}^2,{\mathrm{\σ}}_z^2)\·k(i,j)---\left(5\right)$

Wherein, n represents pixel i, the direction vector between j, and d represents linear depth vector, and k is pattern space filter,Represent pixel i, direction vector at j, z_i, z_jRepresent the degree of depth,For custom variable, after tested,Definition It is 0.2,It is defined as 3% regarding vertebral body the most distant and nearest plane difference degree.

Experimental result

This experiment uses server/customer end framework, and threedimensional model is stored in server end, and server end completes model and reads Data needed for taking and drawing client are extracted and are processed, and are then communicated to client, the client number to receiving According to recovering, when user produces mutual, according to type of interaction, select to be properly calculated response or handle in client Interactive information passes to server end, server end complete the response to this interactive operation.Fig. 4 is that automobile engine model is at clothes The high-definition picture that business device end is drawn, Fig. 5 is the low-resolution image that server end is drawn, and Fig. 6 is that client recovers High-definition picture.

Experiment uses server configures to be Inter Core2 2.2GHz, hosts 4GB, and video card is NVIDIA GeForce9800, uses VC++ exploitation under win7 system, and the interface that drawing uses Ogre to provide completes.Client Being a notebook, be configured to Inter Core2 1.8GHz, host 3GB, video card is the ATI Mobility of video memory 128M Radeon X2300, client-side program uses MFC Development of Framework, it is not necessary to carry out drawing.Test 800*600 (SVGA), 1080*720 (HD720p) and 1920*1080 (HD1080p) three kinds has carried out remote testing under differentiating, mainly use the present invention Method and traditional direct draw full resolution pricture at server end and be sent to the mode of client and contrast, result such as table 1 Shown in.

Context of methods and traditional method Data Comparison in the experiment of table 1 remote plotting

In upper table, traditional Remote method for drafting the response time of server end mainly include full resolution pricture render and H.264 encoding image, the time used uses T respectively_renAnd T_h264Representing, using method includes low resolution image wash with watercolours herein DyeBoundary Extraction, the degree of depth and stream of action information calculate (T_pre) and these contents to be sent are encoded (T_enc), Speedup represents the speed-increasing effect using this method, it can be seen that use the response time of methods herein server end to obtain Significant raising, as a example by car engine, the execution time of every frame has 422.4ms to bring up to 124.1ms, and execution efficiency obtains To significantly improving.In method employed herein, client adds some response times to the reconstruction of full resolution pricture, But relatively in the time that server end is saved within the scope of acceptable.

In a word, the present invention has dropped the workload rendering part taking server ample resources to get off, and only draws low point Resolution image, saves the time then carries out taking the less Boundary Extraction of resource and the degree of depth, the work of stream of action information retrieval, greatly Reduce greatly server end load, also reduce the demand to bandwidth；Simultaneously as given server end drawing Carry out, so the requirement that client is configured by the present invention is the lowest.

The content not being described in detail in description of the invention belongs to prior art known to professional and technical personnel in the field.

The above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art For Yuan, under the premise without departing from the principles of the invention, it is also possible to make some improvements and modifications, these improvements and modifications also should It is considered as protection scope of the present invention.

Claims (3)

1., based on the degree of depth and a remote plotting method for stream of action information, first have to carry out the degree of depth and stream of action at server end Information retrieval, boundary information extracts and low-resolution image is drawn, and then carries out data compression, and last client obtains high-resolution Rate image, it is characterised in that step is as follows:

(1) server dependence buffering carries out Boundary Extraction；

(2) according to step (1) calculated boundary information, extract border stream of action information and boundary depth information, count Process according to compression and be sent to client；

(3) draw low-resolution image, use H.264 standard to be sent to client；

(4) client accepts the information that server end is sent to, and first passes through border stream of action information and boundary depth information Expansion is penetrated algorithm and is recovered, and obtains complete stream of action information and depth information；

(5) data obtained according to step (4), use space-time upwards sampling algorithm to obtain full resolution pricture；

In described step (2), boundary depth information and the extraction of border stream of action information and data compression, its step is as follows: first Boundary depth information and border stream of action information, then to two kinds of elements, one are extracted according to boundary information in first dependence relief area Whether whether individual pixel is border sample, border sample value, encodes: first with the form of bianry image to a pixel be Border sample encodes, and 1 represents that this pixel is border sample, and 0 expression is not；For the value of border sample, with by row scanning Mode store；

Step (4) including: its core is a kind of " push-and-pull mechanism ", first passes through reduction image resolution ratio and carrys out filling cavity, specifically Operation is by continuous down sample and obtains pyramid shape image, and each rank increases a pixel wide of boundary sample, empty Hole is filled up rapidly；Then the cavity filled up is propagated back to high-definition picture；By top-down mode, rough rank Sample be " pulled " back to oneself corresponding exact position；Owing to remaining the value of even number position in the stage pushed away, so drawing Stage only from even bit value, at each layer of pyramid, residual pixel uses the method for local diffusion replicate, at this The mode of coordinate axes after the first diagonal of employing during individual；

Step (5) comprises the following steps that the previous frame image first passing through stream of action information and be stored in client is inferred and works as Each location of pixels of front frame；Then the bidirectional weighting scheme using Herzog to propose differentiates figure to previous frame image and present frame are low Pixel adjacent on space-time in Xiang gives weight respectively, obtains present frame H by weighted sum_tIn the value of pixel, finally give Full resolution pricture.

The most according to claim 1 based on the degree of depth and the remote plotting method of stream of action information, it is characterised in that: described step (1) extraction of boundary information in, the steps include: to use Laplace operator, carries out border according to the data in attribute buffering Extracting, the border extracted is the width of 2 pixels；For the less region of change that prevents loss, after Boundary Extraction terminates, Every 32*32 pixel is added a Pixel Information again in data boundary.

The most according to claim 1 based on the degree of depth and the remote plotting method of stream of action information, it is characterised in that: described step (3) in, first low-resolution image is drawn according to boundary information, i.e. only to boundary pixel and one of every 32*32 pixel decimation Pixel carries out rendering obtaining low-resolution image, then to send after H.264 standard encodes.