WO2016029283A1

WO2016029283A1 - System for creating images, videos and sounds in an omnidimensional virtual environment from real scenes using a set of cameras and depth sensors, and playback of images, videos and sounds in three-dimensional virtual environments using a head-mounted display and a movement sensor

Info

Publication number: WO2016029283A1
Application number: PCT/BR2015/000128
Authority: WO
Inventors: Samuel MUNIZ
Original assignee: Muniz Samuel
Priority date: 2014-08-27
Filing date: 2015-08-26
Publication date: 2016-03-03

Abstract

The invention relates to a system for creating images, videos and sounds in an omnidimensional virtual environment from real scenes using a set of cameras and depth sensors, and to the playback of images, videos and sounds in three-dimensional virtual environments using a head-mounted display and a movement sensor, for enabling the creation and playback of images, videos, sounds and films in a virtual omnidirectional environment from real scenes, allowing the spectator to control autonomously the position and direction of viewing during playback of the digital file, said system comprising a set of image positioning cameras (2) including image cameras (8) parallel to depth cameras (9), sound capture devices (13) and an inner computer (18). The images of the inner environment formed by the image positioning cameras can thus be captured and positioned in a virtual environment by a central computer (35). An example is the playback of a virtual environment file by a head-mounted display (14) coupled to a movement sensor (15) and a computer program for decoding the data received from the movement sensor (15) and transmitting the data to the sound emitter and to the user display screen, with a remote depth control (19) that allows the spectator to determine autonomously the desired virtual position during playback of the files, giving the user the sensation of total immersion in the virtual environment.

Description

"IMAGE, VIDEO AND SOUND CREATION SYSTEM ONI DI SINGLE VIRTUAL ENVIRONMENT FROM REAL SCENES WITH CAMERA SET AND DEPTH SENSORS, È SQf ^ S VIDEO IN THREE DUAL SADD END VIDEO ENVIRONMENTS YE MOVEMENT SENSOR, "

1 - The object of the present invention is an "IMAGE, VIDEO & SONG CREATING SYSTEM FROM ONE-DIMENSIONAL VIRTUAL ENVIRONMENT WITH CAMERA SET AND DEPTH SENSORS _" . IMAGE, VIDEO AND SONG PLAYBACK 3-D VIRTUAL HEAD MOUNTED DISPLAY WITH MOTION SENSOR "makes full immersion within recorded scenes possible, giving the viewer full autonomy of positioning control and control of vision direction.

U.S. Patent 5,495,576 issued February 27, 1096 discloses an audio visual system in which a set of cameras provided with shape sensors and a sound capture device; ¹ As described devices are positioned around ym target element, wherein each chamber ger image gives a side of the element © sensors define the sounds surface shape, thereby enabling reproduction of eompíeto element three-dimensional shape, however eese invention is limited the three-dimensional reproduction of a target element and not of the entire census environment in which it is reproduced 2D.

can be captured in mm of an image, causing a duplication of this element in the final product of the image.

4 - Solving the above-cited problem US Patent 2001/001055 $, issued August 2, 2001, included a convex mirror in a panoramic camera and digitally matching the panoramic image captured without the time difference between the individual Images.

5 - As reported in U.S. US20Q1 / 0020976 13 d September 2001, was developed stereo panoramic eâmera, which ^comes' are two panoramic cameras using appropriate mirrors attached to a stereomicroscope, Providing this invention to produce panoramic images 3 Dim nsões

6 - Published Patent EP 26144Q5 $ m 2013, a depth chamber was presented from the structured choir light; In stereo vision, such an invention enables a camera to capture the distance between it and the obfetGS in its field of view by activating these data in real time, thereby enabling the reproduction of stereo images and capture of depth data from the scene.

7 - It is common knowledge to capture three-dimensional and pan-wide images of cameras, so we can reproduce stereo-quality images, videos and the like and provide the user with autonomous viewing direction from external commands, either by dragging mouse, a knob of some control or simply turning the head when equipped with a Heasl ounted Dispiay- however none of the techniques presented provides virtual positioning autonomy within the captured image or scene, in all cases the user will necessarily be virtually static - at the scene from where the camera was positioned during the shot. Such inconvenience impairs the immersion of the user to the virtual scene, because unlike the real world to the common techniques the user is always an immovable element.

8 ^»In view of these problems and the purpose of them was to overcome the des ^nvolvido: - ^{'^} ^ T¾ ;, E, ^O' íeçÁ0 ^D: E f ^ S AQE AND VIDEO ^'tones and BIE and VL Tuai 0NlDIM' SiÕMAL Á PiM¾f IR AND REAL SENAS -COM CAMERA SET AND DEPTH SENSORS, IMAGE, VIDEO AND SONG PLAYBACK IN VIRTUAL ENVIRONMENTS TftiDIMENCIO ALS WITH HEAD MOUHTEO DISFLAY ", the subject matter of this patent, which consists of producing video, image, movie or similar files m a virtual omni-clonal environment from a set of conventional positional panoramic cameras to create an area between them, each camera captures the environment around itself while parallel depth cameras panoramic cameras begin to digitize the distance between the panoramic cameras and panoramic image pixels and ambient sound pickup devices preferably positioned at the same point and positioning of the cameras so that both the ambient sound vibrations and the sound pickup positioning point are part of the digitally created data files. mode allows the creation of a: contend file o information such as _» panoramic images from different points of the same environment, distance from each pixel of the image with the camera, sound vibrations caused in the ambient scene at the moment of capturing its locations relative to panoramic images. Since the file types! created virtual environment has all the information pertinent to the scene captured, the reproduction of this file can be performed on ordinary televisions, personal computers and preferably on a Head Wounted Display using a built-in or not motion sensor, which captures Tilt and rotation that the user makes on the HMD, causing the sound and image transmitted on the web to move the same simultaneously, © a device control may be coupled to the MMD or a remote control device.

9 - This new system solves all the drawbacks already mentioned, since the creation of the three-dimensional virtual environment together with the appropriate player, gives the viewer the full possibility, and visual direction control and virtual positioning during the playback of the files. archives, giving the viewer the sensation of a leta-immersion in the reproduced environment.

10 - The accompanying drawings Hu & tr the data capture system (images, distances and sounds) of the environment and its reproduction by bush of an HMD with orskiifeeional range, in which,

11 - The Fig. 1 shows in isometric view, a simulated model of panning camera with panoramic cameras and depth cameras with panoramic range in this case adjacent to a single elm.

12 - Fig. 1a shows in isometric view a simplified model of image positioning camera with panoramas and depth cameras with panoramic range, in this case adjacent to a single axis, with full range of each camera range and image overlay,

13 - Â Fig. 2 shows in isometric view a model of image positioning camera with panoramic cameras and depth cameras with global panoramic range, ie unidirectional range,

Fig. 3 shows in side view an example of the file creation system capturing the real environment data for the virtual environment file creation.

Fig. 4 shows in plan the example cited in Fig. 3,

16 - Fig. 5 shows in side view a user using a Head foynt d Display for playback of virtual environment files, as well as the view screen reproduced in the Head ounted Display,

17 - Â Fig ;, 8 shows in perspective a model¾ó file creation system capturing real environment data in one _. octagon of fights

18 - Fig,? shows an example of sound intensity proportionality for ambient reproduction according to the viewer's virtual positioning in relation to the distance of the sound capture,

Fig. 8 shows an example of a dc-Î²-Î²iÃ n nàmeniò ^' -resnòtD control device.

Fig. 9 shows a background scheme of an image positioning camera as well as co-operative devices.

21 Fig. 10 is a diagram of the operation of the data transmission system from the image position cameras to the central computer.

22 - As illustrated by the trapped figures, "IMAGE, VIDEO, AND SOUND CREATING SESSIONS IN ONE-DIMENSIONAL VIRTUAL ENVIRONMENT FROM REAL SCENES, DEPTH, VIDEO, AND SONG SENSORS In three-dimensional virtual environments with HEAD MOUNTED DISPLAY and PEOPLE SENSOR ^w _s object of the present invention, consists of amenfo posidot emers as shown in Figs. 1 and 2 as positioned to form an internal area therebetween as shown. In Figs. 3- Fig. 6, the image positioning camera is composed of a number of cameras (8) about the same positioning in different directions to form a panoramic range on a single axis as shown in Fig. 1 or about several axes forming a panoramic ^'global reach as illustrated in Fig .. 2 parallel to each imaging camera (8) a depth camera (&) with the same angle al cance in the same direction of the imaging camera (8) so the depth camera (9) also has a wide range, preferably imante depth camera (9) may be the Infrared type transmitter and receiver, which as ^is' knowledge common determines the distance of the elements within range based on the. infrared frequency reached at the receiver, but. This intent is not imitated for a single type of depth camera, as this method is suitable for ^" any type of depth camera. An internal computer (1 $) in the image position camera (2) Illustrated in Fig. 9. understood by, inter ace for data communication with them, interface to react (25) cameras, interface to slide ^■ sound .captação Ç2 ^■■ ára interface IC eras: ^'PRA dídadè- this ínferfaoes poclem be t. them soo medid the utísad models .oon eRcidnais ^as; m fi re, may: It is further reduced these three interfaces (25 _f 26 and 27) to a single communication interface an internal battery (28), a RAM (29) memory, poskâona system ym then (30), a clock (31), a memory itself azenamento (32) _»a processor (34) an interface for external communication (33) may be any type of interface using comyrtlcâ io via wireless telephone or communication, the onboard computer (18) synchronizes the imaging cameras ( 8) With depth cameras each imaging camera (8) captures individually! the image according to its range angle (11), the internal computer (18) compares the image generated from each camera with the image (8) of the adjunct deíé mirtandô cameras. The Image overlay area, after determining the overlap areas, the internal computer (18) deletes these image areas, aligns and joins the adjacent images of each camera to create a panoramic image. Depth cameras (9) capture s digitize the distance from any environment to the cameras. Since they are in the same position, the Internal computer (18) with the aid of a clock (31) synchronizes the procedures of. image capture @ element distance scanning / Based on the overlapping area determined in the image capture, the internal computer (18) deletes the distance data for this area. A GPS global positioning system (30) is used to inform to the internal user (16) locating the image positioning camera (2). The Image positioning camera further comprises a pedestal (3) for height adjustment support and a sound pickup device (13) such as, for example, one or more microphones attached to the pedestal, each sound pickup device ( 1) captures the ambient sound vibrations individually synchronized by the internal computer (18) _. with depth cameras (9) and image _{(8). .} . ¾

23 - The area formed between the image positioning cameras 2} _{s is} delimited by the boundary perimeter of the movement environment (1). as Fi, 3¾ illustrate: f¾ 6 each ci ra d _; position: Imaging Ç2) captures soa im gem: pan, s data d © - ro || ndldade s vibration mode environment indivi UAI _are themselves © processed Peto internal computer (18) in each, these data can be stored in the storage memory (32) internal to each computer or sent in real time to a central computer interface melo for external communication (33), this communication can be done by cable .or via wireless communication,

24 - The internal computers (18) send via the external communication interface (33) the generated data to a central computer (35), illustrated in Fig. 10 consisting of: Image Positioning Camera Interface (42) be custom made or used conventional models such as USB or Firewire, a battery (36), RAM (37) _s watches (39) _? storage memory (40),. processor {41} and interface for external communication (38) can be any type of interface using cable communication or wireless communication. The Central Computer (35) merges the data received by the internal computers (18) into a single file, defines the Jiniite perimeter of the moving environment (1) by drawing a virtual perimeter line between the image positioning cameras (2). according to the information provided by

Camera rofurididadé - (9), s s ñoras vibrations are emlidai Give lanyard © 0: 5 POSITIONS namenl the device to ca: l ^ nor captures: (1.3) durante- of GFA _are ACFO wherein the scene of ioeaf somfá uptake becomes the focal point of sound fesio noamblentev ríuai

¾ - Occasionally the infrared signal from a depth camera (9) may interfere with the reception of another camera camera: {©) sj by iOi ό © ^' · intended for _. sun c tion this one. optionally after splitting the positioning ^, imaging, and m etations (¾ at the same time as before); iodine the system in a thesis mode, so the central computer (35) first drives an individual Image positioning camera (2) and then another and so on until all of them have recorded the test environment data, If this is true, the oscillator triggers all Image positioning cameras (2) at once »any interference reading or if | _If it does not coincide with the first work, it is terminated by the ceptral computation 3S. ! QPE and turn ab í¾ ap d ar¾o sincfQfíízaçio- give: ^■ êâmems ^of:; depth ($), ye ,. ^■ 8ssarn 'of- work Iron: Ada always up coi¾ ^{dr¾çençj'"the} minimum point, and nio be, for repro ercebidas the UCI rquivo Another example is only feasible ^ ck ULR below the emitter v¾rmelèO:. $ T i cameras ClaI (Ne e tne uir dli ossfeo ^ e

depth camera (9) as long as they reach the entire reach area in the positioning camera d Image (2) thus avoiding overlapping the infrared handle area. The central computer can be an external element, or one of the internal computers. It is one of the Image positioning cameras.

BMP, with a computer that is capable of decoding the information received from the motion sensor and transmitting it to the HMD web (14). When starting HMD (14) the computer program records the Inclination Initiate on the initial axes {23} of the sensor (14), from this moment on when the user tilts the head forms a new inclination for the axes (24) which consequently creates angles of the inclination difference for the

A new image is then reproduced on the user's screen after Tilt or Rotation (17), for positioning autonomy the depth contract (19) built into the ao or remote can be used as shown in Fig <8 with an on / off button ( 22) synchronization knob (20) to synchronize it to its respective HMD (14) and autonomy buttons (21) such as movement, volume, pause and the like control, when a remote control is used before playback starts. file type! virtual environment the user synchronizes his depth remote control (19) with his HIVID and can now disable file playback during playback ^■ when the user wants to change their virtual positioning within the environment by simply clicking the autonomy buttons . (2) ^! Since the data received by the serious control emitted to the HMÚ program by reiess communication which in turn will adjust the onrdiracionaí image (12) according to the new virtual positioning of the viewer, the adjustment of party sound reproduction by the program _s is The. from the sound proportionality lines of the axis "" (?) of the sound proportionality lines of the esxo y (6), which contract the sound reproduction intensity according to the virtual distance that the user is in relation to the sound emission locations. in the virtual environment, as the example in Hg illustrates,? of sound reproduction for the sound location in the indicated virtual environment (43).

Claims

mwmmcAÇÃQ

1) A panoramic image positioning camera comprising cameras parallel to depth cameras connected to a computer. Characterized by enabling the computer to form a panoramic image and position each pixel of that panoramic image according to the distance captured in the depth beam,

The method of claim 1 wherein there is one or more sound pickup devices supported on a pedestal or remote positioning chamber. Characterized by digitizing sound vibrations to the positioning camera's internal computer,

The method of claim 1, wherein the internal positioning computer comprises the image positioning camera comprising; processor, battery, RAM, storage memory ^ Camera interface, depth camera interface, sound device interface,

(4) The method of Indication 3 also comprises a clock, characterized in that it allows the synchronization of the capture of images with the depth of the environment, and also allows the alternation of infrared emission from the depth camera.

5) The method of claim 3 also comprising a positioning SYSTEM (GPS) _"wherein the computer possioitar define its own position within an _environment. relative to other external elements that also have GPS,

(6) a real scene data creation system, characterized by a set of image positioning cameras distributed to form an area between them,

7) the method of claim 6. _A computer containing a central composed of picture positioning the camera interface, battery, RAM, external communication interface, storage memory, clock and processor, characterized by receiving data each positioning camera, create a virtual environment from this data, sifKjonize the positioning cameras by alternating the emission of system infmemmes,

8) Reproduction of vlrl ai environment files on Head Mounted Display with motion sensor, characterized by matching the image transmitted on the Head Mounted Display screen to the movements performed by the user,

9} Playing environment files. Virtual TV in common _are characterized by adjusting the usual remote positioning commands and image direction, thereby enabling the common TV reproduces images of the virtual environment with a range of direction and positioning visyaltzaçio _*

10) Positioning control device for playback of virtual environment files with positioning autonomy, characterized by the fact that it allows the user a virtual movement within the environment created by commands contained on the remote control to the playback display;

11) Sound reproduction system in virtual environment, characterized by the fact that the system controls the sound emission by feeding or proportionally decreasing its intensity according to the user's posidonamenfo in relation to the sound emission.