US20020145660A1 - System and method for manipulating the point of interest in a sequence of images - Google Patents

System and method for manipulating the point of interest in a sequence of images Download PDF

Info

Publication number
US20020145660A1
US20020145660A1 US10/074,313 US7431302A US2002145660A1 US 20020145660 A1 US20020145660 A1 US 20020145660A1 US 7431302 A US7431302 A US 7431302A US 2002145660 A1 US2002145660 A1 US 2002145660A1
Authority
US
United States
Prior art keywords
image
point
camera systems
images
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US10/074,313
Other versions
US7106361B2 (en
Inventor
Takeo Kanade
Robert Collins
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Carnegie Mellon University
Original Assignee
Carnegie Mellon University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Carnegie Mellon University filed Critical Carnegie Mellon University
Priority to US10/074,313 priority Critical patent/US7106361B2/en
Assigned to CARNEGIE MELLON UNIVERSITY reassignment CARNEGIE MELLON UNIVERSITY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: COLLINS, ROBERT, KANADE, TAKEO
Publication of US20020145660A1 publication Critical patent/US20020145660A1/en
Application granted granted Critical
Publication of US7106361B2 publication Critical patent/US7106361B2/en
Adjusted expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2625Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect
    • H04N5/2627Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect for providing spin image effect, 3D stop motion effect or temporal freeze effect
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B41/00Special techniques not covered by groups G03B31/00 - G03B39/00; Apparatus therefor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/67Focus control based on electronic image sensor signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/695Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Definitions

  • the present invention relates generally to image and video processing.
  • fly-around effect For applications such as advertising, sports and entertainment, it is often desirable to take a set of images of an object from a large number of cameras that surround the object, and then play back those images in sequence to create an effect as if one is flying around the object.
  • This special effect is sometimes referred to as the “fly-around” effect.
  • a subset of the fly-around effect is when the displayed images are all from the same instant in time; this is sometimes referred to as the “3D stop-motion” effect. If the cameras are positioned in a closed-ended configuration, such as a circle or ellipse, the effect is sometimes referred to as the “spin-image” effect.
  • FIG. 1 illustrates one known technique for realizing this effect.
  • multiple cameras are set up in a ring, fixated on a single point of interest (PO) in space. Playing back one frame from each camera creates the appearance of spinning around the POI. Furthermore, playing back frames from a single time step, across all cameras, yields the appearance of freezing the action in time while a virtual camera spins around the frozen actor.
  • PO point of interest
  • the present invention is directed to a method of generating a video image sequence.
  • the method includes positioning a plurality of camera systems relative to a scene such that the camera systems define a gross trajectory.
  • the method further includes transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory.
  • the method includes displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
  • the present invention is directed to a system for generating a video image sequence of an object within a scene.
  • the system includes a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory and a video storage unit in communication with the camera systems.
  • the system also includes a frame-sequencing module in communication with the video storage unit.
  • the frame-sequencing module is for transforming images of the camera systems retrieved from the video storage unit to superimpose a secondary induced motion on the gross trajectory.
  • the system may also include means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed a target within the scene and a size of the target in the images from the camera systems is substantially the same over time.
  • the present invention is directed to a computer readable medium.
  • the computer readable medium has stored thereon instructions which, when executed by a processor, cause the processor to transform images from a plurality of camera systems positioned relative to a scene to define a gross trajectory to superimpose a secondary induced motion on the gross trajectory, and to output the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
  • FIG. 1 is a diagram illustrating a technique for achieving the so-called “fly-around” effect
  • FIG. 2 is a diagram illustrating a number of cameras arranged around a scene
  • FIG. 3 is a diagram illustrating the spatial “neighbor” relations for a closed-ended configuration of cameras
  • FIG. 4 is a diagram illustrating the spatial “neighbor” relations for an array configuration of cameras
  • FIG. 5 is a diagram illustrating a set of local camera motions superimposed on top of a gross camera trajectory
  • FIGS. 6 and 7 are diagrams of a system for generating a video image sequence of an object within a scene according to one embodiment of the present invention
  • FIG. 8 is a diagram illustrating the concept of correcting for physically misaligned camera systems using an embodiment of the present invention
  • FIG. 9 is a diagram illustrating the concept of changing the point of interest (POI) when the camera systems are fixated on a different point in space according to one embodiment of the present invention
  • FIG. 9 a is a diagram illustrating the concept of changing the point of interest (POI) when the camera systems are aimed on a different point in space, but physically misaligned, according to one embodiment of the present invention
  • FIG. 10 is a diagram illustrating the process flow through the master control unit of the systems of FIGS. 6 and 7 according to one embodiment of the present invention
  • FIG. 11 is a diagram illustrating a portion of the system of FIGS. 6 and 7 according to another embodiment of the present invention.
  • FIG. 12 is a diagram illustrating a portion of the system of FIGS. 6 and 7 according to another embodiment of the present invention.
  • FIG. 13 is a diagram illustrating the relationship between the principal viewing ray of the master camera system and the servo fixation point (SFP) according to one embodiment of the present invention.
  • FIG. 14 is a diagram illustrating the process flow through the image sequence generator according to one embodiment of the present invention.
  • the present invention concerns a technique for generating virtual camera trajectories and motions from the available camera views.
  • virtual refers to giving the viewer the appearance that he is looking at video from a single camera that is moving through the scene even though there is no such camera.
  • the technique concerns generating gross virtual camera trajectories with superimposed secondary induced motions that change the point in the scene at which the virtual camera appears to be looking.
  • the technique includes taking video from a set of cameras arranged relative to (such as surrounding) a dynamic, 3D scene, and to generate a new video corresponding to a smooth, virtual camera trajectory.
  • Two processes may be used to achieve this result. The first includes specification of the gross camera trajectory by selecting a sequence of neighboring physical cameras. Second, image transformations may be applied to video frames from these cameras to superimpose a secondary set of local camera motions on top of this trajectory, resulting in a video sequence that appears to have been taken from a camera undergoing smooth, continuous motion around the scene.
  • the gross trajectory aspect of the technique is described in conjunction with FIG. 2.
  • a set of cameras C 1 , C 2 , . . . Cn are arranged around a dynamic scene 2 .
  • the cameras C 1 . . . Cn may be, for example, static cameras or pan/tilt cameras.
  • the cameras C 1 . . . Cn may also have motorized zoom lenses that provide remote control of the field of view (zoom) and depth of field (focus).
  • Video from each camera C 1 . . . Cn may be composed of a set of video frames or images.
  • image (j,t) denote the video frame from camera Cj that is taken at time t.
  • the cameras may be provided a common genlock signal such that images indexed by time t are synchronized across all cameras to be taken at precisely the same time instant, i.e., t is a common temporal index into all camera videos.
  • All the cameras C 1 . . . Cn may be controlled to take video of interesting events that are visible to them, as described in more detail herein. This may involve active, remote control of their individual pan (P), tilt (T), zoom (Z) and focus (F) parameters in order to keep the desired object within their collective field of view. All of the video may be captured in a video storage device, as described in more detail hereinafter, in such a way that individual image frames I(j,t) can be efficiently retrieved either by camera number j (spatial retrieval) or by time step t (temporal retrieval).
  • a neighborhood topology may be defined across all the cameras C 1 . . . Cn encoding the notion of which cameras are spatial “neighbors.” This topology may be represented as a graph where cameras are nodes and “neighbor” relations are links between nodes, as illustrated in FIGS. 3 and 4.
  • FIG. 3 shows the spatial neighbor relationship for a ring of cameras
  • FIG. 4 shows the spatial neighbor relationship for an array of cameras.
  • spatial neighbors are shown with thin lines and the thick arrows represent sample trajectories.
  • a trajectory may be defined as a sequence of M camera indices (j( 1 ), j( 2 ), . . . , j(M)) defining a sequence of cameras (Cj( 1 ), Cj( 2 ), . . . Cj(M)) such that adjacent cameras are neighbors, i.e., there is a link between node Cj(k) and Cj(k+1) in the neighborhood topology graph.
  • a trajectory may be cyclic, in which case Cj(M) and Cj( 1 ) are neighbors.
  • a “freeze-frame” or “stop-motion” video By playing back a sequence of frames I (j(k), t) for a sequence of neighboring cameras with k start ⁇ k ⁇ k end , and for a constant t, a “freeze-frame” or “stop-motion” video can be created that shows a frozen moment in time viewed from what appears to be a camera moving spatially through the scene.
  • the trajectory may appear to be jumpy, depending upon the precision of the alignment of the cameras and the positioning of the cameras along the sequence trajectory.
  • a video showing moving events may be created, viewed from what appears to be a camera moving spatially through the scene.
  • the trajectory may appear to be jumpy due to camera misalignment and/or camera positioning (such as whether the cameras are evenly spaced along the trajectory).
  • the secondary induced motion aspect of the technique is described in conjunction with FIG. 5. As discussed previously, this aspect involves a superimposition of secondary, local camera motions on top of the gross trajectory.
  • Local camera motion may be parameterized by six parameters.
  • the three degrees of camera rotation may be specified by pitch, roll and yaw, and the three degrees of camera translation may be specified by displacement Tx, Ty and Tz, where Tz is directed along the central viewing ray of the camera, and Tx and Ty are perpendicular to the central viewing ray in the x and y directions respectively.
  • each transformation may be represented as a 2D homography, i.e., a 3 ⁇ 3 transformation matrix in homogenous 2D film plane coordinates.
  • the homography reduces either exactly or approximately to simpler image plane transforms such as similarity transformation (translation, rotation and scale), translation only, and scale only. These cases may be important if fast implementations are desired, such as to reduce the processing time needed to create a video sequence after an event has completed.
  • camera rotation induces a 2D homography in the image frame.
  • small corrections and adjustments of camera rotation may be performed after-the-fact, purely by applying a 2D image transformation to each video frame in a sequence.
  • corrections or adjustments to camera yaw and pitch can be specified by selecting one point correspondence, called the point of interest (POI). This defines a homography that brings the POI to the center of the image by simulating the effects of changing yaw and pitch.
  • POI point of interest
  • the yaw, pitch, roll and Tz homography of the prior statement can be approximated as the 2D similarity transformation, i.e., an image translation, rotation and isotropic scale.
  • corrections or adjustments to yaw, pitch, roll, Tx, Ty and Tz can be specified by selecting a three point correspondence, a point of interest (POI), a vertical unit point VI, and a translation point CO. These define a homography that brings the POI to point CO and maps VI one unit vertically above point CO, by simulating the effects of changing yaw, pitch and roll rotation angles, displacements Tx and Ty perpendicularly to the camera's central viewing ray, and displacement Tz along the camera's central viewing ray.
  • POI point of interest
  • the yaw, pitch, roll, Tx, Ty and Tz homography statement of the prior statement can be approximate as the 2D similarity transformation, i.e., an image translation, rotation and isotropic scale.
  • FIGS. 6 and 7 are block diagrams of a system 10 according to one embodiment.
  • the system 10 includes a number of camera systems positioned around the dynamic scene 12 .
  • the camera systems may be variable pointing camera systems including a master variable pointing camera system 14 and a number of slave variable camera systems 16 .
  • the variable pointing camera systems 14 , 16 may be, for example, pan/tilt camera systems, as explained further herein.
  • the camera systems 14 , 16 are sometimes referred to as pan/tilt camera systems 14 , 16 , although it should be recognized that the variable pointing camera systems 14 , 16 may be any camera system having the ability to point at different targets within the scene 12 .
  • the camera systems 14 , 16 may be fixed (i.e., nonvariable pointing) camera systems.
  • the master pan/tilt camera system 14 may include a video camera 18 and a pan/tilt device 20 for panning and tilting the camera 18 .
  • the slave pan/tilt camera systems 16 may include a video camera 18 and pan/tilt devices 20 .
  • the system 10 may include any number of camera systems 14 , 16 positioned around the scene, and the quantity of camera systems may be determined based upon the system requirements and applications. According to one embodiment, the camera systems 14 , 16 are equally spaced about the scene 12 . According to another embodiment, some or all of the camera systems 14 , 16 may be static (or fixed) cameras, i.e., cameras 18 with no pan/tilt device 20 .
  • the system 10 additionally includes a master control unit 24 in communication with the master camera system 14 .
  • the system 10 also includes a number of slave camera control units 26 in communication with the master control unit 24 by, for example, a computer network 28 such as, for example, a LAN.
  • Each slave camera control unit 26 is for controlling one or more slave camera systems 16 .
  • each slave camera control unit 26 is shown as being in communication with only one slave camera system 16 ; however, according to other embodiments, more than one slave camera system 16 may be in communication with one slave camera control unit 26 for the purpose of having that one slave camera control unit 26 control multiple slave camera systems 16 .
  • the master control unit 24 and the slave camera control units 26 may be implemented as computing devices such as, for example, a personal computer, a laptop computer, a workstation, a minicomputer, a mainframe or a supercomputer, depending upon the application requirements.
  • Each of the control units 24 , 26 may include a video storage unit 30 for storing digitized, time-stamped video image frames from the respective camera systems 14 , 16 .
  • the video storage units 30 may be, for example, DAT drives utilizing a Digital Video Data Storage (DVDS) format.
  • the system 10 may include analog-to-digital (A/D) converters 32 to convert the analog video from the cameras 18 to a digital format.
  • A/D analog-to-digital
  • the camera systems need not be in close proximity to their respective control units.
  • the slave camera systems 16 are shown as being in communication with their respective slave camera control units 26 via a fiber optic cable 34 .
  • the system 10 may include multiplexers/demultiplexers (MUX) 36 to multiplex and demultiplex the data onto and off of the fiber optic cables 34 .
  • MUX multiplexers/demultiplexers
  • the master camera system 14 is not illustrated as being in communication with the master control unit via a fiber optic cable, but according to other embodiments these components may be in communication via, for example, a fiber optic cable.
  • the master camera system 14 may be operated by an operator (not shown), which may be, for example, a human operator or a computer vision system, as described hereinafter. Accordingly, the operator may focus the master camera system 14 on the point of interest (or target) within the scene 12 . Parameters of the master camera system 14 are communicated to the master control unit 24 . According to one embodiment, the relevant parameters include pointing parameters, such as pan (P) and tilt (T) angles for the pan/tilt devices 20 , optical parameters, such as zoom (Z) and focus (F) parameters for the cameras 18 , and mechanical parameters, such as speed and accuracy.
  • pointing parameters such as pan (P) and tilt (T) angles for the pan/tilt devices 20
  • optical parameters such as zoom (Z) and focus (F) parameters for the cameras 18
  • mechanical parameters such as speed and accuracy.
  • These parameters may be digitally encoded by an encoder 38 and communicated to the master control unit 24 , such as by using a RS232 link 40 .
  • the relevant parameters are limited to pan, tilt, zoom and focus, although it should be recognized that other parameters might also be used by the system 10 .
  • the encoder 38 is sometimes referred to as the PTZF encoder 38 .
  • the master control unit 24 may also include a target determination module 42 and a slave control module 43 .
  • the modules 42 , 43 may be implemented as software code to be executed by the master control unit 24 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques.
  • the software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • the target determination module 42 reads the current PTZF parameters received from the master camera system 14 . Based on the pan/tilt angles, the target determination module 42 may compute the position of the desired target within the scene 12 , and based on the zoom and focus parameters the target determination 42 may compute the size of the target at the position in images from the master camera system 14 .
  • the slave control module 43 may compute the desired pan, tilt, zoom and focus parameters for each slave camera system 16 . As described further hereinbelow, this calculation may also be dependent on master/slave mapping -data, which may be ascertained during a calibration process.
  • the master/slave mapping data may be stored in a network database 50 , as illustrated in FIG. 7. According to another embodiment, the master/slave mapping data may be stored in a memory unit (not shown) of the master control unit 24 .
  • the parameters are communicated, via the network 28 , to the slave camera control units 26 that control the slave camera systems 16 . Commands may be sent from the master control unit 24 to each slave camera control unit 26 at a high update rate in order to be responsive to movements made by the operator of the master camera system 14 .
  • each slave camera control unit 26 includes a servo control module 44 .
  • the servo control modules 44 may be implemented as software code to be executed by the slave camera control units 26 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques.
  • the software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • the servo control modules 44 execute a servo control loop to compute commands to control the pan, tilt, zoom and focus of the slave camera systems 16 in order that the slave camera systems 16 may track the same target as the master camera system 14 and with the same focus to smoothly and accurately track the scene position designated by the master camera system 14 .
  • the PTZF commands for the slave camera systems 16 may be communicated from the slave camera control units 26 via, for example, the fiber optic cable 34 and RS- 232 links.
  • the pan and tilt commands may be input to the pan/tilt device 20 of the slave camera system 16 and the zoom/focus commands may be input to the camera 18 of the slave camera system 16 .
  • a 3 D servo-fixation point may be chosen, which is the desired target of each camera system 14 , 16 .
  • Each slave camera system 16 is then directed to view this fixation point.
  • each slave camera system 16 is controlled to continuously servo on the moving fixation point.
  • the zoom and focus of each slave camera system 16 is also controlled, based on their distance to the desired servo-fixation point.
  • the system 10 may include an image sequence generator 60 which may, according to one embodiment, be implemented by a computing device such as, for example, a personal computer, a laptop computer, a workstation, a minicomputer, a mainframe or a supercomputer, depending upon the application requirements.
  • the image sequence generator 60 may include a video reviewer interface module 62 and a frame-sequencing module 64 .
  • the modules 62 , 64 may be implemented as software code to be executed by a processor of the generator 60 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques.
  • the software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • a computer readable medium such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • Video from the master and slave camera systems may be continuously stored in the video storage units 30 .
  • the video storage units 30 may be such that the video frames are retrievable both spatially and temporally.
  • the video reviewer interface module 62 may be a graphic-based man-machine interface that provides continuous video from at least one of the camera systems 14 , 16 to a video review operator and which allows the video review operator to select a point in time in which to create a 3 D stop-motion video image sequence of the target.
  • the reviewer interface module 62 may also allow the reviewer to retrieve video frames temporally (i.e., sequential frames in time from a single camera system) or spatially (i.e., the same time frame, retrieved from a sequence of cameras).
  • the frame-sequencing module 64 may retrieve image frames from the video storage units 30 for certain (i.e., all or less than all) of the camera systems 14 , 16 and output images in a sequence corresponding to the position of the corresponding camera systems 14 , 16 .
  • the frame-sequencing module 64 may output images from each of the camera systems 14 , 16 corresponding to the position of the camera systems around the scene 12 , either clockwise or counter-clockwise, to generate the fly-around video image sequence.
  • each image may be from the same instant in time.
  • each camera system 14 , 16 may be synchronized to a common genlock signal, so that the shutter for each camera 18 fires at precisely the same time, resulting in video frames taken at the same time instant, thus heightening the apparent stop-motion effect.
  • the frame-sequencing module 64 may provide correction for misalignment of the cameras, as illustrated in FIG. 8, through the secondary, induced camera motion by allowing specification of the point of interest (POI) and vertical unit point V 1 , as described herein, for each frame of the sequence.
  • POI and vertical unit point VI may be specified by an operator, such as through the video reviewer interface 62 , by allowing the operator to click on the points with a mouse or stylus, or by entering character strings corresponding to the coordinates of these points in each frame.
  • these points may be specified by a software application.
  • the frame-sequencing module 64 may provide the ability to change the fixation point after the fact (i.e., after the images have been captured) to, for example, focus attention on other objects in the scene. This may be done whether or not the camera systems 14 , 16 are perfectly aligned by specification of the POI in each image frame, as illustrated in FIGS. 9 and 9 a.
  • simulating the effects of changing yaw, pitch and roll rotation angles, as well as displacements Tx, Ty, and Tz, in the image sequence may be realizable by the frame-sequencing module 64 through specification of the POI, vertical unit point V 1 , and translation point C 0 .
  • FIG. 10 is a diagram illustrating the process flow through the master control unit 24 according to one embodiment of the present invention.
  • the process initiates at block 70 where the master control unit 24 reads the pan, tilt, zoom and focus (PTZF) parameters of the master camera system 14 .
  • the target determination module 42 determines the position and size of the target. As described previously, the target determination module 42 may determine the position from the pan and tilt parameters and the size from the zoom and focus parameter.
  • the slave control module 43 may compute the PTZF parameters for each of the slave camera systems 16 based on the determined target position and size, and based on the master/slave mapping data as determined in the calibration process.
  • each camera system 14 , 16 may be calibrated so that its relationship to the scene 12 and to the other camera systems is known. According to one embodiment, this requires determining the pose (i.e., location and orientation) of each camera system 14 , 16 with respect to a scene coordinate system, determining the relationship of the zoom control parameter to angular field of view, and determining the relationship of the focus control parameter to the distance of objects in the scene.
  • Camera pose may be determined by measuring the pan/tilt angles toward a set of distinguished points or “landmarks” with known 3 D coordinates. “Sighting” the landmarks involves rotating the pan/tilt device from a user interface, until the landmark point is centered within the field of view of the camera. The pan/tilt parameters are the stored with the X,Y,Z coordinates of the landmark to form one pose calibration measurement.
  • Camera orientation and location can be determined by an optimization procedure, using three or more landmark measurements in a nondegenerate configuration.
  • These values can be measured directly and/or solved for using an optimization procedure based on more than three landmark measurements.
  • Computer control of motorized zoom lenses may involve sending commands to the camera system containing parameters specifying the desired zoom and focus. The effect of the value of these parameters on physical lens settings may be determined through calibration.
  • the zoom parameter may be calibrated by stepping through the allowable values and measuring the field of view after the motorized zoom is complete.
  • User control of the pan/tilt devices 20 can be used to actively and directly measure the field of view at each setting.
  • the focus parameter may be calibrated by focusing on objects at different distances from the camera systems 14 , 16 , and deriving either an explicit or implicit relationship between focus value and distance.
  • an implicit relationship can be determined using a lookup table of focus parameter settings, indexed by inverse distance to the desired focal distance in the scene. Focus to points at intermediate distances can be determined via interpolation of these stored table values.
  • the operator may select any camera system in the system 10 to act as a master camera system 14 .
  • the operator may change which camera system is the master camera system 14 at any time.
  • the cameraman may control the pan, tilt, zoom and focus of the master camera system 14 remotely through a remote operator interface unit 80 , as illustrated in FIG. 11.
  • the remote operator interface unit 80 may be implemented as a computing device such as, for example, a personal computer, a laptop computer or a workstation, providing a graphical user interface to allow the cameraman to specify the pan, tilt, zoom and focus parameter for the master camera system 14 .
  • a decoder 82 may decode these parameters for use by the master camera system 14 . These parameters may also be input to the master control unit 24 , either directly from the user interface, as illustrated in FIG. 11, or as feedback from the master camera system after it has executed a movement, as shown in FIG. 7.
  • FIG. 12 is a diagram of a portion of the system 10 according to such an embodiment.
  • the system 10 includes a computer vision control unit 84 for controlling the master camera system 14 .
  • the computer vision control unit 84 may be inplemented as a computing device such as, for example, a personal computer, a laptop computer or a workstation, configured with computer vision software that when executed by the computer vision control unit automatically detects and tracks moving objects in the scene 12 by processing video from the master camera system 14 .
  • the computer vision control unit 84 may receive the video from and be in communication with each camera system 14 , 16 , and may automatically select a different camera system to be the master computer system to decrease the distance to, or increase the visibility of, an object being tracked by the computer vision control unit 84 .
  • the desired servo-fixation point (SFP) for the spin-image effect is defined to be some point on the principal viewing ray 90 of the master camera system 14 . Choosing which point is the SFP is equivalent to choosing a value for parameter k in the above line equation.
  • the SFP may be determined by specifying k directly through a user interface such as, for example, the video reviewer interface 62 or the remote operator interface unit 80 . Note that k represents the distance or range of the desired SFP from the master camera system 14 . It may be selected using a one-degree of freedom mechanism, by the cameraman or a second operator.
  • the SFP may be determined by intersecting the principal-viewing ray 90 with an equation or set of equations representing a real surface of the scene 92 .
  • the real surface of the scene 92 may be approximately represented by the equation of a plane.
  • a more accurate approximation may be to represent the field by a nonplanar, triangulated mesh, or an explicit nonplanar surface equation.
  • the SFP may be determined by intersecting the principal-viewing ray 90 with an equation or set of equations representing a virtual (nonphysical) surface 94 in the scene. For example, it may be desirable to intersect the viewing ray 90 with a virtual surface 94 located a certain distance H, e.g. four feet, above the real surface of the scene 92 .
  • the SFP may be determined by intersecting the principal-viewing ray 90 with a set composed of any arbitrary combination real and virtual surfaces in the scene, for example the floor, walls and ceiling of a room.
  • the SFP is determined by intersecting the principal-viewing ray 90 with a surface or set of surfaces. Because there is more than one mathematical intersection point, various methods may be used to determine which point is the desired SFP. One such method is to always choose the intersection point that is closest to the master camera system 14 . If there is no mathematical intersection point, an alternate method must be used to determine the SFP. One example is to use the last known valid point of intersection.
  • the 3D position of the SFP is used to compute the pan and tilt angle parameters that bring the slave camera system principal-viewing ray 96 into alignment with the SFP. These values are used to command the pan/tilt device 20 of the respective slave camera systems 16 to move. After this movement, the SFP may appear in the center of the camera image.
  • the zoom of each slave camera system 16 may be controlled to keep the object of interest (a person, for example) substantially the same size in all the images (such as within error margins caused by servoing errors and misalignment), even though the slave camera systems 16 may be different distances away from the object.
  • r be the desired radius of a virtual sphere subtending the entire vertical field of view of each image.
  • d i be the distance from slave camera system 16 i to the SFP.
  • the zoom parameter that achieves this desired field of view is then computed by the servo control module 44 i from data collected during the prior zoom camera calibration procedure.
  • each slave camera system 16 may be controlled to achieve sharp focus at the SFP.
  • the focus parameter that achieves sharp focus at distance d i may be computed for slave camera system 16 i using the distance versus focus parameters equations or tables derived from the prior focus camera calibration procedure.
  • each servo control module 44 of the slave camera control units 26 may have to command the pan/tilt device 20 of the slave camera systems 16 as well as the camera/lens systems thereof at an even higher rate than it is receiving commands from the slave control module 43 of the master control unit. This may be achieved by interpolating between the last-received command and the current command, thereby controlling the pan, tilt, zoom and focus in smaller increments, more frequently.
  • the frame-sequencing module 64 may allow an operator to select, for example, the point of interest (POI), the vertical unit point V 1 , and/or the translation point C 0 for the sequence of images used in the image sequence. These points may be selected, for example, to correct for misalignment errors in the camera systems and/or select a POI that is different than the object on which the camera systems are servoing.
  • POI point of interest
  • V 1 the vertical unit point V 1
  • the translation point C 0 for the sequence of images used in the image sequence.
  • FIG. 14 is a diagram of a process flow through the image sequence generator 60 according to one embodiment in which the POI, the vertical unit point V 1 , and the translation point CO are specified for each image of the sequence to, for example, simulate the effects of changing yaw, pitch, roll and translation Tz along the cameras' central viewing rays.
  • the process flow illustrated in FIG. 14 is for the generation of a 3D stop-motion image sequence in which all of the displayed images are from the same instant in time (t).
  • the image sequence generator 60 may also output images from different time steps.
  • the process illustrated in FIG. 14 initiates at block 110 where the image sequence generator 60 reads the time (t) for which the 3D stop-motion image sequence is to be generated. As described previously, the video reviewer may specify this instant in time through the video reviewer interface module 62 .
  • the frame-sequencing module 64 may retrieve from the video storage units 30 the image (images I 1-N ) for all of the camera systems 14 , 16 to be used in the sequence, which may also be specified through the video reviewer interface module 62 and may be all or less than all of the camera systems.
  • the image sequence generator 60 may read the POI, the vertical unit point V 1 and the translation point C 0 for each image I 1-N .
  • an operator may enter the point for each image I 1-N through the video reviewer interface module 62 such as by, for example, clicking on the point in the image with a mouse or stylus.
  • the operator may enter a character string corresponding to the coordinates of the respective points for each image I 1-N through the video reviewer interface module 62 .
  • the frame-sequencing module 64 transforms the images I 1-N according to the homography defined by the POI, the V 1 , and the C 0 . That is, as described previously, the POI may be mapped to the point C 0 , and the vertical unit point V 1 may be mapped to one vertical unit above C 0 .
  • the frame-sequencing module 64 may output the transformed images I 1′-N′ in sequence corresponding to the order of the placement of the corresponding camera systems 14 , 16 around the scene 12 , either clockwise or counter-clockwise, to generate the video image sequence.
  • the frame-sequencing module 64 may also generate new (“virtual”) video images to “fill in” between the image frames from existing views to, for example, produce a new video image sequence that appears to be smoother and more continuous.
  • the new video images may be generated using induced camera motion, as described previously, to “interpolate” camera positions between the actual camera system locations. For example, according to one embodiment, a sequence of new images may be generated that smoothly varies the rotation and translation from one camera viewpoint into another, resulting in an apparent smooth motion of a single camera from the first position into the second.
  • the master control unit 24 and the slave camera control units 26 may be integrated into one computer device.
  • the master control unit 24 may therefore further include a servo control module 44 for computing the PTZF commands for each slave camera system 16 .
  • the image sequence generator 60 may be integrated with the computing device of the master control unit 24 , as may the remote operator interface unit 80 or the computer vision control unit 84 . According to another embodiment, the image sequence generator 60 may be distributed across more than one computing device. In addition, according to another embodiment, the slave control module 43 may be distributed among the slave camera control units 26 . According to such an embodiment, the appropriate master/slave mapping data may be stored in a memory unit of the slave camera control units 26 .
  • the system 10 may include a plurality of master camera systems 14 , each one controlling a subset of the slave camera systems 16 .
  • the system 10 may include a plurality of master control units 24 , one for each master camera system 14 .
  • each of the master control units 24 may be centralized in one computing device.
  • w be a 3D point in world coordinates
  • c be the location of the camera in that coordinate system
  • R be a rotation matrix representing the orientation of the camera (see FIG. A 1 ).
  • a local Cartesian camera coordinate system is specified by axes X-Y-Z, where the Z axis lies along the central viewing ray of the camera and X and Y are perpendicular to that ray.
  • film coordinates are related to image pixel coordinates through an affine transformation that is a function of linear camera intrinsic parameters, and also perhaps by an additional nonlinear transformation representing lens distortion.
  • Working in film coordinates amounts to assuming a calibrated camera, for which intrinsic parameters and lens distortion are known and invertible.
  • a 1 Camera Rotation Induces a 2D Homography in Film Coordinates.
  • A2 Change in Zoom Induces an Isotropic Sag of Film Coordinates.
  • A4 Translation Tx and Ty is Approximately a 2D Image Translation
  • A7 Specifying Yaw and Pitch Correction by Selecting One Point Correspondence.
  • the 2D homography H above can be simplified to a translation, rotation and isotropic scale, i.e. a single similarity transformation, leading to a great computational savings.
  • the derivation is similar to the case A8, above.
  • A11 Specifying Yaw, Pitch, Roll, Tx, Ty and Tz by Selecting Three Point Correspondences.
  • Tx and Ty are the correction for Tx and Ty.
  • A12 Efficient Approximation for Changing Yaw, Pitch, Roll, Tx, Ty and Tz
  • the 2D homography H above can be simplified to a translation, rotation and isotropic scale, i.e. a single similarity transformation, leading to a great computational savings.
  • the derivation is similar to cases A8 and A10, above.

Abstract

A method and a system of generating a video image sequence. According to one embodiment, the method includes positioning a plurality of camera systems relative to a scene such that the camera systems define a gross trajectory. The method further includes transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory. And the method includes displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • This application claims priority under 35 U.S.C. § 119 to U.S. provisional patent applications Serial No. 60/268,205 and Serial No. 60/268,206, both filed Feb. 12, 2001, which are incorporated herein by reference.[0001]
  • BACKGROUND OF THE INVENTION
  • 1. Field of the Invention [0002]
  • The present invention relates generally to image and video processing. [0003]
  • 2. Description of the Background [0004]
  • For applications such as advertising, sports and entertainment, it is often desirable to take a set of images of an object from a large number of cameras that surround the object, and then play back those images in sequence to create an effect as if one is flying around the object. This special effect is sometimes referred to as the “fly-around” effect. A subset of the fly-around effect is when the displayed images are all from the same instant in time; this is sometimes referred to as the “3D stop-motion” effect. If the cameras are positioned in a closed-ended configuration, such as a circle or ellipse, the effect is sometimes referred to as the “spin-image” effect. [0005]
  • FIG. 1 illustrates one known technique for realizing this effect. As illustrated in FIG. 1, multiple cameras are set up in a ring, fixated on a single point of interest (PO) in space. Playing back one frame from each camera creates the appearance of spinning around the POI. Furthermore, playing back frames from a single time step, across all cameras, yields the appearance of freezing the action in time while a virtual camera spins around the frozen actor. [0006]
  • The process of taking images for this purpose is tedious and costly. First, all cameras must be aligned with great precision so that their central viewing rays pass through the same POI on the object. Otherwise, the set of images when played back will appear bumpy and jittery. In addition, after the set of images are taken, one may want to alter the POI around which to create the fly-around effect. This typically involves reorienting the cameras and retaking a whole new set of images. These two difficulties are compounded when dealing with an unsupervised moving object or a dynamic scene (rather than an actor following instructions). There may not be time to align all of the cameras to satisfy the condition that all central rays intersect at the POI, and the object motion may not occur again in the same place. It may also not be possible to align some of the cameras with the POI due to constraints on their allowed motions. [0007]
  • BRIEF SUMMARY OF THE INVENTION
  • In one general respect, the present invention is directed to a method of generating a video image sequence. According to one embodiment, the method includes positioning a plurality of camera systems relative to a scene such that the camera systems define a gross trajectory. The method further includes transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory. And the method includes displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory. [0008]
  • In another general respect, the present invention is directed to a system for generating a video image sequence of an object within a scene. According to one embodiment, the system includes a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory and a video storage unit in communication with the camera systems. The system also includes a frame-sequencing module in communication with the video storage unit. The frame-sequencing module is for transforming images of the camera systems retrieved from the video storage unit to superimpose a secondary induced motion on the gross trajectory. According to another embodiment, the system may also include means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed a target within the scene and a size of the target in the images from the camera systems is substantially the same over time. [0009]
  • In another general respect, the present invention is directed to a computer readable medium. The computer readable medium has stored thereon instructions which, when executed by a processor, cause the processor to transform images from a plurality of camera systems positioned relative to a scene to define a gross trajectory to superimpose a secondary induced motion on the gross trajectory, and to output the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory. [0010]
  • DESCRIPTION OF THE FIGURES
  • Embodiments of the present invention are described in conjunction with the following figures, wherein: [0011]
  • FIG. 1 is a diagram illustrating a technique for achieving the so-called “fly-around” effect; [0012]
  • FIG. 2 is a diagram illustrating a number of cameras arranged around a scene; [0013]
  • FIG. 3 is a diagram illustrating the spatial “neighbor” relations for a closed-ended configuration of cameras; [0014]
  • FIG. 4 is a diagram illustrating the spatial “neighbor” relations for an array configuration of cameras; [0015]
  • FIG. 5 is a diagram illustrating a set of local camera motions superimposed on top of a gross camera trajectory; [0016]
  • FIGS. 6 and 7 are diagrams of a system for generating a video image sequence of an object within a scene according to one embodiment of the present invention; [0017]
  • FIG. 8 is a diagram illustrating the concept of correcting for physically misaligned camera systems using an embodiment of the present invention; [0018]
  • FIG. 9 is a diagram illustrating the concept of changing the point of interest (POI) when the camera systems are fixated on a different point in space according to one embodiment of the present invention; [0019]
  • FIG. 9[0020] a is a diagram illustrating the concept of changing the point of interest (POI) when the camera systems are aimed on a different point in space, but physically misaligned, according to one embodiment of the present invention; FIG. 10 is a diagram illustrating the process flow through the master control unit of the systems of FIGS. 6 and 7 according to one embodiment of the present invention;
  • FIG. 11 is a diagram illustrating a portion of the system of FIGS. 6 and 7 according to another embodiment of the present invention; [0021]
  • FIG. 12 is a diagram illustrating a portion of the system of FIGS. 6 and 7 according to another embodiment of the present invention; [0022]
  • FIG. 13 is a diagram illustrating the relationship between the principal viewing ray of the master camera system and the servo fixation point (SFP) according to one embodiment of the present invention; and [0023]
  • FIG. 14 is a diagram illustrating the process flow through the image sequence generator according to one embodiment of the present invention.[0024]
  • DETAILED DESCRIPTION OF THE INVENTION
  • In one general respect, the present invention concerns a technique for generating virtual camera trajectories and motions from the available camera views. The term “virtual” refers to giving the viewer the appearance that he is looking at video from a single camera that is moving through the scene even though there is no such camera. In particular, the technique concerns generating gross virtual camera trajectories with superimposed secondary induced motions that change the point in the scene at which the virtual camera appears to be looking. [0025]
  • According to one embodiment, the technique includes taking video from a set of cameras arranged relative to (such as surrounding) a dynamic, 3D scene, and to generate a new video corresponding to a smooth, virtual camera trajectory. Two processes may be used to achieve this result. The first includes specification of the gross camera trajectory by selecting a sequence of neighboring physical cameras. Second, image transformations may be applied to video frames from these cameras to superimpose a secondary set of local camera motions on top of this trajectory, resulting in a video sequence that appears to have been taken from a camera undergoing smooth, continuous motion around the scene. [0026]
  • The gross trajectory aspect of the technique is described in conjunction with FIG. 2. As illustrated in FIG. 2, a set of cameras C[0027] 1, C2, . . . Cn are arranged around a dynamic scene 2. The cameras C1 . . . Cn may be, for example, static cameras or pan/tilt cameras. The cameras C1 . . . Cn may also have motorized zoom lenses that provide remote control of the field of view (zoom) and depth of field (focus).
  • Video from each camera C[0028] 1 . . . Cn may be composed of a set of video frames or images. Let image (j,t) denote the video frame from camera Cj that is taken at time t. To facilitate the cameras C1 . . . Cn taking images at the same instances in time, the cameras may be provided a common genlock signal such that images indexed by time t are synchronized across all cameras to be taken at precisely the same time instant, i.e., t is a common temporal index into all camera videos.
  • All the cameras C[0029] 1 . . . Cn may be controlled to take video of interesting events that are visible to them, as described in more detail herein. This may involve active, remote control of their individual pan (P), tilt (T), zoom (Z) and focus (F) parameters in order to keep the desired object within their collective field of view. All of the video may be captured in a video storage device, as described in more detail hereinafter, in such a way that individual image frames I(j,t) can be efficiently retrieved either by camera number j (spatial retrieval) or by time step t (temporal retrieval).
  • A neighborhood topology may be defined across all the cameras C[0030] 1 . . . Cn encoding the notion of which cameras are spatial “neighbors.” This topology may be represented as a graph where cameras are nodes and “neighbor” relations are links between nodes, as illustrated in FIGS. 3 and 4. FIG. 3 shows the spatial neighbor relationship for a ring of cameras, and FIG. 4 shows the spatial neighbor relationship for an array of cameras. In FIG. 4, spatial neighbors are shown with thin lines and the thick arrows represent sample trajectories.
  • A trajectory may be defined as a sequence of M camera indices (j([0031] 1), j(2), . . . , j(M)) defining a sequence of cameras (Cj(1), Cj(2), . . . Cj(M)) such that adjacent cameras are neighbors, i.e., there is a link between node Cj(k) and Cj(k+1) in the neighborhood topology graph. A trajectory may be cyclic, in which case Cj(M) and Cj(1) are neighbors.
  • By playing back a sequence of frames I(j, t(k)) with constant camera index j and a sequence of times t(k) such that t[0032] start≦k≦tend, a standard video subsequence can be created from a particular camera.
  • By playing back a sequence of frames I (j(k), t) for a sequence of neighboring cameras with k[0033] start≦k≦kend, and for a constant t, a “freeze-frame” or “stop-motion” video can be created that shows a frozen moment in time viewed from what appears to be a camera moving spatially through the scene. The trajectory may appear to be jumpy, depending upon the precision of the alignment of the cameras and the positioning of the cameras along the sequence trajectory.
  • By playing back a sequence of frames I(j(k), t(p)) for a sequence of neighboring cameras with k[0034] start≦k≦kend, and for a sequence of time t(p) with tstart≦p≦tend, a video showing moving events may be created, viewed from what appears to be a camera moving spatially through the scene. Again, the trajectory may appear to be jumpy due to camera misalignment and/or camera positioning (such as whether the cameras are evenly spaced along the trajectory).
  • The secondary induced motion aspect of the technique is described in conjunction with FIG. 5. As discussed previously, this aspect involves a superimposition of secondary, local camera motions on top of the gross trajectory. Local camera motion may be parameterized by six parameters. The three degrees of camera rotation may be specified by pitch, roll and yaw, and the three degrees of camera translation may be specified by displacement Tx, Ty and Tz, where Tz is directed along the central viewing ray of the camera, and Tx and Ty are perpendicular to the central viewing ray in the x and y directions respectively. [0035]
  • These local motions may be induced by purely 2D image transformations that require no knowledge of the 3D scene structure. In general, each transformation may be represented as a 2D homography, i.e., a 3×3 transformation matrix in homogenous 2D film plane coordinates. In some cases the homography reduces either exactly or approximately to simpler image plane transforms such as similarity transformation (translation, rotation and scale), translation only, and scale only. These cases may be important if fast implementations are desired, such as to reduce the processing time needed to create a video sequence after an event has completed. [0036]
  • Several goals are achievable by applying 2D transforms that simulate local camera motions. As a first example, misalignment errors between multiple cameras can be corrected so that they appear to fixate precisely on a given POI in the video sequence, even though they did not all point at a single 3D point in the scene. A second example is that new points of interest may be chosen for camera fixation, even though they were not anticipated during recording of the event. Third, additional frames can be generated between actual image frames from a sparse set of cameras along the gross trajectory, thereby transforming a jumpy video sequence into one that appears to be smooth and continuous. [0037]
  • Provided below are some aspects of the secondary induced motion aspect of the technique. Mathematical justification for each of these points is included in the Appendix attached hereto. In the Appendix, mathematical justification for the first point listed below is provided at the heading denoted A1, mathematical justification for the second point is provided at A2, and so on. [0038]
  • First, camera rotation induces a 2D homography in the image frame. As a result, small corrections and adjustments of camera rotation may be performed after-the-fact, purely by applying a 2D image transformation to each video frame in a sequence. [0039]
  • Second, change in camera zoom induces an isotropic scaling of the image frame. As a result, corrections or adjustments of camera zoom may be performed after-the-fact, purely by applying a 2D image transformation to each video frame in a sequence. [0040]
  • Third, a small translation along the camera's central viewing ray approximately induces an isotropic scaling in the image frame. Particularly, this is a good approximation when an object being fixated on is “shallow” in depth, i.e., the range of z values across the object is small with respect to the mean distance z from the camera. [0041]
  • Fourth, a small translation perpendicular to the camera's central viewing ray is approximately a 2D image translation. As in the prior statement, this approximation may be appropriate when fixating on “shallow” objects. [0042]
  • Fifth, 2D image transformations corresponding to multiple changes of camera rotation or translation can be composed into a single, new 2D transformation. [0043]
  • Sixth, small, local changes in camera rotation and translation can be parameterized by six parameters: pitch, roll, and yaw angles for rotation, and Tx, Ty and Tz displacements perpendicular (Tx and Ty) and parallel (Tz) to the camera's central viewing ray. [0044]
  • Seven, corrections or adjustments to camera yaw and pitch can be specified by selecting one point correspondence, called the point of interest (POI). This defines a homography that brings the POI to the center of the image by simulating the effects of changing yaw and pitch. [0045]
  • Eight, for cameras at high zoom viewing distant objects, the yaw and pitch homography of the prior statement can be approximated as the 2D image translation. [0046]
  • Nine, corrections or adjustments to yaw, pitch, roll and Tz can be specified by selecting two point correspondences, a point of interest (POI) and a vertical unit point VI. These define a homography that may, for example, bring the POI to the center of the image and map Vi one unit vertically above the center of the image, by simulating the effects of changing yaw, pitch, roll and translation Tz along the camera's central viewing ray. Other vertical unit points may be defined instead such as, for example, two vertical unit points above, one vertical unit point below, etc. [0047]
  • Ten, for cameras at high zoom viewing distant objects, the yaw, pitch, roll and Tz homography of the prior statement can be approximated as the 2D similarity transformation, i.e., an image translation, rotation and isotropic scale. [0048]
  • Eleven, corrections or adjustments to yaw, pitch, roll, Tx, Ty and Tz can be specified by selecting a three point correspondence, a point of interest (POI), a vertical unit point VI, and a translation point CO. These define a homography that brings the POI to point CO and maps VI one unit vertically above point CO, by simulating the effects of changing yaw, pitch and roll rotation angles, displacements Tx and Ty perpendicularly to the camera's central viewing ray, and displacement Tz along the camera's central viewing ray. [0049]
  • Twelve, for cameras at high zoom viewing distant objects, the yaw, pitch, roll, Tx, Ty and Tz homography statement of the prior statement can be approximate as the 2D similarity transformation, i.e., an image translation, rotation and isotropic scale. [0050]
  • The techniques described herein can be used in many applications. One application is to provide spin-image stabilization in systems designed to generate a “spin-image” video sequence. FIGS. 6 and 7 are block diagrams of a [0051] system 10 according to one embodiment. The system 10 includes a number of camera systems positioned around the dynamic scene 12. According to one embodiment, the camera systems may be variable pointing camera systems including a master variable pointing camera system 14 and a number of slave variable camera systems 16. According to one embodiment, the variable pointing camera systems 14, 16 may be, for example, pan/tilt camera systems, as explained further herein. For purposes of convenience in the description to follow, the camera systems 14, 16 are sometimes referred to as pan/ tilt camera systems 14, 16, although it should be recognized that the variable pointing camera systems 14, 16 may be any camera system having the ability to point at different targets within the scene 12. In addition, according to another embodiment, as described further herein, the camera systems 14, 16 may be fixed (i.e., nonvariable pointing) camera systems.
  • The master pan/[0052] tilt camera system 14 may include a video camera 18 and a pan/tilt device 20 for panning and tilting the camera 18. Similarly, the slave pan/tilt camera systems 16 may include a video camera 18 and pan/tilt devices 20. The system 10 may include any number of camera systems 14, 16 positioned around the scene, and the quantity of camera systems may be determined based upon the system requirements and applications. According to one embodiment, the camera systems 14, 16 are equally spaced about the scene 12. According to another embodiment, some or all of the camera systems 14, 16 may be static (or fixed) cameras, i.e., cameras 18 with no pan/tilt device 20.
  • As illustrated in FIGS. 6 and 7, the [0053] system 10 additionally includes a master control unit 24 in communication with the master camera system 14. The system 10 also includes a number of slave camera control units 26 in communication with the master control unit 24 by, for example, a computer network 28 such as, for example, a LAN. Each slave camera control unit 26 is for controlling one or more slave camera systems 16. For purposes of simplicity, in FIGS. 6 and 7 each slave camera control unit 26 is shown as being in communication with only one slave camera system 16; however, according to other embodiments, more than one slave camera system 16 may be in communication with one slave camera control unit 26 for the purpose of having that one slave camera control unit 26 control multiple slave camera systems 16.
  • The [0054] master control unit 24 and the slave camera control units 26 may be implemented as computing devices such as, for example, a personal computer, a laptop computer, a workstation, a minicomputer, a mainframe or a supercomputer, depending upon the application requirements. Each of the control units 24, 26 may include a video storage unit 30 for storing digitized, time-stamped video image frames from the respective camera systems 14, 16. The video storage units 30 may be, for example, DAT drives utilizing a Digital Video Data Storage (DVDS) format. For an embodiment where the cameras 18 are not digital video cameras, the system 10 may include analog-to-digital (A/D) converters 32 to convert the analog video from the cameras 18 to a digital format.
  • The camera systems need not be in close proximity to their respective control units. For example, in FIG. 7 the [0055] slave camera systems 16 are shown as being in communication with their respective slave camera control units 26 via a fiber optic cable 34. For such an embodiment, the system 10 may include multiplexers/demultiplexers (MUX) 36 to multiplex and demultiplex the data onto and off of the fiber optic cables 34. In FIG. 7 the master camera system 14 is not illustrated as being in communication with the master control unit via a fiber optic cable, but according to other embodiments these components may be in communication via, for example, a fiber optic cable.
  • The [0056] master camera system 14 may be operated by an operator (not shown), which may be, for example, a human operator or a computer vision system, as described hereinafter. Accordingly, the operator may focus the master camera system 14 on the point of interest (or target) within the scene 12. Parameters of the master camera system 14 are communicated to the master control unit 24. According to one embodiment, the relevant parameters include pointing parameters, such as pan (P) and tilt (T) angles for the pan/tilt devices 20, optical parameters, such as zoom (Z) and focus (F) parameters for the cameras 18, and mechanical parameters, such as speed and accuracy. These parameters may be digitally encoded by an encoder 38 and communicated to the master control unit 24, such as by using a RS232 link 40. For purposes of convenience in the description to follow, the relevant parameters are limited to pan, tilt, zoom and focus, although it should be recognized that other parameters might also be used by the system 10. Also, hereinafter the encoder 38 is sometimes referred to as the PTZF encoder 38.
  • As illustrated in FIG. 7, the [0057] master control unit 24 may also include a target determination module 42 and a slave control module 43. The modules 42, 43 may be implemented as software code to be executed by the master control unit 24 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • The [0058] target determination module 42 reads the current PTZF parameters received from the master camera system 14. Based on the pan/tilt angles, the target determination module 42 may compute the position of the desired target within the scene 12, and based on the zoom and focus parameters the target determination 42 may compute the size of the target at the position in images from the master camera system 14.
  • Based on the determined target position and size, the slave control module [0059] 43 may compute the desired pan, tilt, zoom and focus parameters for each slave camera system 16. As described further hereinbelow, this calculation may also be dependent on master/slave mapping -data, which may be ascertained during a calibration process. The master/slave mapping data may be stored in a network database 50, as illustrated in FIG. 7. According to another embodiment, the master/slave mapping data may be stored in a memory unit (not shown) of the master control unit 24. Once computed by the slave control module 42, the parameters are communicated, via the network 28, to the slave camera control units 26 that control the slave camera systems 16. Commands may be sent from the master control unit 24 to each slave camera control unit 26 at a high update rate in order to be responsive to movements made by the operator of the master camera system 14.
  • Also, as illustrated in FIG. 7, each slave [0060] camera control unit 26 includes a servo control module 44. The servo control modules 44 may be implemented as software code to be executed by the slave camera control units 26 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • Based on the PTZF parameters received from the slave control unit [0061] 43, the servo control modules 44 execute a servo control loop to compute commands to control the pan, tilt, zoom and focus of the slave camera systems 16 in order that the slave camera systems 16 may track the same target as the master camera system 14 and with the same focus to smoothly and accurately track the scene position designated by the master camera system 14. The PTZF commands for the slave camera systems 16 may be communicated from the slave camera control units 26 via, for example, the fiber optic cable 34 and RS-232 links. The pan and tilt commands may be input to the pan/tilt device 20 of the slave camera system 16 and the zoom/focus commands may be input to the camera 18 of the slave camera system 16.
  • Thus, according to one embodiment, based on feedback from the [0062] master camera system 14 and knowledge of the geometry of the scene, a 3D servo-fixation point may be chosen, which is the desired target of each camera system 14, 16. Each slave camera system 16 is then directed to view this fixation point. As the operator moves the master camera system 14 in real-time, each slave camera system 16 is controlled to continuously servo on the moving fixation point. The zoom and focus of each slave camera system 16 is also controlled, based on their distance to the desired servo-fixation point.
  • Also, as illustrated in FIGS. 6 and 7, the [0063] system 10 may include an image sequence generator 60 which may, according to one embodiment, be implemented by a computing device such as, for example, a personal computer, a laptop computer, a workstation, a minicomputer, a mainframe or a supercomputer, depending upon the application requirements. The image sequence generator 60 may include a video reviewer interface module 62 and a frame-sequencing module 64. The modules 62, 64 may be implemented as software code to be executed by a processor of the generator 60 using any suitable computer language such as, for example, Java, C or C++ using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM.
  • Video from the master and slave camera systems may be continuously stored in the [0064] video storage units 30. As described previously, the video storage units 30 may be such that the video frames are retrievable both spatially and temporally. The video reviewer interface module 62 may be a graphic-based man-machine interface that provides continuous video from at least one of the camera systems 14, 16 to a video review operator and which allows the video review operator to select a point in time in which to create a 3D stop-motion video image sequence of the target. The reviewer interface module 62 may also allow the reviewer to retrieve video frames temporally (i.e., sequential frames in time from a single camera system) or spatially (i.e., the same time frame, retrieved from a sequence of cameras).
  • The frame-sequencing [0065] module 64 may retrieve image frames from the video storage units 30 for certain (i.e., all or less than all) of the camera systems 14, 16 and output images in a sequence corresponding to the position of the corresponding camera systems 14, 16. For example, the frame-sequencing module 64 may output images from each of the camera systems 14, 16 corresponding to the position of the camera systems around the scene 12, either clockwise or counter-clockwise, to generate the fly-around video image sequence. To generate a 3D stop-motion image sequence, each image may be from the same instant in time. For that and similar time-dependent purposes, each camera system 14, 16 may be synchronized to a common genlock signal, so that the shutter for each camera 18 fires at precisely the same time, resulting in video frames taken at the same time instant, thus heightening the apparent stop-motion effect.
  • For such a system, the gross trajectory is predefined by the cyclic neighborhood topology of the [0066] camera systems 14, 16. Accordingly, the frame-sequencing module 64 may provide correction for misalignment of the cameras, as illustrated in FIG. 8, through the secondary, induced camera motion by allowing specification of the point of interest (POI) and vertical unit point V1, as described herein, for each frame of the sequence. The POI and vertical unit point VI may be specified by an operator, such as through the video reviewer interface 62, by allowing the operator to click on the points with a mouse or stylus, or by entering character strings corresponding to the coordinates of these points in each frame. According to another embodiment, these points may be specified by a software application.
  • According to another embodiment, the frame-sequencing [0067] module 64 may provide the ability to change the fixation point after the fact (i.e., after the images have been captured) to, for example, focus attention on other objects in the scene. This may be done whether or not the camera systems 14, 16 are perfectly aligned by specification of the POI in each image frame, as illustrated in FIGS. 9 and 9a.
  • In addition, according to another embodiment, simulating the effects of changing yaw, pitch and roll rotation angles, as well as displacements Tx, Ty, and Tz, in the image sequence may be realizable by the frame-sequencing [0068] module 64 through specification of the POI, vertical unit point V1, and translation point C0.
  • FIG. 10 is a diagram illustrating the process flow through the [0069] master control unit 24 according to one embodiment of the present invention. The process initiates at block 70 where the master control unit 24 reads the pan, tilt, zoom and focus (PTZF) parameters of the master camera system 14. Next, at block 72, the target determination module 42 determines the position and size of the target. As described previously, the target determination module 42 may determine the position from the pan and tilt parameters and the size from the zoom and focus parameter. Next, at block 74, the slave control module 43 may compute the PTZF parameters for each of the slave camera systems 16 based on the determined target position and size, and based on the master/slave mapping data as determined in the calibration process.
  • Before operation of the [0070] system 10, each camera system 14, 16 may be calibrated so that its relationship to the scene 12 and to the other camera systems is known. According to one embodiment, this requires determining the pose (i.e., location and orientation) of each camera system 14, 16 with respect to a scene coordinate system, determining the relationship of the zoom control parameter to angular field of view, and determining the relationship of the focus control parameter to the distance of objects in the scene.
  • Camera pose may be determined by measuring the pan/tilt angles toward a set of distinguished points or “landmarks” with known [0071] 3D coordinates. “Sighting” the landmarks involves rotating the pan/tilt device from a user interface, until the landmark point is centered within the field of view of the camera. The pan/tilt parameters are the stored with the X,Y,Z coordinates of the landmark to form one pose calibration measurement.
  • Camera orientation and location can be determined by an optimization procedure, using three or more landmark measurements in a nondegenerate configuration. For high-precision pointing, it may also be necessary to measure the pitch and yaw of the sensor as mounted on the pan/[0072] tilt devices 20, and the offset of the sensor focal point from the center of rotation of the pan/tilt device 20. These values can be measured directly and/or solved for using an optimization procedure based on more than three landmark measurements.
  • Computer control of motorized zoom lenses may involve sending commands to the camera system containing parameters specifying the desired zoom and focus. The effect of the value of these parameters on physical lens settings may be determined through calibration. The zoom parameter may be calibrated by stepping through the allowable values and measuring the field of view after the motorized zoom is complete. User control of the pan/[0073] tilt devices 20 can be used to actively and directly measure the field of view at each setting.
  • The focus parameter may be calibrated by focusing on objects at different distances from the [0074] camera systems 14, 16, and deriving either an explicit or implicit relationship between focus value and distance. For example, an implicit relationship can be determined using a lookup table of focus parameter settings, indexed by inverse distance to the desired focal distance in the scene. Focus to points at intermediate distances can be determined via interpolation of these stored table values.
  • During system operation, the operator may select any camera system in the [0075] system 10 to act as a master camera system 14. According to one embodiment, the operator may change which camera system is the master camera system 14 at any time.
  • For an embodiment in which the operator of the [0076] master camera system 14 is a human operator, i.e., a “cameraman,” the cameraman may control the pan, tilt, zoom and focus of the master camera system 14 remotely through a remote operator interface unit 80, as illustrated in FIG. 11. The remote operator interface unit 80 may be implemented as a computing device such as, for example, a personal computer, a laptop computer or a workstation, providing a graphical user interface to allow the cameraman to specify the pan, tilt, zoom and focus parameter for the master camera system 14. A decoder 82 may decode these parameters for use by the master camera system 14. These parameters may also be input to the master control unit 24, either directly from the user interface, as illustrated in FIG. 11, or as feedback from the master camera system after it has executed a movement, as shown in FIG. 7.
  • As described previously, the operator of the [0077] master camera system 14 may also be a computer vision application. FIG. 12 is a diagram of a portion of the system 10 according to such an embodiment. As illustrated in FIG. 12, the system 10 includes a computer vision control unit 84 for controlling the master camera system 14. The computer vision control unit 84 may be inplemented as a computing device such as, for example, a personal computer, a laptop computer or a workstation, configured with computer vision software that when executed by the computer vision control unit automatically detects and tracks moving objects in the scene 12 by processing video from the master camera system 14. According to another embodiment, the computer vision control unit 84 may receive the video from and be in communication with each camera system 14, 16, and may automatically select a different camera system to be the master computer system to decrease the distance to, or increase the visibility of, an object being tracked by the computer vision control unit 84.
  • With reference to FIG. 13, based on the pan/tilt angle parameters from the [0078] master camera system 14, the master control unit 24 may determine the equation of a 3D line specifying the principal-viewing ray 90 of the master camera system 14. All points on this line can be represented as {right arrow over (p)}={right arrow over (c)}+k{right arrow over (v)}, where {right arrow over (p)} is a 3D point on the line, {right arrow over (c)} is the focal point of the master camera system, v is a unit vector representing the orientation of the principal axis, directed out from the focal point, and k is a scalar parameter that selects different points on the line. Only points on the line that are in front of the focal point (i.e., k>0) are considered to be on the master camera system principal viewing ray 90.
  • The desired servo-fixation point (SFP) for the spin-image effect is defined to be some point on the [0079] principal viewing ray 90 of the master camera system 14. Choosing which point is the SFP is equivalent to choosing a value for parameter k in the above line equation. The SFP may be determined by specifying k directly through a user interface such as, for example, the video reviewer interface 62 or the remote operator interface unit 80. Note that k represents the distance or range of the desired SFP from the master camera system 14. It may be selected using a one-degree of freedom mechanism, by the cameraman or a second operator. According to one embodiment, the SFP may be determined by intersecting the principal-viewing ray 90 with an equation or set of equations representing a real surface of the scene 92. For example, the real surface of the scene 92 may be approximately represented by the equation of a plane. Alternatively, a more accurate approximation may be to represent the field by a nonplanar, triangulated mesh, or an explicit nonplanar surface equation.
  • Similarly, the SFP may be determined by intersecting the principal-[0080] viewing ray 90 with an equation or set of equations representing a virtual (nonphysical) surface 94 in the scene. For example, it may be desirable to intersect the viewing ray 90 with a virtual surface 94 located a certain distance H, e.g. four feet, above the real surface of the scene 92. According to another embodiment, the SFP may be determined by intersecting the principal-viewing ray 90 with a set composed of any arbitrary combination real and virtual surfaces in the scene, for example the floor, walls and ceiling of a room.
  • If the SFP is determined by intersecting the principal-[0081] viewing ray 90 with a surface or set of surfaces. Because there is more than one mathematical intersection point, various methods may be used to determine which point is the desired SFP. One such method is to always choose the intersection point that is closest to the master camera system 14. If there is no mathematical intersection point, an alternate method must be used to determine the SFP. One example is to use the last known valid point of intersection.
  • For each slave camera system, the 3D position of the SFP is used to compute the pan and tilt angle parameters that bring the slave camera system principal-viewing ray [0082] 96 into alignment with the SFP. These values are used to command the pan/tilt device 20 of the respective slave camera systems 16 to move. After this movement, the SFP may appear in the center of the camera image.
  • The distance d between a slave camera system position {right arrow over (c)} and SFP {right arrow over (x)} may be computed. Let vector (a,b,)={right arrow over (x)}−{right arrow over (c)}. Then d may be computed as d={square root}{square root over (a[0083] 2+b2+c2)}.
  • The zoom of each [0084] slave camera system 16 may be controlled to keep the object of interest (a person, for example) substantially the same size in all the images (such as within error margins caused by servoing errors and misalignment), even though the slave camera systems 16 may be different distances away from the object. Let r be the desired radius of a virtual sphere subtending the entire vertical field of view of each image. Let di be the distance from slave camera system 16 i to the SFP. The desired vertical field of view angle αi can be computed as αi=2*arctan(r/di). The zoom parameter that achieves this desired field of view is then computed by the servo control module 44 i from data collected during the prior zoom camera calibration procedure.
  • The focus of each [0085] slave camera system 16 may be controlled to achieve sharp focus at the SFP. The focus parameter that achieves sharp focus at distance di may be computed for slave camera system 16 i using the distance versus focus parameters equations or tables derived from the prior focus camera calibration procedure.
  • According to another embodiment, in order to achieve smooth motion, each [0086] servo control module 44 of the slave camera control units 26 may have to command the pan/tilt device 20 of the slave camera systems 16 as well as the camera/lens systems thereof at an even higher rate than it is receiving commands from the slave control module 43 of the master control unit. This may be achieved by interpolating between the last-received command and the current command, thereby controlling the pan, tilt, zoom and focus in smaller increments, more frequently.
  • As mentioned previously, the frame-sequencing [0087] module 64 may allow an operator to select, for example, the point of interest (POI), the vertical unit point V1, and/or the translation point C0 for the sequence of images used in the image sequence. These points may be selected, for example, to correct for misalignment errors in the camera systems and/or select a POI that is different than the object on which the camera systems are servoing.
  • FIG. 14 is a diagram of a process flow through the [0088] image sequence generator 60 according to one embodiment in which the POI, the vertical unit point V1, and the translation point CO are specified for each image of the sequence to, for example, simulate the effects of changing yaw, pitch, roll and translation Tz along the cameras' central viewing rays. For purposes of illustration, the process flow illustrated in FIG. 14 is for the generation of a 3D stop-motion image sequence in which all of the displayed images are from the same instant in time (t). According to other embodiments, however, as explained previously, the image sequence generator 60 may also output images from different time steps.
  • The process illustrated in FIG. 14 initiates at block [0089] 110 where the image sequence generator 60 reads the time (t) for which the 3D stop-motion image sequence is to be generated. As described previously, the video reviewer may specify this instant in time through the video reviewer interface module 62. Next, at block 112 the frame-sequencing module 64 may retrieve from the video storage units 30 the image (images I1-N) for all of the camera systems 14, 16 to be used in the sequence, which may also be specified through the video reviewer interface module 62 and may be all or less than all of the camera systems.
  • At block [0090] 114, the image sequence generator 60 may read the POI, the vertical unit point V1 and the translation point C0 for each image I1-N. As described previously, an operator may enter the point for each image I1-N through the video reviewer interface module 62 such as by, for example, clicking on the point in the image with a mouse or stylus. According to another embodiment, the operator may enter a character string corresponding to the coordinates of the respective points for each image I1-N through the video reviewer interface module 62.
  • At [0091] block 116, the frame-sequencing module 64 transforms the images I1-N according to the homography defined by the POI, the V1, and the C0. That is, as described previously, the POI may be mapped to the point C0, and the vertical unit point V1 may be mapped to one vertical unit above C0. Next, at block 118, the frame-sequencing module 64 may output the transformed images I1′-N′ in sequence corresponding to the order of the placement of the corresponding camera systems 14, 16 around the scene 12, either clockwise or counter-clockwise, to generate the video image sequence.
  • When the [0092] video systems 14, 16 are mounted only sparsely along the specified gross trajectory, a sequence of frames retrieved spatially and played back will appear discrete and discontinuous as the views jump from one camera location to the next. To overcome this effect, the frame-sequencing module 64 may also generate new (“virtual”) video images to “fill in” between the image frames from existing views to, for example, produce a new video image sequence that appears to be smoother and more continuous. The new video images may be generated using induced camera motion, as described previously, to “interpolate” camera positions between the actual camera system locations. For example, according to one embodiment, a sequence of new images may be generated that smoothly varies the rotation and translation from one camera viewpoint into another, resulting in an apparent smooth motion of a single camera from the first position into the second.
  • Although the present invention has been described herein with respect to certain embodiments, those of ordinary skill in the art will recognize that many modifications and variations of the present invention may be implemented. For example, rather than employing a distributed architecture, the [0093] master control unit 24 and the slave camera control units 26 may be integrated into one computer device. According to such an embodiment, the master control unit 24 may therefore further include a servo control module 44 for computing the PTZF commands for each slave camera system 16.
  • According to one embodiment, the [0094] image sequence generator 60 may be integrated with the computing device of the master control unit 24, as may the remote operator interface unit 80 or the computer vision control unit 84. According to another embodiment, the image sequence generator 60 may be distributed across more than one computing device. In addition, according to another embodiment, the slave control module 43 may be distributed among the slave camera control units 26. According to such an embodiment, the appropriate master/slave mapping data may be stored in a memory unit of the slave camera control units 26.
  • According to another embodiment, one may chose to make one of the [0095] slave camera systems 16 the master camera system. Accordingly, the original master camera system 14 would then be under the control of one of the slave camera control units 26. This may be realized, for example, by connecting each of the camera systems 14, 16 to a network such that each camera system 14, 16 is in communication with the master control unit 24 and at least one slave camera control unit.
  • According to another embodiment, the [0096] system 10 may include a plurality of master camera systems 14, each one controlling a subset of the slave camera systems 16. According to such an embodiment, the system 10 may include a plurality of master control units 24, one for each master camera system 14. According to one embodiment, each of the master control units 24 may be centralized in one computing device.
  • The foregoing description and the following claims are intended to cover all such modifications and variations. [0097]
  • Appendix [0098]
  • Let w be a 3D point in world coordinates, c be the location of the camera in that coordinate system, R be a rotation matrix representing the orientation of the camera (see FIG. A[0099] 1). A local Cartesian camera coordinate system is specified by axes X-Y-Z, where the Z axis lies along the central viewing ray of the camera and X and Y are perpendicular to that ray. We represent points in the image using film coordinates (u, v) on a film plane displaced from the optic center by a distance f, the focal length. Note that film coordinates are related to image pixel coordinates through an affine transformation that is a function of linear camera intrinsic parameters, and also perhaps by an additional nonlinear transformation representing lens distortion. Working in film coordinates amounts to assuming a calibrated camera, for which intrinsic parameters and lens distortion are known and invertible.
  • In homogeneous coordinates, we have the projection equation [0100]
  • [uvf]′˜R(w−c)
  • relating film coordinates to 3D world coordinates, where ˜ denotes equality up to a scale factor and ′ denotes a vector or matrix transpose. [0101]
  • A[0102] 1: Camera Rotation Induces a 2D Homography in Film Coordinates.
  • Let [u[0103] 1 v1 f]′ be film coordinates for the image taken with orientation R1, and let [u2 V2 f]′ be film coordinates for the image taken with orientation R2. Then
  • [u1v1f]′˜R1(w−c)
  • and [0104]
  • [u2v2f]′˜R2 (w−c)R2R1′ [u1V1f]′
  • Therefore he change in camera orientation induces a 2D homography R[0105] 2 R1′ on points in the film plane, and the image can be transformed with no knowledge of scene structure.
  • A2: Change in Zoom Induces an Isotropic Sag of Film Coordinates. [0106]
  • Let [u[0107] 1, v1, f1]′ be film coordinates for the image taken with focal length f1, and let [u2, v2, f2]′ be film coordinates for the image taken with focal length f2. Then
  • [u2 v2f2]′=(f2/f1)[u1v1f1]′
  • and thus change in focal length induces an isotropic scaling of film coordinates by the magnification factor f[0108] 2/f1.
  • A3: Translation Tz is Approximately Art Isotropic Scaling of Film Coordinates [0109]
  • Let [xy,z]′=R(w−c) be a 3D world pointp represented within the camera coordinate system and let [u[0110] 1, v1, f1]=[y/z, x/z, f] be its film coordinates. If the camera is translated a small distance Tz along the camera's Z axis, then the new coordinates of p are [x,y,z−Tz]′, which map to film coordinates [u2, v2, f]′=[y/z−Tz), x/(z−Tz), f], so that
  • [u2v2]=(z/(z−Tz))[u1, v1]
  • and the point is transformed by an isotropic scaling of film coordinates Although this is strictly true only for this point, and other points at scene depth z, if the object we are looking at is “shallow” with respect to distance from the camera (variation in z coordinates is small with respect to magnitude of z), then this formula is approximately true for all other points on the object as well. Therefore, translation by a small amount Tz towards or away from a distant object is approximated by an isotropic scaling of film coordinates. [0111]
  • A4: Translation Tx and Ty is Approximately a 2D Image Translation [0112]
  • Let [x,y,z]′=(w−c) be a 3D world point p represented within the camera coordinate system, and let [u[0113] 1, v1, f]′=[y/z, x/z, f] be its film coordinates. If the camera is translated a small distance Tx and Ty along the camera's X and Y axes, then the new coordinates of p are [x−x,y−Ty,z]′, which map to film coordinates [u2, v2, f]′=[(y−Ty)/z), (x−Tx)/z, f], so that
  • [u 2v2 ]=[u 1 −Ty/z; v 1 −Tx/z]
  • and the point is transformed by a 2D translation of film coordinates. Although this is strictly true only for this point, and other points at scene depth z, if the object we are looking at is “shallow” with respect to distance from the camera (variation in z coordinates is small with respect to magnitude of z), then this formula is approximately true for all other points on the object as well. Therefore, translation by a small amount Tx and Ty perpendicular to the central viewing axis of the camera (the Z axis) can approximated for distant objects by a 2D translation in film coordinates [0114]
  • A5: Composition of Transformations [0115]
  • The separate transformations for changing orientation and zoom (or translation Tz) can be combined as [0116]
  • [u 2 v 2 f 2]′˜(f 2 /f 1)R 2 R 1′[u1 v 1 f 1]′
  • A6: Local Parameterization of Changing Orientation and Translation [0117]
  • Given an image, we do not necessarily know it's absolute orientation R[0118] 1 with respect to the world, nor the absolute orientation R2 of the desired image. We thus write the homography for changing camera orientation in terms of the current, local camera coordinate. Note that 2D homography R2R1′ has the form of a 3D rotation matrix R (up to a scale factor). We can therefore specify the homography R in local camera coordinates as a rotation by angle θ around axis a=[abc]′. By Rodrigues formula, we can write R as
  • R=I+sin θA+(1−cos θ)A 2
  • where I is the identity matrix and A is the skew-symmetric matrix [0119] A = [ 0 - c b c 0 - a - b a 0 ]
    Figure US20020145660A1-20021010-M00001
  • Note, for sufficiently small rotation angles θ, we can use the small angle approximations sin θ≈θ and cos θ≈1 to write [0120] R I + θ A = [ 1 - - c θ b θ c θ 1 - a θ - b θ a θ 1 ]
    Figure US20020145660A1-20021010-M00002
  • Comparing to the small angle approximation formula for a rotation matrix in terms of Euler angles roll. pitch and yaw about the camera x, y and z axes, respectively (see FIG. A[0121] 1), we see that for small angles: yaw≈θa, pitch≈θb and roll≈θc.
  • Similar to the case of changing camera orientation, we do not necessarily know the focal length f[0122] 1 of a given image, nor the focal length f2 we would like it to have after a simulated camera zoom. Instead, we parameterize a zoom transformation in terms of a desired magnification factor M=f2/f1.
  • In the case of translation by Tz along the central viewing ray, we do not know the distance z to object points, or the distance Tz of the translation, but we likewise can specify the isotropic scaling induced by the translation as a magnification factor m. Furthermore, translation by Tx and Ty perpendicular to the viewing ray can be specified locally as 2D translation parameters du and dv, which are functions of the unknown object distance z and magnitude of translation components Tx and Ty. [0123]
  • A7: Specifying Yaw and Pitch Correction by Selecting One Point Correspondence. [0124]
  • Here we show that correction of camera yaw and pitch can be specified by selecting one point correspondence (FIG. A[0125] 2). Point (u,v) is selected, and we wish to determine the camera orientation correction that brings this point to the center of the image (u=0, v=0). That is, we wish to compute the transformation that rotates ray [uvf]′ into alignment with ray [uvf]′. The unit vector axis of this rotation is
  • [abc]=[uvf]′×[00f]′=[v−u0]′/sqrt(u 2 +v 2)
  • and the clockwise angle of rotation is [0126]
  • θ=a tan(sqrt(u 2 +v 2)/f)
  • from which we can write the 2D homography R using Rodrigues formula, above By construction, the third component of the rotation axis is always zero, i.e. c=0 so that for small angles this is a correction in yaw and pitch alone, with no roll angle correction. [0127]
  • A8: Efficient Approximation for Implementing Yaw and Pitch Correction [0128]
  • A great implementation speedup for yaw and pitch correction is possible in the case that we have both small rotation angle and a large focal length In this case, θ≈(sqr(u[0129] 2+v2)/f) and we can use R≈I+θA. After some manipulation we find that the orientation correction homography
  • [u2v2f]′˜R[u1v1f]′
  • reduces to [0130]
  • u[0131] 2=u1˜u
  • v[0132] 2≈v1˜v
  • which is simply the image shift (translation) that takes POI (u,v) to the center of the image. [0133]
  • A9 Specifying Yaw, Pitch, Roll and Tz by Selecting Two Point Correspondences. [0134]
  • Here we show that correction of camera yaw, pitch, roll and Tz can be specified by selecting two point correspondences (FIG. A[0135] 3). As previously (case A7) we select one point as the PO and determine the 2D homogaphy R that brings this point to the image center (u=0, v=0) In addition to the POI, we also choose a second point, called V1 to denote “vertical unit”, which will be mapped one “unit” vertically above the image center. The definition of what distance μ in pixels corresponds to 1 unit is specified by the user (e.g. 1 unit=μ=100 pixels).
  • Assumed point V1 is mapped to film coordinate (s,t) by homography R. The coordinate system must be rotated and isotropically scaled so that point (s,t) maps to point (O,μ) which is located verticaly above the image center at distance μ. It can easily be shown that this transformation can be written as [0136] S = [ t μ s 2 + t 2 - s μ s 2 + t 2 0 s μ s 2 + t 2 t μ s 2 + t 2 0 0 0 1 ]
    Figure US20020145660A1-20021010-M00003
  • Roll and Tz correction S can be composed with pitch and yaw correction R to yield a single 2D homography H=S R that performs pitch, yaw, roll and Tz correction. [0137]
  • A10: Efficient Approximation for Changing Yaw, Pitch, Roll and Tz [0138]
  • When implementing for small angles and large focal lengths, the 2D homography H above can be simplified to a translation, rotation and isotropic scale, i.e. a single similarity transformation, leading to a great computational savings. The derivation is similar to the case A8, above. [0139]
  • A11: Specifying Yaw, Pitch, Roll, Tx, Ty and Tz by Selecting Three Point Correspondences. [0140]
  • Here we show that full correction of camera yaw, pitch, roll, Tx, Ty, and Tz can be specified by selecting three point correspondences (FIG. A[0141] 4). As previously (case A9), we select two points to be the POI and VI, and compute the homography that corresponds to correcting yaw, pitch, roll and Tz by bringing the POI to the image center (u=0,v=0) and V1 to point (0,μ) one vertical unit above the center. Now select a third point C0=(p,q) in this connected image. We then want to apply the 2D translation that maps the origin (0,0) to point C0=(p,q), namely
  • u2=u+p [0142]
  • v2=v+q [0143]
  • is the correction for Tx and Ty. Let this transformation be written as matrix T. Then the Tx and Ty correction can be composed with the previous pitch, yaw, roll and Tz corrections (R and S) to form a single 2D homography H=TSR that performs yaw, pitch, roll, Tx, Ty, and Tz. [0144]
  • A12: Efficient Approximation for Changing Yaw, Pitch, Roll, Tx, Ty and Tz [0145]
  • When implementing for small angles and large focal lengths, the 2D homography H above can be simplified to a translation, rotation and isotropic scale, i.e. a single similarity transformation, leading to a great computational savings. The derivation is similar to cases A8 and A10, above. [0146]
    Figure US20020145660A1-20021010-P00001
    Figure US20020145660A1-20021010-P00002

Claims (53)

What is claimed is:
1. A method of generating a video image sequence, comprising:
positioning a plurality of camera systems relative to a scene such that the camera systems define a gross trajectory;
transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory; and
displaying the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
2. The method of claim 1, wherein positioning the plurality of camera systems includes positioning a plurality of pan/tilt camera systems relative to the scene.
3. The method of claim 1, wherein positioning the plurality of camera systems includes positing a plurality of static camera systems relative to the scene.
4. The method of claim 1, wherein positioning the plurality of camera systems includes positioning at least one pan/tilt camera system and one static camera system relative to the scene.
5. The method of claim 1, wherein transforming images from the camera systems includes applying a 2D image transformation to the images .
6. The method of claim 5, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a one point correspondence between the images.
7. The method of claim 6, wherein applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes mapping a point of interest in each image to the center of the image.
8. The method of claim 5, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a two point correspondence between the images.
9. The method of claim 8, wherein applying the 2D image transformation according to a homography defined by a two point correspondence between the images includes:
mapping a point of interest in each image to the center of the image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.
10. The method of claim 5, wherein applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a three point correspondence between the images.
11. The method of claim 10, wherein applying the 2D image transformation according to a homography defined by a three point correspondence between the images includes:
mapping a translation point in each image to the center of the image;
mapping a point of interest in each image to the translation point in each image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.
12. The method of claim 1, further comprising:
generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and
displaying the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.
13. The method of claim 12, wherein generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems includes generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.
14. The method of claim 1, wherein positioning the plurality of camera systems relative to the scene includes positing the plurality of camera systems in a close-ended configuration relative to the scene.
15. The method of claim 1, wherein positioning the plurality of camera systems relative to the scene includes positing the plurality of camera systems in an array configuration.
16. A system for generating a video image sequence of an object within a scene, comprising:
means for capturing an image from a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory; and
means for transforming images from the camera systems to superimpose a secondary induced motion on the gross trajectory.
17. The system of claim 16, further comprising means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed at a target within the scene and a size of the target in the images from the camera systems is substantially the same over time.
18. The system of claim 17, further comprising means for outputting the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
19. The system of claim 18, wherein the means for transforming images from the camera systems includes means for applying a 2D image transformation to the images.
20. The system of claim 19, wherein the means for applying the 2D image transformation includes means for applying a 2D image transformation according to a homography defined by a one-point correspondence between the images.
21. The system of claim 20, wherein the means for applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes means for mapping a point of interest in each image to the center of the image.
22. The system of claim 21, wherein the point of interest is not a point of the target.
23. The system of claim 21, wherein the point of interest is a point of the target.
24. The system of claim 19, wherein the means for applying the 2D image transformation includes applying a 2D image transformation according to a homography defined by a two-point correspondence between the images.
25. The system of claim 24, wherein the means for applying the 2D image transformation according to a homography defined by a two-point correspondence between the images includes:
means for mapping a point of interest in each image to the center of the image; and
means for mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.
26. The system of claim 19, wherein the means for applying the 2D image transformation includes means for applying a 2D image transformation according to a homography defined by a three-point correspondence between the images.
27. The system of claim 26, wherein the means for applying the 2D image transformation according to a nomography defined by a three-point correspondence between the images includes:
means for mapping a translation point in each image to the center of the image;
means for mapping a point of interest in each image to the translation point in each image; and
means for mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.
28. The system of claim 16, further comprising:
means for generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and
means for outputting the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.
29. The system of claim 28, wherein the means for generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems includes means for generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.
30. A system for generating a video image sequence of an object within a scene, comprising:
a plurality of camera systems positioned relative to the scene such that the camera systems define a gross trajectory;
a video storage unit in communication with the camera systems; and
a frame-sequencing module in communication with the video storage unit for transforming images of the camera systems retrieved from the video storage unit to superimpose a secondary induced motion on the gross trajectory.
31. The system of claim 30, further comprising means for controlling the plurality of camera systems such that the camera systems are simultaneously aimed a target within the scene and a size of the target in the images from the camera systems is substantially the same over time.
32. The system of claim 31, wherein the frame-sequencing module is further for outputting the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
33. The system of claim 32, wherein the frame-sequencing module is further for:
generating an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and
outputting the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.
34. The system of claim 33, wherein the frame-sequencing module is for generating the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems by generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.
35. The system of claim 30, wherein the frame-sequencing module is for transforming images from the camera systems by applying a 2D image transformation to the images.
36. The system of claim 35, wherein the 2D image transformation includes a 2D image transformation according to a nomography defined by a one-point correspondence between the images.
37. The system of claim 36, wherein frame-sequencing module is for applying the 2D image transformation according to a homography defined by a one point correspondence between the images includes by mapping a point of interest in each image to the center of the image.
38. The system of claim 37, wherein the point of interest is not a point of the target.
39. The system of claim 37, wherein the point of interest is a point of the target.
40. The system of claim 35, wherein the 2D image transformation includes a 2D image transformation according to a homography defined by a two-point correspondence between the images.
41. The system of claim 40, wherein the frame-sequencing module is for applying the 2D image transformation according to a homography defined by a two-point correspondence between the images by:
mapping a point of interest in each image to the center of the image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.
42. The system of claim 30, wherein the 2D image transformation includes a 2D image transformation according to a homography defined by a three-point correspondence between the images.
43. The system of claim 42, wherein the frame-sequencing module is for applying the 2D image transformation according to a homography defined by a three-point correspondence between the images by:
mapping a translation point in each image to the center of the image;
mapping a point of interest in each image to the translation point in each image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.
44. A computer readable medium, having stored thereon instructions which, when executed by a processor, cause the processor to:
transform images from a plurality of camera systems positioned relative to a scene to define a gross trajectory to superimpose a secondary induced motion on the gross trajectory; and
output the transformed images in sequence corresponding to the position of the corresponding camera systems along the gross trajectory.
45. The computer readable medium of claim 44, having further stored thereon instructions which, when executed by the processor, cause the processor to transform the images from the camera systems by applying a 2D image transformation to the images.
46. The computer readable medium of claim 45, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation by applying a 2D image transformation according to a homography defined by a one point correspondence between the images.
47. The computer readable medium of claim 46, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a nomography defined by a one point correspondence between the images by mapping a point of interest in each image to the center of the image.
48. The computer readable medium of claim 45, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation by applying a 2D image transformation according to a homography defined by a two point correspondence between the images.
49. The computer readable medium of claim 48, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a homography defined by a two point correspondence between the images by mapping a point o f interest in each image to the center of the image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the center of the image.
50. The computer readable medium of claim 45, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation includes applying a 2D image transformation according to a homography defined by a three point correspondence between the images.
51. The computer readable medium of claim 50, having further stored thereon instructions which, when executed by the processor, cause the processor to apply the 2D image transformation according to a homography defined by a three point correspondence between the images by:
mapping a translation point in each image to the center of the image;
mapping a point of interest in each image to the translation point in each image; and
mapping a vertical unit point in each image to a point at a predetermined vertical relationship to the translation point.
52. The computer readable medium of claim 1, having further stored thereon instructions which, when executed by the processor, cause the processor to:
generate an image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems; and
display the image between display of the transformed image from the first camera system and display of the transformed image from the second camera system.
53. The computer readable medium of claim 52, having further stored thereon instructions which, when executed by the processor, cause the processor to generate the image corresponding to an image from a virtual camera system positioned along the gross trajectory between first and second camera systems of the plurality of camera systems by generating an image corresponding to an image from a virtual camera system having a rotation and translation interpolated from a rotation and translation of the first and second camera systems.
US10/074,313 2001-02-12 2002-02-12 System and method for manipulating the point of interest in a sequence of images Expired - Lifetime US7106361B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10/074,313 US7106361B2 (en) 2001-02-12 2002-02-12 System and method for manipulating the point of interest in a sequence of images

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US26820601P 2001-02-12 2001-02-12
US26820501P 2001-02-12 2001-02-12
US10/074,313 US7106361B2 (en) 2001-02-12 2002-02-12 System and method for manipulating the point of interest in a sequence of images

Publications (2)

Publication Number Publication Date
US20020145660A1 true US20020145660A1 (en) 2002-10-10
US7106361B2 US7106361B2 (en) 2006-09-12

Family

ID=26952939

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/074,315 Expired - Fee Related US7102666B2 (en) 2001-02-12 2002-02-12 System and method for stabilizing rotational images
US10/074,313 Expired - Lifetime US7106361B2 (en) 2001-02-12 2002-02-12 System and method for manipulating the point of interest in a sequence of images

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/074,315 Expired - Fee Related US7102666B2 (en) 2001-02-12 2002-02-12 System and method for stabilizing rotational images

Country Status (2)

Country Link
US (2) US7102666B2 (en)
WO (2) WO2002065763A2 (en)

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020118286A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for servoing on a moving fixation point within a dynamic scene
US20020118969A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for stabilizing rotational images
US20030076413A1 (en) * 2001-10-23 2003-04-24 Takeo Kanade System and method for obtaining video of multiple moving fixation points within a dynamic scene
US20030185302A1 (en) * 2002-04-02 2003-10-02 Abrams Thomas Algie Camera and/or camera converter
US20030185301A1 (en) * 2002-04-02 2003-10-02 Abrams Thomas Algie Video appliance
US20030208638A1 (en) * 2002-04-02 2003-11-06 Abrams Thomas Algie Digital production services architecture
US20050046626A1 (en) * 2003-09-02 2005-03-03 Fuji Photo Film Co., Ltd. Image generating apparatus, image generating method and image generating program
US20060066722A1 (en) * 2004-09-28 2006-03-30 Objectvideo, Inc. View handling in video surveillance systems
US20060152435A1 (en) * 2002-06-05 2006-07-13 Tetsujiro Kondo Display device and display method
US20070019943A1 (en) * 2005-07-21 2007-01-25 Takahiko Sueyoshi Camera system, information processing device, information processing method, and computer program
US20070058048A1 (en) * 2005-09-13 2007-03-15 Toshiro Kinugasa Photographing system for a moving apparatus
US20070274705A1 (en) * 2004-05-13 2007-11-29 Kotaro Kashiwa Image Capturing System, Image Capturing Device, and Image Capturing Method
US20080036864A1 (en) * 2006-08-09 2008-02-14 Mccubbrey David System and method for capturing and transmitting image data streams
US20080143821A1 (en) * 2006-12-16 2008-06-19 Hung Yi-Ping Image Processing System For Integrating Multi-Resolution Images
US20090284585A1 (en) * 2008-05-15 2009-11-19 Industrial Technology Research Institute Intelligent multi-view display system and method thereof
US20100021152A1 (en) * 2005-02-03 2010-01-28 Gurpal Sandhu Apparatus and method for viewing radiographs
US20120044373A1 (en) * 2010-08-20 2012-02-23 Canon Kabushiki Kaisha Imaging system and image capturing apparatus
US20120127319A1 (en) * 2010-11-19 2012-05-24 Symbol Technologies, Inc. Methods and apparatus for controlling a networked camera
US20130038723A1 (en) * 2011-08-11 2013-02-14 Canon Kabushiki Kaisha Image acquisition apparatus and image processing apparatus
US20150010287A1 (en) * 2011-09-29 2015-01-08 Teppei Eriguchi Video image display device, video image display method, program, and video image processing/display system
US20150067499A1 (en) * 2013-09-02 2015-03-05 Ken Lai System and Method for Sequencing Rotatable Images
US20150124171A1 (en) * 2013-11-05 2015-05-07 LiveStage°, Inc. Multiple vantage point viewing platform and user interface
US20150302655A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Using a map of the world for augmented or virtual reality systems
US9242379B1 (en) * 2015-02-09 2016-01-26 The Trustees Of The University Of Pennysylvania Methods, systems, and computer readable media for producing realistic camera motion for stop motion animation
US20160227128A1 (en) * 2015-01-29 2016-08-04 Electronics And Telecommunications Research Institute Multi-camera control apparatus and method to maintain location and size of object in continuous viewpoint switching service
WO2016138507A1 (en) * 2015-02-27 2016-09-01 Leia Inc. Multiview camera
US20160263479A1 (en) * 2015-03-11 2016-09-15 Nintendo Co., Ltd. Storage medium having stored therein game program, game apparatus, game system, and game processing method
WO2016203584A1 (en) * 2015-06-17 2016-12-22 オリンパス株式会社 Image pickup control device, image pickup system, image pickup control method, and program
WO2017120802A1 (en) * 2016-01-12 2017-07-20 Shanghaitech University Stitching method and apparatus for panoramic stereo video system
EP3175424A4 (en) * 2014-12-23 2017-08-02 Huawei Technologies Co. Ltd. Computational multi-camera adjustment for smooth view switching and zooming
US10029180B2 (en) 2015-03-11 2018-07-24 Nintendo Co., Ltd. Storage medium having stored therein game program, game apparatus, game system, and game processing method
US20190066376A1 (en) * 2017-08-31 2019-02-28 Hs R & A Co., Ltd. Method for interpreting layout of tube by using three-dimensional coordinates and recording medium thereof
WO2020247906A1 (en) * 2019-06-06 2020-12-10 Ovad Custom Stages, Llc Vehicle photographic system for identification of surface imperfections
US11030733B2 (en) * 2018-12-24 2021-06-08 Beijing Dajia Internet Information Technology Co., Ltd. Method, electronic device and storage medium for processing image
US11138760B2 (en) * 2019-11-06 2021-10-05 Varjo Technologies Oy Display systems and methods for correcting drifts in camera poses
US11277596B2 (en) * 2018-10-26 2022-03-15 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and storage medium
US11310478B2 (en) 2017-10-02 2022-04-19 Leia Inc. Multiview camera array, multiview system, and method having camera sub-arrays with a shared camera
US20220224822A1 (en) * 2018-10-24 2022-07-14 Sony Corporation Multi-camera system, control value calculation method, and control apparatus

Families Citing this family (78)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040239763A1 (en) * 2001-06-28 2004-12-02 Amir Notea Method and apparatus for control and processing video images
JP4043258B2 (en) * 2002-03-13 2008-02-06 オリンパス株式会社 3D imaging device
GB2416457B (en) * 2002-08-23 2006-07-05 Time Slice Cameras Ltd Image capture system enabling special effects
CN100523715C (en) * 2002-12-27 2009-08-05 有泽博 Multi-view-point video capturing system
US7446797B2 (en) * 2003-02-10 2008-11-04 Activeye, Inc. User assisted customization of automated video surveillance systems
US20050134685A1 (en) * 2003-12-22 2005-06-23 Objectvideo, Inc. Master-slave automated video-based surveillance system
JP3700707B2 (en) * 2003-03-13 2005-09-28 コニカミノルタホールディングス株式会社 Measuring system
GB2413720B (en) * 2003-03-14 2006-08-02 British Broadcasting Corp Video processing
JP4635477B2 (en) * 2003-06-10 2011-02-23 カシオ計算機株式会社 Image photographing apparatus, pseudo three-dimensional image generation method, and program
JP2005050037A (en) * 2003-07-31 2005-02-24 Canon Inc Image processing method and apparatus
US7880766B2 (en) * 2004-02-03 2011-02-01 Panasonic Corporation Detection area adjustment apparatus
JP2005295004A (en) * 2004-03-31 2005-10-20 Sanyo Electric Co Ltd Stereoscopic image processing method and apparatus thereof
US7542588B2 (en) * 2004-04-30 2009-06-02 International Business Machines Corporation System and method for assuring high resolution imaging of distinctive characteristics of a moving object
JP4516791B2 (en) * 2004-07-22 2010-08-04 パナソニック株式会社 Camera interlocking system, camera device, and camera interlocking control method
US20060028476A1 (en) * 2004-08-03 2006-02-09 Irwin Sobel Method and system for providing extensive coverage of an object using virtual cameras
US20070058717A1 (en) * 2005-09-09 2007-03-15 Objectvideo, Inc. Enhanced processing for scanning video
US8112325B2 (en) * 2005-09-15 2012-02-07 Manheim Investments, Inc. Method and apparatus for automatically capturing multiple images of motor vehicles and other items for sale or auction
JP2007208781A (en) * 2006-02-03 2007-08-16 Olympus Corp Imaging apparatus
US8560047B2 (en) 2006-06-16 2013-10-15 Board Of Regents Of The University Of Nebraska Method and apparatus for computer aided surgery
US20070291985A1 (en) * 2006-06-20 2007-12-20 Nils Oliver Krahnstoever Intelligent railyard monitoring system
US10298834B2 (en) 2006-12-01 2019-05-21 Google Llc Video refocusing
WO2008147913A2 (en) * 2007-05-22 2008-12-04 Vidsys, Inc. Tracking people and objects using multiple live and recorded surveillance camera video feeds
JP4998156B2 (en) * 2007-08-30 2012-08-15 ソニー株式会社 Information presenting system, information presenting apparatus, information presenting method, program, and recording medium recording the program
EP2076055B1 (en) * 2007-12-27 2012-10-24 Saab AB Method for displaying a virtual image
US20100321475A1 (en) * 2008-01-23 2010-12-23 Phillip Cox System and method to quickly acquire three-dimensional images
US8520054B2 (en) * 2008-01-23 2013-08-27 Techtol Holdings, Llc System and method to quickly acquire images
US9071626B2 (en) 2008-10-03 2015-06-30 Vidsys, Inc. Method and apparatus for surveillance system peering
KR101594048B1 (en) * 2009-11-09 2016-02-15 삼성전자주식회사 3 device and method for generating 3 dimensional image using cooperation between cameras
KR101367820B1 (en) * 2009-12-21 2014-02-27 한국전자통신연구원 Portable multi view image acquisition system and method
CN102148952B (en) * 2010-02-05 2013-12-11 鸿富锦精密工业(深圳)有限公司 Video image compression method and playing method thereof
DE102010024042B4 (en) * 2010-06-16 2012-03-29 Florian Maier Method and device for recording three-dimensional image material for different presentation sizes using the full depth budget
US11911117B2 (en) 2011-06-27 2024-02-27 Board Of Regents Of The University Of Nebraska On-board tool tracking system and methods of computer assisted surgery
US9498231B2 (en) 2011-06-27 2016-11-22 Board Of Regents Of The University Of Nebraska On-board tool tracking system and methods of computer assisted surgery
CA2840397A1 (en) 2011-06-27 2013-04-11 Board Of Regents Of The University Of Nebraska On-board tool tracking system and methods of computer assisted surgery
US9100587B2 (en) * 2011-07-22 2015-08-04 Naturalpoint, Inc. Hosted camera remote control
US9224063B2 (en) 2011-08-02 2015-12-29 Viewsiq Inc. Apparatus and method for digital microscopy imaging
TWI524735B (en) * 2012-03-30 2016-03-01 華晶科技股份有限公司 Method and device for generating three-dimensional image
US9858649B2 (en) 2015-09-30 2018-01-02 Lytro, Inc. Depth-based image blurring
CN104904200B (en) 2012-09-10 2018-05-15 广稹阿马斯公司 Catch the unit and system of moving scene
US9087386B2 (en) 2012-11-30 2015-07-21 Vidsys, Inc. Tracking people and objects using multiple live and recorded surveillance camera video feeds
US9001226B1 (en) * 2012-12-04 2015-04-07 Lytro, Inc. Capturing and relighting images using multiple devices
US9462301B2 (en) 2013-03-15 2016-10-04 Google Inc. Generating videos with multiple viewpoints
US9294757B1 (en) * 2013-03-15 2016-03-22 Google Inc. 3-dimensional videos of objects
US10037474B2 (en) * 2013-03-15 2018-07-31 Leap Motion, Inc. Determining the relative locations of multiple motion-tracking devices
US10105149B2 (en) 2013-03-15 2018-10-23 Board Of Regents Of The University Of Nebraska On-board tool tracking system and methods of computer assisted surgery
US10334151B2 (en) 2013-04-22 2019-06-25 Google Llc Phase detection autofocus using subaperture images
US10033992B1 (en) 2014-09-09 2018-07-24 Google Llc Generating a 3D video of an event using crowd sourced data
JP6452386B2 (en) * 2014-10-29 2019-01-16 キヤノン株式会社 Imaging apparatus, imaging system, and imaging apparatus control method
US9998655B2 (en) * 2014-12-23 2018-06-12 Quallcomm Incorporated Visualization for viewing-guidance during dataset-generation
US10469873B2 (en) 2015-04-15 2019-11-05 Google Llc Encoding and decoding virtual reality video
US10444931B2 (en) 2017-05-09 2019-10-15 Google Llc Vantage generation and interactive playback
US10341632B2 (en) 2015-04-15 2019-07-02 Google Llc. Spatial random access enabled video system with a three-dimensional viewing volume
US10565734B2 (en) 2015-04-15 2020-02-18 Google Llc Video capture, processing, calibration, computational fiber artifact removal, and light-field pipeline
US10540818B2 (en) 2015-04-15 2020-01-21 Google Llc Stereo image generation and interactive playback
US11328446B2 (en) 2015-04-15 2022-05-10 Google Llc Combining light-field data with active depth data for depth map generation
US10440407B2 (en) 2017-05-09 2019-10-08 Google Llc Adaptive control for immersive experience delivery
US10412373B2 (en) 2015-04-15 2019-09-10 Google Llc Image capture for virtual reality displays
US10546424B2 (en) 2015-04-15 2020-01-28 Google Llc Layered content delivery for virtual and augmented reality experiences
US10567464B2 (en) 2015-04-15 2020-02-18 Google Llc Video compression with adaptive view-dependent lighting removal
US10419737B2 (en) 2015-04-15 2019-09-17 Google Llc Data structures and delivery methods for expediting virtual reality playback
US10275898B1 (en) 2015-04-15 2019-04-30 Google Llc Wedge-based light-field video capture
US9979909B2 (en) 2015-07-24 2018-05-22 Lytro, Inc. Automatic lens flare detection and correction for light-field images
WO2017087551A1 (en) * 2015-11-16 2017-05-26 Cuica Llc Inventory management and monitoring
JP6672075B2 (en) 2016-05-25 2020-03-25 キヤノン株式会社 CONTROL DEVICE, CONTROL METHOD, AND PROGRAM
JP6429829B2 (en) 2016-05-25 2018-11-28 キヤノン株式会社 Image processing system, image processing apparatus, control method, and program
US10275892B2 (en) 2016-06-09 2019-04-30 Google Llc Multi-view scene segmentation and propagation
US11025921B1 (en) 2016-09-22 2021-06-01 Apple Inc. Providing a virtual view by streaming serialized data
US10679361B2 (en) 2016-12-05 2020-06-09 Google Llc Multi-view rotoscope contour propagation
US10594945B2 (en) 2017-04-03 2020-03-17 Google Llc Generating dolly zoom effect using light field image data
US10474227B2 (en) 2017-05-09 2019-11-12 Google Llc Generation of virtual reality with 6 degrees of freedom from limited viewer data
US10354399B2 (en) 2017-05-25 2019-07-16 Google Llc Multi-view back-projection to a light-field
US10545215B2 (en) 2017-09-13 2020-01-28 Google Llc 4D camera tracking and optical stabilization
JP2019103067A (en) 2017-12-06 2019-06-24 キヤノン株式会社 Information processing device, storage device, image processing device, image processing system, control method, and program
US10965862B2 (en) 2018-01-18 2021-03-30 Google Llc Multi-camera navigation interface
JP7297515B2 (en) * 2019-04-25 2023-06-26 キヤノン株式会社 Information processing device, information processing method and program
JP2021136511A (en) * 2020-02-25 2021-09-13 キヤノン株式会社 Imaging apparatus, imaging system, control method, program, and, storage medium
JP2022006912A (en) * 2020-06-25 2022-01-13 キヤノン株式会社 Control apparatus, image pickup apparatus, monitoring system, and control method
CN113115110B (en) * 2021-05-20 2022-04-08 广州博冠信息科技有限公司 Video synthesis method and device, storage medium and electronic equipment

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5164827A (en) * 1991-08-22 1992-11-17 Sensormatic Electronics Corporation Surveillance system with master camera control of slave cameras
US5489886A (en) * 1992-08-07 1996-02-06 Alos-Officiating Tennis System Limited Automatic line officiating system and method thereof
US5714997A (en) * 1995-01-06 1998-02-03 Anderson; David P. Virtual reality television system
US5912700A (en) * 1996-01-10 1999-06-15 Fox Sports Productions, Inc. System for enhancing the television presentation of an object at a sporting event
US5917937A (en) * 1997-04-15 1999-06-29 Microsoft Corporation Method for performing stereo matching to recover depths, colors and opacities of surface elements
US6005610A (en) * 1998-01-23 1999-12-21 Lucent Technologies Inc. Audio-visual object localization and tracking system and method therefor
US6084979A (en) * 1996-06-20 2000-07-04 Carnegie Mellon University Method for creating virtual reality
US6094198A (en) * 1994-01-10 2000-07-25 Cognitens, Ltd. System and method for reconstructing surface elements of solid objects in a three-dimensional scene from a plurality of two dimensional images of the scene
US6100925A (en) * 1996-11-27 2000-08-08 Princeton Video Image, Inc. Image insertion in video streams using a combination of physical sensors and pattern recognition
US6137491A (en) * 1998-06-05 2000-10-24 Microsoft Corporation Method and apparatus for reconstructing geometry using geometrically constrained structure from motion with points on planes
US6157747A (en) * 1997-08-01 2000-12-05 Microsoft Corporation 3-dimensional image rotation method and apparatus for producing image mosaics
US6259853B1 (en) * 1999-09-03 2001-07-10 Agilent Technologies, Inc. Optical element having electrically controllable refractive index
US20010028399A1 (en) * 1994-05-31 2001-10-11 Conley Gregory J. Array-camera motion picture device, and methods to produce new visual and aural effects
US20010052131A1 (en) * 1999-07-17 2001-12-13 Hobson Gregory L. Digital video recording system
US20020030741A1 (en) * 2000-03-10 2002-03-14 Broemmelsiek Raymond M. Method and apparatus for object surveillance with a movable camera
US20020118969A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for stabilizing rotational images
US20020118286A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for servoing on a moving fixation point within a dynamic scene
US20030076413A1 (en) * 2001-10-23 2003-04-24 Takeo Kanade System and method for obtaining video of multiple moving fixation points within a dynamic scene
US6608923B1 (en) * 1999-06-19 2003-08-19 Microsoft Corporation System and method for rectifying images of three dimensional objects
US6674461B1 (en) * 1998-07-07 2004-01-06 Matthew H. Klapman Extended view morphing

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6133946A (en) * 1998-01-06 2000-10-17 Sportvision, Inc. System for determining the position of an object
WO1999052288A1 (en) 1998-04-02 1999-10-14 Kewazinga Corp. A navigable telepresence method and system utilizing an array of cameras
US7193645B1 (en) 2000-07-27 2007-03-20 Pvi Virtual Media Services, Llc Video system and method of operating a video system
WO2002087218A2 (en) 2001-04-20 2002-10-31 Kewazinga Corp. Navigable camera array and viewer therefore
WO2002096096A1 (en) 2001-05-16 2002-11-28 Zaxel Systems, Inc. 3d instant replay system and method
US20030210329A1 (en) 2001-11-08 2003-11-13 Aagaard Kenneth Joseph Video system and methods for operating a video system

Patent Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5164827A (en) * 1991-08-22 1992-11-17 Sensormatic Electronics Corporation Surveillance system with master camera control of slave cameras
US5489886A (en) * 1992-08-07 1996-02-06 Alos-Officiating Tennis System Limited Automatic line officiating system and method thereof
US6094198A (en) * 1994-01-10 2000-07-25 Cognitens, Ltd. System and method for reconstructing surface elements of solid objects in a three-dimensional scene from a plurality of two dimensional images of the scene
US20010028399A1 (en) * 1994-05-31 2001-10-11 Conley Gregory J. Array-camera motion picture device, and methods to produce new visual and aural effects
US5714997A (en) * 1995-01-06 1998-02-03 Anderson; David P. Virtual reality television system
US5912700A (en) * 1996-01-10 1999-06-15 Fox Sports Productions, Inc. System for enhancing the television presentation of an object at a sporting event
US6084979A (en) * 1996-06-20 2000-07-04 Carnegie Mellon University Method for creating virtual reality
US6100925A (en) * 1996-11-27 2000-08-08 Princeton Video Image, Inc. Image insertion in video streams using a combination of physical sensors and pattern recognition
US5917937A (en) * 1997-04-15 1999-06-29 Microsoft Corporation Method for performing stereo matching to recover depths, colors and opacities of surface elements
US6157747A (en) * 1997-08-01 2000-12-05 Microsoft Corporation 3-dimensional image rotation method and apparatus for producing image mosaics
US6005610A (en) * 1998-01-23 1999-12-21 Lucent Technologies Inc. Audio-visual object localization and tracking system and method therefor
US6137491A (en) * 1998-06-05 2000-10-24 Microsoft Corporation Method and apparatus for reconstructing geometry using geometrically constrained structure from motion with points on planes
US6674461B1 (en) * 1998-07-07 2004-01-06 Matthew H. Klapman Extended view morphing
US6608923B1 (en) * 1999-06-19 2003-08-19 Microsoft Corporation System and method for rectifying images of three dimensional objects
US20010052131A1 (en) * 1999-07-17 2001-12-13 Hobson Gregory L. Digital video recording system
US6259853B1 (en) * 1999-09-03 2001-07-10 Agilent Technologies, Inc. Optical element having electrically controllable refractive index
US20020030741A1 (en) * 2000-03-10 2002-03-14 Broemmelsiek Raymond M. Method and apparatus for object surveillance with a movable camera
US20020118969A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for stabilizing rotational images
US20020118286A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for servoing on a moving fixation point within a dynamic scene
US20030076413A1 (en) * 2001-10-23 2003-04-24 Takeo Kanade System and method for obtaining video of multiple moving fixation points within a dynamic scene

Cited By (103)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020118286A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for servoing on a moving fixation point within a dynamic scene
US20020118969A1 (en) * 2001-02-12 2002-08-29 Takeo Kanade System and method for stabilizing rotational images
US7102666B2 (en) 2001-02-12 2006-09-05 Carnegie Mellon University System and method for stabilizing rotational images
US7027083B2 (en) 2001-02-12 2006-04-11 Carnegie Mellon University System and method for servoing on a moving fixation point within a dynamic scene
US20030076413A1 (en) * 2001-10-23 2003-04-24 Takeo Kanade System and method for obtaining video of multiple moving fixation points within a dynamic scene
US20030185302A1 (en) * 2002-04-02 2003-10-02 Abrams Thomas Algie Camera and/or camera converter
US20030208638A1 (en) * 2002-04-02 2003-11-06 Abrams Thomas Algie Digital production services architecture
US20030185301A1 (en) * 2002-04-02 2003-10-02 Abrams Thomas Algie Video appliance
US20070160142A1 (en) * 2002-04-02 2007-07-12 Microsoft Corporation Camera and/or Camera Converter
US7212574B2 (en) 2002-04-02 2007-05-01 Microsoft Corporation Digital production services architecture
US8310409B2 (en) * 2002-06-05 2012-11-13 Sony Corporation Display device and display method
US20060152435A1 (en) * 2002-06-05 2006-07-13 Tetsujiro Kondo Display device and display method
US20090262185A1 (en) * 2002-06-05 2009-10-22 Tetsujiro Kondo Display apparatus and display method
US20050046626A1 (en) * 2003-09-02 2005-03-03 Fuji Photo Film Co., Ltd. Image generating apparatus, image generating method and image generating program
US7532215B2 (en) * 2003-09-02 2009-05-12 Fujifilm Corporation Image generating apparatus, image generating method and image generating program
US10999487B2 (en) 2004-05-13 2021-05-04 Sony Group Corporation Image capturing system, image capturing device, and image capturing method
US8023817B2 (en) * 2004-05-13 2011-09-20 Sony Corporation Image capturing system, image capturing device, and image capturing method
US9467610B2 (en) 2004-05-13 2016-10-11 Sony Corporation Image capturing system, image capturing device, and image capturing method
US8965195B2 (en) 2004-05-13 2015-02-24 Sony Corporation Image capturing system, image capturing device, and image capturing method
US8787748B2 (en) 2004-05-13 2014-07-22 Sony Corporation Image capturing system, image capturing device, and image capturing method
US20070274705A1 (en) * 2004-05-13 2007-11-29 Kotaro Kashiwa Image Capturing System, Image Capturing Device, and Image Capturing Method
US8369701B2 (en) 2004-05-13 2013-02-05 Sony Corporation Image capturing system, image capturing device, and image capturing method
US9998647B2 (en) 2004-05-13 2018-06-12 Sony Corporation Image capturing system, image capturing device, and image capturing method
US9936170B2 (en) 2004-09-28 2018-04-03 Avigilon Fortress Corporation View handling in video surveillance systems
WO2006037057A2 (en) * 2004-09-28 2006-04-06 Objectvideo, Inc. View handling in video surveillance systems
US20060066722A1 (en) * 2004-09-28 2006-03-30 Objectvideo, Inc. View handling in video surveillance systems
US8497906B2 (en) 2004-09-28 2013-07-30 Objectvideo, Inc. View handling in video surveillance systems
US7733369B2 (en) * 2004-09-28 2010-06-08 Objectvideo, Inc. View handling in video surveillance systems
US20100225760A1 (en) * 2004-09-28 2010-09-09 Object Video, Inc. View handling in video surveillance systems
WO2006037057A3 (en) * 2004-09-28 2009-06-11 Objectvideo Inc View handling in video surveillance systems
US9204107B2 (en) 2004-09-28 2015-12-01 Avigilon Fortress Corporation View handling in video surveillance systems
US20100021152A1 (en) * 2005-02-03 2010-01-28 Gurpal Sandhu Apparatus and method for viewing radiographs
US20070019943A1 (en) * 2005-07-21 2007-01-25 Takahiko Sueyoshi Camera system, information processing device, information processing method, and computer program
US7693413B2 (en) * 2005-07-21 2010-04-06 Sony Corporation Camera system, information processing device, information processing method, and computer program
US7612798B2 (en) * 2005-09-13 2009-11-03 Xanavi Informatics Corporation Photographing system for a moving apparatus
US20070058048A1 (en) * 2005-09-13 2007-03-15 Toshiro Kinugasa Photographing system for a moving apparatus
US20080036864A1 (en) * 2006-08-09 2008-02-14 Mccubbrey David System and method for capturing and transmitting image data streams
US7719568B2 (en) * 2006-12-16 2010-05-18 National Chiao Tung University Image processing system for integrating multi-resolution images
US20080143821A1 (en) * 2006-12-16 2008-06-19 Hung Yi-Ping Image Processing System For Integrating Multi-Resolution Images
US20090284585A1 (en) * 2008-05-15 2009-11-19 Industrial Technology Research Institute Intelligent multi-view display system and method thereof
US20160309139A1 (en) * 2010-08-20 2016-10-20 Canon Kabushiki Kaisha Imaging system and image capturing apparatus
US20120044373A1 (en) * 2010-08-20 2012-02-23 Canon Kabushiki Kaisha Imaging system and image capturing apparatus
US10085010B2 (en) * 2010-08-20 2018-09-25 Canon Kabushiki Kaisha Imaging system for capturing an image of the same object using a plurality of image capturing apparatuses that can communicate with each other
US10560621B2 (en) * 2010-11-19 2020-02-11 Symbol Technologies, Llc Methods and apparatus for controlling a networked camera
US20120127319A1 (en) * 2010-11-19 2012-05-24 Symbol Technologies, Inc. Methods and apparatus for controlling a networked camera
US20130038723A1 (en) * 2011-08-11 2013-02-14 Canon Kabushiki Kaisha Image acquisition apparatus and image processing apparatus
US9076214B2 (en) * 2011-08-11 2015-07-07 Canon Kabushiki Kaisha Image acquisition apparatus and image processing apparatus using selected in-focus image data
US9129657B2 (en) * 2011-09-29 2015-09-08 Teppei Eriguchi Video image display apparatus, video image display method, non-transitory computer readable medium, and video image processing/display system for video images of an object shot from multiple angles
US20150010287A1 (en) * 2011-09-29 2015-01-08 Teppei Eriguchi Video image display device, video image display method, program, and video image processing/display system
US9367221B2 (en) * 2013-09-02 2016-06-14 Ken Lai System and method for sequencing rotatable images
US20150067499A1 (en) * 2013-09-02 2015-03-05 Ken Lai System and Method for Sequencing Rotatable Images
US20150124171A1 (en) * 2013-11-05 2015-05-07 LiveStage°, Inc. Multiple vantage point viewing platform and user interface
US9996977B2 (en) 2014-04-18 2018-06-12 Magic Leap, Inc. Compensating for ambient light in augmented or virtual reality systems
US10115232B2 (en) * 2014-04-18 2018-10-30 Magic Leap, Inc. Using a map of the world for augmented or virtual reality systems
US11205304B2 (en) 2014-04-18 2021-12-21 Magic Leap, Inc. Systems and methods for rendering user interfaces for augmented or virtual reality
US20150302655A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Using a map of the world for augmented or virtual reality systems
US9761055B2 (en) 2014-04-18 2017-09-12 Magic Leap, Inc. Using object recognizers in an augmented or virtual reality system
US9767616B2 (en) 2014-04-18 2017-09-19 Magic Leap, Inc. Recognizing objects in a passable world model in an augmented or virtual reality system
US9766703B2 (en) 2014-04-18 2017-09-19 Magic Leap, Inc. Triangulation of points using known points in augmented or virtual reality systems
US10909760B2 (en) 2014-04-18 2021-02-02 Magic Leap, Inc. Creating a topological map for localization in augmented or virtual reality systems
US9852548B2 (en) 2014-04-18 2017-12-26 Magic Leap, Inc. Systems and methods for generating sound wavefronts in augmented or virtual reality systems
US10846930B2 (en) 2014-04-18 2020-11-24 Magic Leap, Inc. Using passable world model for augmented or virtual reality
US9881420B2 (en) 2014-04-18 2018-01-30 Magic Leap, Inc. Inferential avatar rendering techniques in augmented or virtual reality systems
US9911234B2 (en) 2014-04-18 2018-03-06 Magic Leap, Inc. User interface rendering in augmented or virtual reality systems
US9911233B2 (en) 2014-04-18 2018-03-06 Magic Leap, Inc. Systems and methods for using image based light solutions for augmented or virtual reality
US9922462B2 (en) 2014-04-18 2018-03-20 Magic Leap, Inc. Interacting with totems in augmented or virtual reality systems
US9928654B2 (en) 2014-04-18 2018-03-27 Magic Leap, Inc. Utilizing pseudo-random patterns for eye tracking in augmented or virtual reality systems
US10825248B2 (en) * 2014-04-18 2020-11-03 Magic Leap, Inc. Eye tracking systems and method for augmented or virtual reality
US9972132B2 (en) 2014-04-18 2018-05-15 Magic Leap, Inc. Utilizing image based light solutions for augmented or virtual reality
US9984506B2 (en) 2014-04-18 2018-05-29 Magic Leap, Inc. Stress reduction in geometric maps of passable world model in augmented or virtual reality systems
US10665018B2 (en) 2014-04-18 2020-05-26 Magic Leap, Inc. Reducing stresses in the passable world model in augmented or virtual reality systems
US10262462B2 (en) 2014-04-18 2019-04-16 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US10008038B2 (en) 2014-04-18 2018-06-26 Magic Leap, Inc. Utilizing totems for augmented or virtual reality systems
US10013806B2 (en) 2014-04-18 2018-07-03 Magic Leap, Inc. Ambient light compensation for augmented or virtual reality
US10198864B2 (en) 2014-04-18 2019-02-05 Magic Leap, Inc. Running object recognizers in a passable world model for augmented or virtual reality
US10043312B2 (en) 2014-04-18 2018-08-07 Magic Leap, Inc. Rendering techniques to find new map points in augmented or virtual reality systems
US10186085B2 (en) 2014-04-18 2019-01-22 Magic Leap, Inc. Generating a sound wavefront in augmented or virtual reality systems
US10109108B2 (en) 2014-04-18 2018-10-23 Magic Leap, Inc. Finding new points by render rather than search in augmented or virtual reality systems
US10127723B2 (en) 2014-04-18 2018-11-13 Magic Leap, Inc. Room based sensors in an augmented reality system
US10115233B2 (en) 2014-04-18 2018-10-30 Magic Leap, Inc. Methods and systems for mapping virtual objects in an augmented or virtual reality system
EP3175424A4 (en) * 2014-12-23 2017-08-02 Huawei Technologies Co. Ltd. Computational multi-camera adjustment for smooth view switching and zooming
CN107111864A (en) * 2014-12-23 2017-08-29 华为技术有限公司 The computational polyphaser adjustment for switching for smooth view and scaling
US20160227128A1 (en) * 2015-01-29 2016-08-04 Electronics And Telecommunications Research Institute Multi-camera control apparatus and method to maintain location and size of object in continuous viewpoint switching service
US9786064B2 (en) * 2015-01-29 2017-10-10 Electronics And Telecommunications Research Institute Multi-camera control apparatus and method to maintain location and size of object in continuous viewpoint switching service
US9242379B1 (en) * 2015-02-09 2016-01-26 The Trustees Of The University Of Pennysylvania Methods, systems, and computer readable media for producing realistic camera motion for stop motion animation
WO2016138507A1 (en) * 2015-02-27 2016-09-01 Leia Inc. Multiview camera
US10484662B2 (en) 2015-02-27 2019-11-19 Leia Inc. Multiview camera, multiview imaging system, and method of multiview image capture
US10029180B2 (en) 2015-03-11 2018-07-24 Nintendo Co., Ltd. Storage medium having stored therein game program, game apparatus, game system, and game processing method
US9878244B2 (en) * 2015-03-11 2018-01-30 Nintendo Co., Ltd. Storage medium having stored therein game program, game apparatus, game system, and game processing method
US20160263479A1 (en) * 2015-03-11 2016-09-15 Nintendo Co., Ltd. Storage medium having stored therein game program, game apparatus, game system, and game processing method
WO2016203584A1 (en) * 2015-06-17 2016-12-22 オリンパス株式会社 Image pickup control device, image pickup system, image pickup control method, and program
US10133157B2 (en) 2015-06-17 2018-11-20 Olympus Corporation Imaging control device, imaging system, imaging control method, and recording medium
US10636121B2 (en) 2016-01-12 2020-04-28 Shanghaitech University Calibration method and apparatus for panoramic stereo video system
US10643305B2 (en) 2016-01-12 2020-05-05 Shanghaitech University Compression method and apparatus for panoramic stereo video system
WO2017120802A1 (en) * 2016-01-12 2017-07-20 Shanghaitech University Stitching method and apparatus for panoramic stereo video system
US10489886B2 (en) 2016-01-12 2019-11-26 Shanghaitech University Stitching method and apparatus for panoramic stereo video system
US20190066376A1 (en) * 2017-08-31 2019-02-28 Hs R & A Co., Ltd. Method for interpreting layout of tube by using three-dimensional coordinates and recording medium thereof
US11310478B2 (en) 2017-10-02 2022-04-19 Leia Inc. Multiview camera array, multiview system, and method having camera sub-arrays with a shared camera
US20220224822A1 (en) * 2018-10-24 2022-07-14 Sony Corporation Multi-camera system, control value calculation method, and control apparatus
US11277596B2 (en) * 2018-10-26 2022-03-15 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and storage medium
US11030733B2 (en) * 2018-12-24 2021-06-08 Beijing Dajia Internet Information Technology Co., Ltd. Method, electronic device and storage medium for processing image
WO2020247906A1 (en) * 2019-06-06 2020-12-10 Ovad Custom Stages, Llc Vehicle photographic system for identification of surface imperfections
US11138760B2 (en) * 2019-11-06 2021-10-05 Varjo Technologies Oy Display systems and methods for correcting drifts in camera poses

Also Published As

Publication number Publication date
US7102666B2 (en) 2006-09-05
WO2002065763A3 (en) 2002-12-12
WO2002065761A3 (en) 2002-12-19
WO2002065761A2 (en) 2002-08-22
US7106361B2 (en) 2006-09-12
US20020118969A1 (en) 2002-08-29
WO2002065763A2 (en) 2002-08-22

Similar Documents

Publication Publication Date Title
US20020145660A1 (en) System and method for manipulating the point of interest in a sequence of images
US7027083B2 (en) System and method for servoing on a moving fixation point within a dynamic scene
US5949433A (en) Processing image data
US5594845A (en) Method and device for processing an image in order to construct a target image from a plurality of contiguous source images
US20030076413A1 (en) System and method for obtaining video of multiple moving fixation points within a dynamic scene
US6738073B2 (en) Camera system with both a wide angle view and a high resolution view
CN1965578B (en) Digital 3D/360 degree camera system
US8243122B2 (en) Video method for generating free viewpoint video image using divided local regions
US20020063711A1 (en) Camera system with high resolution image inside a wide angle view
US20020075258A1 (en) Camera system with high resolution image inside a wide angle view
US10545215B2 (en) 4D camera tracking and optical stabilization
US6839081B1 (en) Virtual image sensing and generating method and apparatus
McGarrity et al. A method for calibrating see-through head-mounted displays for AR
CN110517209A (en) Data processing method, device, system and computer readable storage medium
JP2023546739A (en) Methods, apparatus, and systems for generating three-dimensional models of scenes
JP4960941B2 (en) Camera calibration device for zoom lens-equipped camera of broadcast virtual studio, method and program thereof
JP2022500793A (en) How to calibrate a multi-degree-of-freedom movable visual system
CN113687627B (en) Target tracking method based on camera robot
CN112312041B (en) Shooting-based image correction method and device, electronic equipment and storage medium
JP5075659B2 (en) Object trajectory calculation device and program thereof
KR100234196B1 (en) Method for generating height map using 2-dimensional video image
GB2312125A (en) Virtual studio with zoom control
JP4135934B2 (en) Electronics
US20230360333A1 (en) Systems and methods for augmented reality video generation
CN113487677A (en) Outdoor medium and long distance scene calibration method of multiple PTZ cameras based on any distributed configuration

Legal Events

Date Code Title Description
AS Assignment

Owner name: CARNEGIE MELLON UNIVERSITY, PENNSYLVANIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KANADE, TAKEO;COLLINS, ROBERT;REEL/FRAME:012890/0936;SIGNING DATES FROM 20020412 TO 20020415

STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: PAT HOLDER NO LONGER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: STOL); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

REMI Maintenance fee reminder mailed
FPAY Fee payment

Year of fee payment: 8

SULP Surcharge for late payment

Year of fee payment: 7

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553)

Year of fee payment: 12