US9609373B2 - Presentation timeline synchronization across audio-video (AV) streams - Google Patents
Presentation timeline synchronization across audio-video (AV) streams Download PDFInfo
- Publication number
- US9609373B2 US9609373B2 US14/148,625 US201414148625A US9609373B2 US 9609373 B2 US9609373 B2 US 9609373B2 US 201414148625 A US201414148625 A US 201414148625A US 9609373 B2 US9609373 B2 US 9609373B2
- Authority
- US
- United States
- Prior art keywords
- streams
- event
- video
- user
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 claims description 48
- 238000003860 storage Methods 0.000 claims description 47
- 230000005540 biological transmission Effects 0.000 claims description 13
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims 10
- 230000000875 corresponding effect Effects 0.000 description 38
- 230000008569 process Effects 0.000 description 37
- 238000005516 engineering process Methods 0.000 description 17
- 230000009471 action Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 230000002596 correlated effect Effects 0.000 description 8
- 230000000153 supplemental effect Effects 0.000 description 5
- 238000009826 distribution Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000006855 networking Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000004065 semiconductor Substances 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 241000258963 Diplopoda Species 0.000 description 1
- 241000510009 Varanus griseus Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/34—Indicating arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N21/2353—Processing of additional data, e.g. scrambling of additional data or processing content descriptors specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/23614—Multiplexing of additional data and video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2365—Multiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Definitions
- live events such as live sporting events
- the owner/provider of the live event or a production entity associated therewith, selects which camera feed is presented to viewers at any given time.
- the owner/provider may also determine an audio feed to provide with the camera feed, e.g. in the form of audio commentary from announcers selected by the owner/provider.
- the owner/provider may determine whether the camera feed should be augmented with any additional information, such as graphical overlays.
- viewers of a live event may only have the option of viewing/hearing an audio-video (AV) stream that is produced by the owner/provider of the live event.
- AV audio-video
- FIG. 1 illustrates an example network environment in which a system for presentation timeline synchronization across AV streams may be implemented in accordance with one or more implementations.
- FIG. 2 illustrates an example timing diagram of a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- FIG. 3 illustrates an example user interface of a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- FIG. 4 illustrates a flow diagram of an example process for creating a marked point-in-time in a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- FIG. 5 illustrates a flow diagram of an example process for retrieving a marked point-in-time in a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- FIG. 6 illustrates a flow diagram of an example process of a user device in a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- Live streaming technologies may allow an owner/provider of a live event to provide multiple different AV streams to viewers of a live event.
- the owner/provider of a live event may authorize multiple individual content producers to generate AV streams for the live event.
- the content producers may be allowed to select from any of the available camera feeds (not just the camera feed selected by the owner/provider of the live event), and the content producers may provide their own audio feed, e.g. commentary, that supplements, or replaces, the audio feed provided by the owner/provider of the live event.
- the content producers may further supplement the video stream provided by a selected camera feed with supplemental video content, such as an overlay and/or a supplemental video stream.
- FIG. 1 illustrates an example network environment 100 in which a system for presentation timeline synchronization across AV streams may be implemented in accordance with one or more implementations. Not all of the depicted components may be required, however, and one or more implementations may include additional components not shown in the figure. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.
- the example network environment 100 includes a network 108 , a metadata server 116 and an associated data store 118 , an audio video (AV) stream storage server 120 and an associated data store 122 , a video stream server 124 and an associated data store 126 , video cameras 128 A-C that may record a live event 130 , a primary content producing (CP) device 132 , alternative content producing (CP) devices 142 A-C, a set-top device 110 , an output device 114 , and a secondary device 112 .
- CP primary content producing
- CP alternative content producing
- the secondary device 112 can be one or more computing devices, such as laptop or desktop computers, smartphones, tablet devices, or other displays with one or more processors coupled thereto and/or embedded therein.
- the secondary device 112 is depicted as a smart phone.
- the set-top device 110 and/or the secondary device 112 may be referred to as a user device or a client device.
- the servers 116 , 120 , 124 may each be individual computing devices such as computer servers, and/or may all be a single computing device.
- the servers 116 , 120 , 124 may represent one or more computing devices (such as a cloud of computers and/or a distributed system) that are communicatively coupled, such as communicatively coupled over the network 108 , that collectively, or individually, perform one or more functions that can be performed server-side.
- the servers 116 , 120 , 124 may each be coupled with various databases, storage services, or other computing devices, such as the data stores 118 , 122 , 126 , respectively.
- the data stores 118 , 122 , 126 may each include one or more storage devices, such as hard drives.
- the video cameras 128 A-C may generate different video streams from the live event 130 .
- the video cameras 128 A-C may be positioned at different angles relative to the live event 130 and/or may be located at different locations relative to the live event 130 .
- the live event 130 may be any live event, such as a sporting event, a music event, a television show recorded live, such as a game show, etc.
- the live event 130 may occur at a geographic location, and the video cameras 128 A-C may be located at, or proximal to, the geographic location.
- the video cameras 128 A-C may be communicably coupled to the video stream server 124 .
- the video cameras 128 A-C may be synchronized, such as by the video stream server 124 , such that the video cameras 128 A-C may insert common reference clock timestamps into the video streams that are correlated to a common time base, such as a starting time of the live event 130 .
- the common reference clock timestamps may be inserted as fields in transport stream packet headers of the video streams and/or as time marker packets that are inserted into the video streams.
- the video stream server 124 may synchronize the video cameras 128 A-C based on a common reference clock maintained by the video stream server 124 that has a time base corresponding to the start of the live event 130 .
- the video cameras 128 A-C may also insert video camera identifiers into the generated video streams, e.g. as a field inserted into transport stream packet headers and/or as inserted video camera identifier packets.
- the video stream server 124 may maintain a common reference clock having a time base corresponding to the start of the live event 130 .
- the video stream server 124 may receive the video streams from the video cameras 128 A-C and may insert common reference clock timestamps based on the common reference clock into the video streams, e.g. as fields inserted into transport stream packet headers and/or by inserting common timestamp packets into the video streams. Since the transmission latency between the video cameras 128 A-C and the video stream server 124 may be minimal and/or generally equivalent across the video cameras 128 A-C, the video stream server 124 may be able to uniformly insert common reference clock timestamps into the video streams.
- the video stream server 124 may insert video camera identifiers into the video streams generated by the video cameras 128 A-C, e.g. as a field inserted into transport stream packet headers and/or as inserted video camera identifier packets.
- the primary CP device 132 may be associated with an owner/provider of the live event, such as a football league for a football game, and/or a production entity associated therewith.
- the primary CP device 132 may select one or more of the video streams from the video stream server 124 and may produce an AV stream that includes the selected video stream, an audio stream, such as audio commentary by one or more announcers selected by the owner/provider of the live event 130 , and/or additional content, such as an overlay.
- the AV stream generated by the primary CP device 132 for a live event 130 may be referred to as a primary AV stream for the live event 130 .
- the AV stream generated by the primary CP device 132 may be provided for transmission to the set-top device 110 and/or the secondary device 112 , via one or more content distribution channels of the network 108 .
- the AV stream generated by the primary CP device 132 may be provided to the set-top device 110 via a television channel of a television network.
- the AV stream generated by the primary CP device 132 may also be transmitted to the AV stream storage server 120 , and the AV stream may be stored, including any inserted timestamps and/or identifiers, by the AV stream storage server 120 , such as in the data store 122 .
- the AV stream may be stored with an AV stream identifier, an event identifier, an identifier of the primary CP device 132 , and/or an identifier of an entity associated therewith, such as the owner/provider of the live event 130 .
- the alternative CP devices 142 A-C may be associated with one or more individuals authorized by the owner/entity of the live event 130 to generate, or produce, an AV stream for the live event 130 .
- the individuals may be, for example, former or current players of a sport corresponding to the live event 130 , former or current officials or referees for a sport corresponding to the live event 130 , persons of pop culture significance, political figures, or generally any individuals who may generally offer a different perspective of the live event 130 than the perspective conveyed by the AV stream generated by the primary CP device 132 .
- the alternative CP devices 142 A-C may each select one or more of the video streams from the video stream server 124 and may each produce an AV stream that includes the selected video stream and an audio stream, such as audio commentary by the authorized individuals, and/or additional content, such as an overlay, additional video content, etc.
- the alternative CP devices 142 A-C may select one of the video streams generated by the cameras 128 A-C, e.g. a video stream corresponding to an angle or position of one of the cameras 128 A-C that is most interesting to the authorized individuals.
- the video stream selected by one or more of the alternative CP devices 142 A-C may not coincide with the video stream selected by the primary CP device 132 , and/or may not coincide with the other alternative CP devices 142 A-C.
- the source video stream included in the AV stream generated by the primary CP device 132 may be different than the source video stream included in the AV stream generated by one or more of the alternative CP devices 142 A-C.
- Example primary AV and alternative AV streams that include different source video streams at different points-in-time are discussed further below with respect to FIG. 2 .
- the alternative CP devices 142 A-C may change video streams as frequently, or as often, as is desirable to the authorized individuals.
- the video streams of the AV streams generated by the alternative CP devices 142 A-C may include concatenated video streams generated by multiple different video cameras 128 A-C.
- the video streams of the AV streams generated by the alternative CP devices 142 A-C may include any common reference clock timestamps and/or video camera identifiers previously inserted into the individual source video streams, e.g. by the video cameras 128 A-C and/or the video stream server 124 .
- the alternative CP devices 142 A-C may insert video camera identifiers associated with the video stream selected by the alternative CP devices 142 A-C into the generated AV streams, e.g. as fields inserted into the transport stream packet headers.
- the AV streams generated by the alternative CP devices 142 A-C for a live event 130 may be referred to as alternative AV streams for the live event 130 .
- the alternative AV streams generated by the alternative CP devices 142 A-C may be provided for transmission to the set-top device 110 and/or the secondary device 112 via one or more content distribution channels of the network 108 , e.g. when requested by the set-top device 110 and/or the secondary device 112 .
- the set-top device 110 and/or the secondary device 112 may provide the user with an option to view one or more of the AV streams generated by the alternative CP devices 142 A-C for the live event 130 .
- an application executing on the set-top device 110 and/or the secondary device 112 may search one of the data stores 118 , 122 , 126 , such as the data store 122 , to identify alternative AV streams available for the live event 130 .
- the user may interact with the set-top device 110 and/or the secondary device 112 to select one of the alternative AV streams.
- the alternative AV stream may then be provided to the set-top device 110 and/or the secondary device 112 , e.g. in place of and/or in combination with, the primary AV stream for the live event 130 .
- the AV streams generated by the alternative CP devices 142 A-C may also be transmitted to the AV stream storage server 120 , and may be stored, including any inserted timestamps and/or identifiers, by the AV stream storage server 120 , such as in the data store 122 .
- the AV streams may be stored with an AV stream identifier, e.g. a unique identifier generated for each AV stream, an event identifier of the live event 130 , an identifier of the corresponding alternative CP device 142 A, and/or an identifier of the authorized individual associated therewith.
- the metadata server 116 may store, e.g. in the data store 118 , metadata items that include information corresponding to, e.g., the live event 130 , the AV stream generated by the primary CP device 132 and/or the entity associated therewith, the AV streams generated by the alternative CP devices 142 A-C and/or the authorized individuals associated therewith, the set-top device 110 , the secondary device 112 , and/or the user associated therewith.
- a metadata item may include an event identifier of a live event 130 and a timing indicator that is indicative of a point-in-time during the live event 130 , e.g. relative to the common reference clock.
- the timing indicator may be, or may include, a common reference clock timestamp, one or more frames of a video stream associated with the point-in-time, and/or statistical information, e.g. fingerprint information, corresponding to one or more frames associated with the point-in-time.
- the metadata item may further include one or more fields that may be indicative of an action that occurred at the point-in-time indicated by the metadata item.
- a user associated with the set-top device 110 and/or the secondary device 112 may mark a point-in-time while viewing the live event 130 .
- the generated metadata item may include an identifier of the user, and/or an identifier of the AV stream being viewed by the user at the marked point-in-time.
- the owner/provider of the live event, or an entity associated therewith may also generate metadata items that relate to action events occurring during the live event 130 , such as plays of a football game.
- a metadata item may further include a description of an action event, e.g. a play, that occurred at the point-in-time indicated by the metadata item.
- a user viewing one of the AV streams associated with a live event 130 via the set-top device 110 may mark a point-in-time while viewing the AV stream, e.g. during or after the live event 130 , via the set-top device 110 and/or the secondary device 112 .
- the set-top device 110 and/or the secondary device 112 may generate a metadata item corresponding to the marked point-in-time.
- the metadata item may include an identifier of the live event 130 , an identifier of the user, a timing indicator, such as a common reference timestamp retrieved from a transport stream packet corresponding to the marked point-in-time, one or more frames of the AV stream being viewed by the user, and/or statistical information describing the same, and/or an identifier of the video camera that generated the source video stream included in the AV stream at the marked point-in-time.
- a timing indicator such as a common reference timestamp retrieved from a transport stream packet corresponding to the marked point-in-time, one or more frames of the AV stream being viewed by the user, and/or statistical information describing the same, and/or an identifier of the video camera that generated the source video stream included in the AV stream at the marked point-in-time.
- the metadata item may be transmitted to the metadata server 116 . If the metadata item does not include a common reference clock timestamp, the metadata server 116 may correlate the timing indicator (e.g. the one or more frames and/or statistical information describing the same) with one or more frames of a video stream of one of the AV streams stored by the AV stream storage server 120 and/or one or more frames of a video stream stored by the video stream server 124 . In one or more implementations, if the metadata item includes the event identifier and a video camera identifier, the metadata server 116 may search for the one or more frames in the video stream corresponding to the video camera identifier, e.g. as stored in the data store 126 .
- the timing indicator e.g. the one or more frames and/or statistical information describing the same
- the metadata server 116 may search for the one or more frames in the AV stream corresponding to the AV stream identifier, e.g. as stored in the data store 122 . In one or more implementations, if the metadata item does not include an AV stream identifier or a video camera identifier, the metadata server 116 may search for the one or more frames in the AV streams associated with the live event 130 and/or the video streams associated with the live event 130 , e.g. one-by-one. In one or more implementations, the metadata server 116 may provide the one or more frames, and/or statistical information describing the same, to the video stream server 124 , the AV stream storage server 120 , and/or another server, to perform the search.
- the metadata server 116 may determine a common reference clock timestamp for the marked point-in-time, e.g. based at least on the common reference timestamp associated with the matched one or more frames. Since the video streams may be stored by the video stream server 124 and/or the AV streams may be stored by the AV stream storage server 120 with the inserted common reference clock timestamps, the metadata server 116 may be able to retrieve the common reference clock timestamp associated with the matching one or more frames.
- the metadata server 116 may retrieve other timing information associated with the one or more frames, such as a presentation time stamp (PTS), a program clock reference (PCR), etc.
- the metadata server 116 may insert the common reference clock timestamp (or other timing information) into the metadata item and may store the metadata item, e.g. in the data store 118 .
- the metadata server 116 may retrieve the corresponding one or more frames, and may insert the corresponding one or more frames into the metadata item. In one or more implementations, the metadata server 116 may search other metadata items associated with the marked point-in-time, and/or associated with a time within a temporal threshold of the marked point-in-time, to identify additional information regarding the marked point-in-time.
- the metadata server 116 may identify a metadata item corresponding to an action event that occurred at the marked point-in-time, and/or temporally proximal to the marked point-in-time.
- the metadata server 116 may insert information describing the action event into the metadata item for the marked point-in-time, such as a description of a play that occurred at the marked point-in-time.
- the metadata server 116 may interface with, e.g. monitor, one or more applications running on the secondary device 112 of the user, such as social networking applications, messaging applications, etc., to allow the user to mark a point in time via these applications.
- the user may generate a message on a social network application that includes an identifier from which the live event 130 can be determined, e.g. a hash tag corresponding to the live event 130 .
- the subject system may monitor the social networking application on the secondary device 112 of the user and/or a publicly available social networking feed associated with the viewer (e.g. without accessing the social networking application on the secondary device 112 ) to determine when such a message has been generated by the user.
- the metadata server 116 may then generate a metadata item that includes an identifier of the live event 130 , an identifier of the user, and/or a timing indicator from which the marked point-in-time can be determined, such as a common reference clock timestamp of an AV stream (or one or more frames thereof) that is being presented to the user, e.g. via the set-top device 110 .
- the metadata server 116 may request the common reference clock timestamp from the set-top device 110 when the metadata server 116 determines that the user marked a point-in-time via the secondary device 112 .
- the user may retrieve the marked point-in-time and may view any of the AV streams that were generated for the live event 130 at the marked point-in-time. Since the source video streams included in the AV streams are synchronized with a common reference clock, the marked point-in-time may be correlatable to any of the AV streams generated for the live event 130 .
- An example user interface for retrieving a marked point-in-time during a live event 130 , and for retrieving the available AV streams corresponding thereto, is discussed further below with respect to FIG. 3 .
- An example process of the metadata server 116 for retrieving AV streams available at a marked point-in-time is discussed further below with respect to FIG.
- FIG. 5 An example process of a device associated with the user, such as the set-top device 110 and/or the secondary device 112 , for marking points-in-time and subsequently retrieving AV streams available at a marked point-in-time is discussed further below with respect to FIG. 6 .
- FIG. 2 illustrates an example timing diagram of a system for presentation timeline synchronization across AV streams in accordance with one or more implementations. Not all of the depicted components may be required, however, and one or more implementations may include additional components not shown in the figure. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.
- the timing diagram includes a primary AV stream 232 generated by a primary CP device 132 , alternative AV streams 242 A-C generated by alternative CP devices 142 A-B, and marked points-in-time 210 A-C.
- the primary AV stream 232 includes a video stream 234 and an audio stream 236 .
- the alternative AV streams 242 A-C include video streams 244 A-C and audio streams 246 A-C.
- the marked points-in-time may have been marked by a user interacting with the set-top device 110 and/or the secondary device 112 , e.g. while viewing one of the AV streams 232 , 242 A-C.
- the video streams 234 , 244 A-C may be a concatenation of source video streams generated by the video cameras 128 A-C.
- VS_A may refer to a source video stream A that was generated by the video camera 128 A
- VS_B may refer to a source video stream B that was generated by the video camera 128 B
- VS_C may refer to a source video stream C that was generated by the video camera 128 C.
- the video streams 234 , 244 A-C of the AV streams 232 , 242 A-C may further include overlays, such as graphical overlays, e.g. as shown in the video stream 244 B of the alternative AV stream 242 B, and/or additional content, such as supplemental video streams, e.g. as shown in the video stream 244 C of the alternative AV stream 242 C.
- the user may have marked the marked point-in-time 210 A while viewing the primary AV stream 232 which, at the marked point-in-time 210 A, included the source video stream C generated by the video camera 128 C, as did the alternative AV stream 242 C.
- the alternative AV stream 242 A included the source video stream B generated by the video camera 128 B
- the alternative AV stream 242 B included the source video stream A generated by the video camera 128 A.
- the common reference clock timestamp may be retrieved by the set-top device 110 from the primary AV stream 232 being presented to the user.
- the set-top device 110 may transmit one or more frames of the primary AV stream 232 that were presented at the marked point-in-time 210 A to the metadata server 116 , e.g. as a timing indicator of a metadata item generated for the marked point-in-time 210 A.
- the metadata server 116 may match the one or more frames to the primary AV stream 232 , e.g. as stored by the AV stream storage server 120 , or to the source video stream C generated by the video camera 128 C, e.g. as stored by the video stream server 124 , to identify the common reference clock timestamp corresponding to the marked point-in-time.
- the example user interface 300 may include a viewed events menu 302 , a marked points-in-time menu 308 , an available AV streams menu 330 and a main display area 350 .
- the example user interface 300 may be presented on the output device 114 , e.g. by the set-top device 110 , on the secondary device 112 , or generally on any device.
- the viewed events menu 302 may include representations 304 A-C of events
- the marked points-in-time menu 308 may include representations 310 A-C of marked points-in-time 210 A-C
- the available AV streams menu 330 may include representations 332 , 342 A-C of available AV streams 232 , 242 A-C.
- the events menu 302 may appear in response to a request therefor, e.g. by a user, and the menus 308 , 330 may successively appear as selections are made on the menus 302 , 308 , respectively.
- the main display area 350 may be reduced in size to accommodate display of the menus 302 , 308 , 330 , or the main display area 350 may extend behind the menus 302 , 308 , 330 .
- the events menu 302 may present representations 304 A-C of events that were previously viewed by the user.
- the representations 304 A-C may include one or more frames of the previously viewed events that may be presented to the user, e.g. the representations 304 A-C may display single still frames, or a video stream that includes multiple frames.
- the user may log in, via the set-top device 110 , to a system hosted by, e.g., the metadata server 116 , to access events previously viewed on any device associated with the user, such as the set-top device 110 and/or the secondary device 112 .
- the set-top device 110 may transmit an identifier of the user to the metadata server 116 and the metadata server 116 may retrieve and transmit the representations 304 A-C to the set-top device 110 .
- the set-top device 110 may store the events previously viewed via the set-top device 110 .
- the user may select one of the representations 304 A-C of the viewed events menu 302 , such as the representation 304 C in FIG. 3 .
- the set-top device 110 may retrieve metadata items corresponding to marked points-in-time 210 A-C during the selected event and may display representations 310 A-C of the marked points-in-time 210 A-C via the marked points-in-time menu 308 .
- the set-top device 110 may transmit an identifier of the selected event, and the user identifier, to the metadata server 116 and the metadata server 116 may retrieve, and transmit to the set-top device 110 , any metadata items corresponding to points-in-time 210 A-C marked by the user during the selected event.
- the metadata items may include one or more frames corresponding to the marked points-in-time 210 A-C.
- the representations 310 A-C of the marked points-in-time 210 A-C may display still frames or video streams corresponding to the AV streams that were presented to the user at the marked points-in-time 210 A-C.
- the representations 310 A-C may include at least a portion of the descriptions, or the descriptions may be displayed when the representations 310 A-C are highlighted or otherwise selected by the user.
- the user may select one of the representations 310 A-C of the marked points-in-time 210 A-C, such as the representation 310 B in FIG. 3 .
- the set-top device 110 may retrieve representations 332 , 342 A-C of AV streams 232 , 242 A-C that were available at the selected marked point-in-time 210 B.
- the set-top device 110 may transmit a metadata item corresponding to the selected marked point-in-time 210 B, an identifier of the selected marked point-in-time 210 B, and/or a common reference clock timestamp corresponding to the selected marked point-in-time 210 B, to the metadata server 116 and the metadata server 116 may retrieve, and transmit to the set-top device 110 , representations 332 , 342 A-C of the AV streams 232 , 242 A-C that were available at the marked point-in-time 210 B.
- the representations 332 , 342 A-C may include one or more frames of the AV streams 232 , 242 A-C corresponding to the selected marked point-in-time 210 B.
- the representations 332 , 342 A-C of the AV streams 232 , 242 A-C may display still frames or video streams corresponding to the marked point-in-time 210 B.
- the representation 342 B of the alternative AV stream 242 B includes an overlay 344 and the representation 342 C of the alternative AV stream 242 C includes supplemental content 346 , such as a supplemental video stream.
- the representations 332 , 342 A-C of the AV streams 232 , 242 A-C may also include an identifier, such as a name, of the individual or entity that generated/produced the corresponding AV streams 232 , 242 A-C.
- the user may select one of the representations 332 , 342 A-C of the AV streams 232 , 242 A-C that were available at the selected marked point-in-time 210 B, such as the representation 332 of the primary AV stream 232 generated by the primary CP device 132 .
- the set-top device 110 may transmit an indication of the selection to the metadata server 116 and/or the AV stream storage server 120 .
- the metadata server 116 may configure the AV stream storage server 120 to provide the selected primary AV stream 232 beginning at the selected marked point-in-time 210 B.
- FIG. 4 illustrates a flow diagram of an example process 400 for creating a marked point-in-time in a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- the example process 400 is primarily described herein with reference to the metadata server 116 of FIG. 1 ; however, the example process 400 is not limited to the metadata server 116 of FIG. 1 , and the example process 400 may be performed by one or more components of the metadata server 116 .
- the blocks of the example process 400 are described herein as occurring in serial, or linearly. However, multiple blocks of the example process 400 may occur in parallel.
- the blocks of the example process 400 need not be performed in the order shown and/or one or more of the blocks of the example process 400 need not be performed.
- the metadata server 116 identifies video streams associated with the live event 130 ( 406 ). For example, the metadata server 116 may identify video streams generated by the video cameras 128 A-C for the live event 130 and stored by the video stream server 124 , e.g. based on the event identifier, and/or the metadata server 116 may identify AV streams (that include video streams) generated for the live event 130 and stored by the AV stream storage server 120 , e.g. based on the event identifier.
- the metadata server 116 may correlate the timing indicator of the received metadata item to one of the identified video streams ( 408 ).
- the received metadata item may include a video camera identifier, and/or a video stream identifier, that can be correlated to one of the video streams stored by the video stream server 124 and/or an AV stream identifier that can be correlated to one of the AV streams stored by the AV stream storage server 120 .
- the metadata server 116 determines whether an action event description is available for the common reference clock timestamp ( 414 ). For example, the metadata server 116 may search for other metadata items that include a common reference clock timestamp that is within a temporal threshold of the common reference clock timestamp of the metadata item for the marked point-in-time. For example, an owner/provider of the live event 130 , or a third party entity, may generate metadata items that describe action events during the live event 130 , such as plays within a sporting event. The metadata items may include a common reference clock timestamp and may include a description of an action event that was occurring during the live event 130 at the corresponding time.
- the metadata server 116 determines that an action event description is available for the common reference clock timestamp ( 414 )
- the metadata server 116 adds at least a portion of the action event description to the metadata item for the marked point-in-time ( 416 ) and stores the metadata item ( 418 ), e.g. in the data store 118 .
- the metadata server 116 determines that no action event descriptions are available for the common reference clock timestamp ( 414 )
- the metadata server 116 stores the metadata item ( 418 ), e.g. in the data store 118 .
- the metadata server 116 provides the representations 304 A-C of the previously viewed events to the set-top device 110 ( 504 ).
- the previously viewed events may only include events for which the user marked a point-in-time, and consequently metadata for the marked point-in-time was generated and stored, e.g. in the data store 118 .
- the metadata server 116 receives a selection of one of the representations 304 A-C of the previously viewed events that were provided to the set-top device 110 ( 506 ).
- the set-top device 110 may present the representations 304 A-C, e.g. via user interface 300 , and the user may select one of the presented representations 304 A-C.
- the selection received by the metadata server 116 may include an event identifier of the selected event.
- the metadata server 116 retrieves metadata items for points-in-time marked 210 A-C by the user for the selected event ( 508 ).
- the metadata server 116 may search, e.g.
- the metadata server 116 may retrieve representations 310 A-C of the marked points-in-time 210 A-C from the metadata items, e.g. the one or more frames included in the metadata items and/or any action event descriptions included in the metadata items.
- the metadata server 116 provides the representations 310 A-C of the marked points-in-time 210 A-C( 510 ), e.g. to the set-top device 110 and/or the secondary device 112 associated with the user.
- the metadata server 116 receives a selection of one of the representations 310 A-C of the marked points-in-time 210 A-C( 512 ), e.g. from the set-top device 110 and/or the secondary device 112 .
- the metadata server 116 determines the AV streams that are available for the event at the marked point-in-time ( 514 ). For example, the metadata server 116 may search the data store 122 of the AV stream storage server 120 for AV streams that are associated with the event identifier.
- the metadata server 116 may determine which of the AV streams include a video stream and/or an audio stream for the marked point-in-time, e.g. based at least on the common reference clock timestamp corresponding to the marked point-in-time.
- the metadata server 116 may retrieve one or more frames from the available AV streams corresponding to the marked point-in-time, and may provide the one or more frames for each of the AV streams 232 , 242 A-C as representations 332 , 342 A-C of the AV streams 232 , 242 A-C( 516 ). In one or more implementations, the metadata server 116 may also provide an identifier, such as a name, of the individual or entity that generated/produced each of the AV streams 232 , 242 A-C. The metadata server 116 receives a selection of one of the representations 332 , 342 A-C of the AV streams 232 , 242 A-C( 518 ), e.g. from the set-top device 110 and/or the secondary device 112 .
- the metadata server 116 facilitates transmission of the selected AV stream to a device associated with the user, such as the set-top device 110 and/or the secondary device 112 ( 520 ).
- the metadata server 116 may transmit a message to the AV stream storage server 120 to transmit the selected AV stream, beginning at the marked point-in-time, to the set-top device 110 and/or the secondary device 112
- the metadata server 116 may retrieve the selected AV stream from the data store 122 and may forward the AV stream to the set-top device 110 and/or the secondary device 112
- the metadata server 116 may transmit an identifier, such as a network identifier, for accessing the AV stream from the data store 122 , beginning at the marked point-in-time, to the set-top device 110 and/or the secondary device 112 .
- FIG. 6 illustrates a flow diagram of an example process 600 of a user device in a system for presentation timeline synchronization across AV streams in accordance with one or more implementations.
- the example process 600 is primarily described herein with reference to the set-top device 110 and/or the secondary device 112 of FIG. 1 ; however, the example process 600 is not limited to the set-top device 110 and/or secondary device 112 of FIG. 1 , and the example process 600 may be performed by one or more components of the set-top device 110 and/or the secondary device 112 .
- the blocks of the example process 600 are described herein as occurring in serial, or linearly. However, multiple blocks of the example process 600 may occur in parallel.
- the blocks of the example process 600 need not be performed in the order shown and/or one or more of the blocks of the example process 600 need not be performed.
- the set-top device 110 and/or the secondary device 112 may receive indications of marked points-in-time for an event, such as the live event 130 , from a user interacting with the set-top device 110 and/or the secondary device 112 ( 602 ). For example, the user may use a remote control to interact with the set-top device 110 to mark a point-in-time.
- the set-top device 110 and/or the secondary device 112 generates a metadata item corresponding to the marked point-in-time and transmits the metadata item to the metadata server 116 ( 604 ).
- the set-top device 110 may generate a metadata item that includes the event identifier, the user identifier, a timing indicator, a video camera identifier and/or an AV stream identifier.
- the secondary device 112 may generate a metadata item that includes the user identifier and the event identifier.
- the secondary device 112 may request one or more of the event identifier, the timing indicator, the video camera identifier, and/or the AV stream identifier from the set-top device 110 , e.g. via a local area network.
- the secondary device 112 may transmit a metadata item that includes the user identifier, or the user identifier and the event identifier, to the metadata server 116 and the metadata server 116 may request one or more of the event identifier, the timing indicator, the video camera identifier, and/or the AV stream identifier from the set-top device 110 , e.g. based at least on the user identifier.
- the set-top device 110 and/or the secondary device 112 may transmit a request for AV streams 232 , 242 A-C available at a selected marked point-in-time during the live event 130 ( 606 ).
- the set-top device 110 may present the user interface 300 to the user, and the user may select one of the representations 310 A-C of the marked points-in-time 210 A-C.
- the set-top device 110 and/or the secondary device 112 may receive and present representations 332 , 342 A-C of AV streams 232 , 242 A-C that were available at a selected marked point-in-time ( 608 ), e.g. via the user interface 300 .
- the set-top device and/or the secondary device 112 may transmit a request for a selected AV stream ( 610 ).
- the user may select one of the representations 332 , 342 A-C of the available AV streams 232 , 242 A-C via the user interface 300 .
- the set-top device 110 and/or the secondary device 112 receives and presents the selected AV stream beginning at the selected marked point-in-time ( 612 ).
- the request for the selected AV stream ( 610 ) may be transmitted via one of the devices, such as the secondary device 112 , and the selected AV stream may be received and presented by the other device, such as the set-top device 110 .
- the user may view the marked points-in-time for an event, and select one of the AV streams available for a selected marked point-in-time via the secondary device 112 and the user may view the selected AV stream on the output device 114 , via the set-top device 110 .
- FIG. 7 conceptually illustrates an electronic system 700 with which one or more implementations of the subject technology may be implemented.
- the electronic system 700 can be a gateway device, a set-top device, a desktop computer, a laptop computer, a tablet computer, a server, a switch, a router, a base station, a receiver, a phone, a personal digital assistant (PDA), or generally any electronic device that transmits signals over a network.
- PDA personal digital assistant
- Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media.
- the electronic system 700 includes a bus 708 , one or more processor(s) 712 , a system memory 704 or buffer, a read-only memory (ROM) 710 , a permanent storage device 702 , an input device interface 714 , an output device interface 706 , and one or more network interface(s) 716 , or subsets and variations thereof.
- a bus 708 one or more processor(s) 712 , a system memory 704 or buffer, a read-only memory (ROM) 710 , a permanent storage device 702 , an input device interface 714 , an output device interface 706 , and one or more network interface(s) 716 , or subsets and variations thereof.
- the bus 708 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 700 .
- the bus 708 communicatively connects the one or more processor(s) 712 with the ROM 710 , the system memory 704 , and the permanent storage device 702 . From these various memory units, the one or more processor(s) 712 retrieve instructions to execute and data to process in order to execute the processes of the subject disclosure.
- the one or more processor(s) 712 can be a single processor or a multi-core processor in different implementations.
- the ROM 710 stores static data and instructions that are needed by the one or more processor(s) 712 and other modules of the electronic system 700 .
- the permanent storage device 702 may be a read-and-write memory device.
- the permanent storage device 702 may be a non-volatile memory unit that stores instructions and data even when the electronic system 700 is off.
- a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as the permanent storage device 702 .
- a removable storage device such as a floppy disk, flash drive, and its corresponding disk drive
- the system memory 704 may be a read-and-write memory device. However, unlike the permanent storage device 702 , the system memory 704 may be a volatile read-and-write memory, such as random access memory.
- the system memory 704 may store any of the instructions and data that one or more processor(s) 712 may need at runtime.
- the processes of the subject disclosure are stored in the system memory 704 , the permanent storage device 702 , and/or the ROM 710 . From these various memory units, the one or more processor(s) 712 retrieve instructions to execute and data to process in order to execute the processes of one or more implementations.
- the bus 708 also connects to the input and output device interfaces 714 and 706 .
- the input device interface 714 enables a user to communicate information and select commands to the electronic system 700 .
- Input devices that may be used with the input device interface 714 may include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”).
- the output device interface 706 may enable, for example, the display of images generated by electronic system 700 .
- Output devices that may be used with the output device interface 706 may include, for example, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.
- printers and display devices such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.
- One or more implementations may include devices that function as both input and output devices, such as a touchscreen.
- feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- bus 708 also couples electronic system 700 to one or more networks (not shown) through one or more network interface(s) 716 .
- One or more network interface(s) may include an Ethernet interface, a WiFi interface, a multimedia over coax alliance (MoCA) interface, a reduced gigabit media independent interface (RGMII), or generally any interface for connecting to a network.
- electronic system 700 can be a part of one or more networks of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system 700 can be used in conjunction with the subject disclosure.
- Implementations within the scope of the present disclosure can be partially or entirely realized using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions.
- the tangible computer-readable storage medium also can be non-transitory in nature.
- the computer-readable storage medium can be any storage medium that can be read, written, or otherwise accessed by a general purpose or special purpose computing device, including any processing electronics and/or processing circuitry capable of executing instructions.
- the computer-readable medium can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM.
- the computer-readable medium also can include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.
- the computer-readable storage medium can include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions.
- the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer-readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.
- Instructions can be directly executable or can be used to develop executable instructions.
- instructions can be realized as executable or non-executable machine code or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code.
- instructions also can be realized as or can include data.
- Computer-executable instructions also can be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those of skill in the art, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without varying the underlying logic, function, processing, and output.
- ASICs application specific integrated circuits
- FPGAs field programmable gate arrays
- integrated circuits execute instructions that are stored on the circuit itself.
- any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
- base station As used in this specification and any claims of this application, the terms “base station”, “receiver”, “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people.
- display or “displaying” means displaying on an electronic device.
- the phrase “at least one of” preceding a series of items, with the term “and” or “or” to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item).
- the phrase “at least one of” does not require selection of at least one of each item listed; rather, the phrase allows a meaning that includes at least one of any one of the items, and/or at least one of any combination of the items, and/or at least one of each of the items.
- phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and/or at least one of each of A, B, and C.
- a processor configured to monitor and control an operation or a component may also mean the processor being programmed to monitor and control the operation or the processor being operable to monitor and control the operation.
- a processor configured to execute code can be construed as a processor programmed to execute code or operable to execute code.
- a phrase such as “an aspect” does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology.
- a disclosure relating to an aspect may apply to all configurations, or one or more configurations.
- An aspect may provide one or more examples of the disclosure.
- a phrase such as an “aspect” may refer to one or more aspects and vice versa.
- a phrase such as an “embodiment” does not imply that such embodiment is essential to the subject technology or that such embodiment applies to all configurations of the subject technology.
- a disclosure relating to an embodiment may apply to all embodiments, or one or more embodiments.
- An embodiment may provide one or more examples of the disclosure.
- a phrase such an “embodiment” may refer to one or more embodiments and vice versa.
- a phrase such as a “configuration” does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology.
- a disclosure relating to a configuration may apply to all configurations, or one or more configurations.
- a configuration may provide one or more examples of the disclosure.
- a phrase such as a “configuration” may refer to one or more configurations and vice versa.
Abstract
Description
Claims (22)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/148,625 US9609373B2 (en) | 2013-10-25 | 2014-01-06 | Presentation timeline synchronization across audio-video (AV) streams |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361895970P | 2013-10-25 | 2013-10-25 | |
US14/148,625 US9609373B2 (en) | 2013-10-25 | 2014-01-06 | Presentation timeline synchronization across audio-video (AV) streams |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150121436A1 US20150121436A1 (en) | 2015-04-30 |
US9609373B2 true US9609373B2 (en) | 2017-03-28 |
Family
ID=52997028
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/148,625 Active 2034-08-27 US9609373B2 (en) | 2013-10-25 | 2014-01-06 | Presentation timeline synchronization across audio-video (AV) streams |
Country Status (1)
Country | Link |
---|---|
US (1) | US9609373B2 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160277465A1 (en) * | 2014-06-13 | 2016-09-22 | Tencent Technology (Shenzhen) Company Limited | Method and system for client-server real-time interaction based on streaming media |
US10199069B1 (en) * | 2012-01-30 | 2019-02-05 | Google Llc | Aggregation on related media content |
US10616665B1 (en) * | 2014-04-23 | 2020-04-07 | Daniel Stieglitz | Automated video logging methods and systems |
US10769446B2 (en) | 2014-02-28 | 2020-09-08 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations |
US10862944B1 (en) | 2017-06-23 | 2020-12-08 | Amazon Technologies, Inc. | Real-time video streaming with latency control |
US10997425B2 (en) | 2014-02-28 | 2021-05-04 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11113887B2 (en) * | 2018-01-08 | 2021-09-07 | Verizon Patent And Licensing Inc | Generating three-dimensional content from two-dimensional images |
US11113535B2 (en) | 2019-11-08 | 2021-09-07 | Second Spectrum, Inc. | Determining tactical relevance and similarity of video sequences |
US11120271B2 (en) | 2014-02-28 | 2021-09-14 | Second Spectrum, Inc. | Data processing systems and methods for enhanced augmentation of interactive video content |
US11380101B2 (en) | 2014-02-28 | 2022-07-05 | Second Spectrum, Inc. | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US11861906B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Data processing systems and methods for enhanced augmentation of interactive video content |
Families Citing this family (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9686581B2 (en) * | 2013-11-07 | 2017-06-20 | Cisco Technology, Inc. | Second-screen TV bridge |
US10754511B2 (en) | 2013-11-20 | 2020-08-25 | Google Llc | Multi-view audio and video interactive playback |
US11275949B2 (en) | 2014-02-28 | 2022-03-15 | Second Spectrum, Inc. | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition |
US10832057B2 (en) | 2014-02-28 | 2020-11-10 | Second Spectrum, Inc. | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition |
CA2961970A1 (en) * | 2014-09-23 | 2016-03-31 | Surgical Safety Technologies Inc. | Operating room black-box device, system, method and computer readable medium |
CN107615395B (en) * | 2015-03-26 | 2021-02-05 | 外科安全技术公司 | Operating room black box apparatus, system, method and computer readable medium for event and error prediction |
US20160316261A1 (en) * | 2015-04-23 | 2016-10-27 | Sorenson Media, Inc. | Automatic content recognition fingerprint sequence matching |
KR102309811B1 (en) * | 2015-08-28 | 2021-10-07 | 한화테크윈 주식회사 | Apparatus and method for providing time information |
US10771423B2 (en) * | 2015-11-24 | 2020-09-08 | Facebook, Inc. | Systems and methods to control event based information |
US10021438B2 (en) | 2015-12-09 | 2018-07-10 | Comcast Cable Communications, Llc | Synchronizing playback of segmented video content across multiple video playback devices |
US10372520B2 (en) | 2016-11-22 | 2019-08-06 | Cisco Technology, Inc. | Graphical user interface for visualizing a plurality of issues with an infrastructure |
US10739943B2 (en) | 2016-12-13 | 2020-08-11 | Cisco Technology, Inc. | Ordered list user interface |
EP3352077A1 (en) * | 2017-01-24 | 2018-07-25 | Da Sheng Inc. | Method for synchronously taking audio and video in order to proceed one-to-multi multimedia stream |
WO2018137040A1 (en) * | 2017-01-26 | 2018-08-02 | D-Box Technologies Inc. | Capturing and synchronizing motion with recorded audio/video |
CN108260016B (en) * | 2018-03-13 | 2020-07-28 | 北京小米移动软件有限公司 | Live broadcast processing method, device, equipment, system and storage medium |
US10862867B2 (en) | 2018-04-01 | 2020-12-08 | Cisco Technology, Inc. | Intelligent graphical user interface |
JP7365212B2 (en) * | 2019-12-03 | 2023-10-19 | 株式会社ソニー・インタラクティブエンタテインメント | Video playback device, video playback system, and video playback method |
WO2022005618A1 (en) * | 2020-06-29 | 2022-01-06 | Gideon Eden | Processing video and audio streaming data |
Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5729471A (en) * | 1995-03-31 | 1998-03-17 | The Regents Of The University Of California | Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene |
US6144375A (en) * | 1998-08-14 | 2000-11-07 | Praja Inc. | Multi-perspective viewer for content-based interactivity |
US20030093790A1 (en) * | 2000-03-28 | 2003-05-15 | Logan James D. | Audio and video program recording, editing and playback systems using metadata |
US20050005308A1 (en) * | 2002-01-29 | 2005-01-06 | Gotuit Video, Inc. | Methods and apparatus for recording and replaying sports broadcasts |
US6920181B1 (en) * | 2000-09-19 | 2005-07-19 | Todd Porter | Method for synchronizing audio and video streams |
US20080216141A1 (en) * | 2007-02-07 | 2008-09-04 | The Directv Group, Inc. | On demand rf video feed for portable video monitor |
US20080320545A1 (en) * | 2007-06-22 | 2008-12-25 | Schwartz Richard T | System and method for providing audio-visual programming with alternative content |
US20090164904A1 (en) * | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | Blog-Based Video Summarization |
US20100251295A1 (en) * | 2009-03-31 | 2010-09-30 | At&T Intellectual Property I, L.P. | System and Method to Create a Media Content Summary Based on Viewer Annotations |
US20110075990A1 (en) * | 2009-09-25 | 2011-03-31 | Mark Kenneth Eyer | Video Bookmarking |
US20110126252A1 (en) * | 2009-11-20 | 2011-05-26 | At&T Intellectual Property I, L.P. | Method and apparatus for presenting media programs |
WO2013013176A1 (en) * | 2011-07-21 | 2013-01-24 | Sirius Xm Radio Inc. | Method and apparatus for delivery of programs and metadata to provide user alerts to tune to corresponding program channels before high interest events occur during playback of programs |
US20130279740A1 (en) * | 2012-04-18 | 2013-10-24 | Dolby Laboratories Licensing Corporation | Identifying Multimedia Objects Based on Multimedia Fingerprint |
-
2014
- 2014-01-06 US US14/148,625 patent/US9609373B2/en active Active
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5729471A (en) * | 1995-03-31 | 1998-03-17 | The Regents Of The University Of California | Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene |
US6144375A (en) * | 1998-08-14 | 2000-11-07 | Praja Inc. | Multi-perspective viewer for content-based interactivity |
US20030093790A1 (en) * | 2000-03-28 | 2003-05-15 | Logan James D. | Audio and video program recording, editing and playback systems using metadata |
US6920181B1 (en) * | 2000-09-19 | 2005-07-19 | Todd Porter | Method for synchronizing audio and video streams |
US20050005308A1 (en) * | 2002-01-29 | 2005-01-06 | Gotuit Video, Inc. | Methods and apparatus for recording and replaying sports broadcasts |
US20080216141A1 (en) * | 2007-02-07 | 2008-09-04 | The Directv Group, Inc. | On demand rf video feed for portable video monitor |
US20080320545A1 (en) * | 2007-06-22 | 2008-12-25 | Schwartz Richard T | System and method for providing audio-visual programming with alternative content |
US20090164904A1 (en) * | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | Blog-Based Video Summarization |
US20100251295A1 (en) * | 2009-03-31 | 2010-09-30 | At&T Intellectual Property I, L.P. | System and Method to Create a Media Content Summary Based on Viewer Annotations |
US20110075990A1 (en) * | 2009-09-25 | 2011-03-31 | Mark Kenneth Eyer | Video Bookmarking |
US20110126252A1 (en) * | 2009-11-20 | 2011-05-26 | At&T Intellectual Property I, L.P. | Method and apparatus for presenting media programs |
WO2013013176A1 (en) * | 2011-07-21 | 2013-01-24 | Sirius Xm Radio Inc. | Method and apparatus for delivery of programs and metadata to provide user alerts to tune to corresponding program channels before high interest events occur during playback of programs |
US20140157307A1 (en) * | 2011-07-21 | 2014-06-05 | Stuart Anderson Cox | Method and apparatus for delivery of programs and metadata to provide user alerts to tune to corresponding program channels before high interest events occur during playback of programs |
US20130279740A1 (en) * | 2012-04-18 | 2013-10-24 | Dolby Laboratories Licensing Corporation | Identifying Multimedia Objects Based on Multimedia Fingerprint |
Non-Patent Citations (18)
Title |
---|
"B@TV-Multiple Camera Event Streaming - Paperheads", retrieved from <http://www.paperheads.co.uk/projects/b-tv-multiple-camera-event-streaming.go> on Dec. 3, 2013, 2 pages. |
"Live Streaming | Streaming Tank | Live Video Streaming", retrieved from <http://www.streamingtank.com/services/live-streaming/> on Dec. 3, 2013, 2 pages. |
"Livestream Store-Livestream Studio-Live Production Switchers", retrieved from <http://store.livestream.com/pages/livestream-studio> on Dec. 3, 2013, 8 pages. |
"Livestream-Live Blogging Tools, Tell the story of your event", retrieved from <http://new.livestream.com/live-blogging-tools> on Dec. 3, 2013, 11 pages. |
"Livestream-Live Video Tools, Everything you need to live stream video", retrieved from <http://new.livestream.com//live-video-tools> on Dec. 3, 2013, 10 pages. |
"Livestream-Watch Thousands of Live Events & Live Stream Your Events", retrieved from <http://new.livestream.com/> on Dec. 3, 2013, 12 pages. |
"NeuLion-Internet Television Technology Innovators (IPTV)-What We Do", retrieved from <http://www.neulion.com/> on Dec. 3, 2013, 2 pages. |
"Rihanna Live Concert Streaming Tank | Live Video Streaming", retrieved from <http://www.streamingtank.com/client/case-studies/rihanna-live-concert/> on Dec. 3, 2013, 2 pages. |
"What We Do-NeuLion Sports-NeuLion-Internet Television Technology Innovators (IPTV)-Features", retrieved from <http://www.neulion.com/SportSelect.dbml?DB-OEM-ID=30000&SPID=122708&SPSID=731710> on Dec. 3, 2013, 2 pages. |
"What We Do-NeuLion Sports-NeuLion-Internet Television Technology Innovators (IPTV)-Interaction", retrieved from <http://www.neulion.com/Links.dbml?DB-OEM-ID=30000&SPID=122708&SPSID=725510&CONTENT-MODULE-CONTENT-ID=420912&content-link-name=Interaction> on Dec. 3, 2013, 2 pages. |
"B@TV—Multiple Camera Event Streaming - Paperheads", retrieved from <http://www.paperheads.co.uk/projects/b-tv-multiple-camera-event-streaming.go> on Dec. 3, 2013, 2 pages. |
"Livestream Store—Livestream Studio—Live Production Switchers", retrieved from <http://store.livestream.com/pages/livestream-studio> on Dec. 3, 2013, 8 pages. |
"Livestream—Live Blogging Tools, Tell the story of your event", retrieved from <http://new.livestream.com/live-blogging-tools> on Dec. 3, 2013, 11 pages. |
"Livestream—Live Video Tools, Everything you need to live stream video", retrieved from <http://new.livestream.com//live-video-tools> on Dec. 3, 2013, 10 pages. |
"Livestream—Watch Thousands of Live Events & Live Stream Your Events", retrieved from <http://new.livestream.com/> on Dec. 3, 2013, 12 pages. |
"NeuLion—Internet Television Technology Innovators (IPTV)—What We Do", retrieved from <http://www.neulion.com/> on Dec. 3, 2013, 2 pages. |
"What We Do—NeuLion Sports—NeuLion—Internet Television Technology Innovators (IPTV)—Features", retrieved from <http://www.neulion.com/SportSelect.dbml?DB—OEM—ID=30000&SPID=122708&SPSID=731710> on Dec. 3, 2013, 2 pages. |
"What We Do—NeuLion Sports—NeuLion—Internet Television Technology Innovators (IPTV)—Interaction", retrieved from <http://www.neulion.com/Links.dbml?DB—OEM—ID=30000&SPID=122708&SPSID=725510&CONTENT—MODULE—CONTENT—ID=420912&content—link—name=Interaction> on Dec. 3, 2013, 2 pages. |
Cited By (18)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10770112B2 (en) | 2012-01-30 | 2020-09-08 | Google Llc | Aggregation of related media content |
US10199069B1 (en) * | 2012-01-30 | 2019-02-05 | Google Llc | Aggregation on related media content |
US20190172495A1 (en) * | 2012-01-30 | 2019-06-06 | Google Llc | Aggregation of related media content |
US11335380B2 (en) | 2012-01-30 | 2022-05-17 | Google Llc | Aggregation of related media content |
US10769446B2 (en) | 2014-02-28 | 2020-09-08 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations |
US11120271B2 (en) | 2014-02-28 | 2021-09-14 | Second Spectrum, Inc. | Data processing systems and methods for enhanced augmentation of interactive video content |
US11861906B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Data processing systems and methods for enhanced augmentation of interactive video content |
US10997425B2 (en) | 2014-02-28 | 2021-05-04 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11023736B2 (en) | 2014-02-28 | 2021-06-01 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11861905B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Methods and systems of spatiotemporal pattern recognition for video content development |
US11380101B2 (en) | 2014-02-28 | 2022-07-05 | Second Spectrum, Inc. | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US11373405B2 (en) | 2014-02-28 | 2022-06-28 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations to produce augmented video |
US10616665B1 (en) * | 2014-04-23 | 2020-04-07 | Daniel Stieglitz | Automated video logging methods and systems |
US20160277465A1 (en) * | 2014-06-13 | 2016-09-22 | Tencent Technology (Shenzhen) Company Limited | Method and system for client-server real-time interaction based on streaming media |
US10862944B1 (en) | 2017-06-23 | 2020-12-08 | Amazon Technologies, Inc. | Real-time video streaming with latency control |
US11113887B2 (en) * | 2018-01-08 | 2021-09-07 | Verizon Patent And Licensing Inc | Generating three-dimensional content from two-dimensional images |
US11113535B2 (en) | 2019-11-08 | 2021-09-07 | Second Spectrum, Inc. | Determining tactical relevance and similarity of video sequences |
US11778244B2 (en) | 2019-11-08 | 2023-10-03 | Genius Sports Ss, Llc | Determining tactical relevance and similarity of video sequences |
Also Published As
Publication number | Publication date |
---|---|
US20150121436A1 (en) | 2015-04-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9609373B2 (en) | Presentation timeline synchronization across audio-video (AV) streams | |
US20150128174A1 (en) | Selecting audio-video (av) streams associated with an event | |
US11805291B2 (en) | Synchronizing media content tag data | |
US11792444B2 (en) | Dynamic viewpoints of live event | |
CN108713321B (en) | It is identified using more matching detections with the media channel of the disambiguation based on broadcast time | |
US11910066B2 (en) | Providing interactive advertisements | |
US9875318B2 (en) | Concepts for providing an enhanced media presentation | |
US11275804B2 (en) | Systems and methods for providing timely and relevant social media updates for a person of interest in a media asset who is unknown simultaneously with the media asset | |
US10622020B2 (en) | Point of view video processing and curation platform | |
US9215080B2 (en) | Adaptive bit rate distribution of multicast streams | |
US20110289532A1 (en) | System and method for interactive second screen | |
US9036092B2 (en) | Video channel change system | |
US9668023B1 (en) | Systems and methods for providing real-time presentation of timely social chatter of a person of interest depicted in media simultaneous with presentation of the media itself | |
CN111343476A (en) | Video sharing method and device, electronic equipment and storage medium | |
US20200177971A1 (en) | Systems and methods for summarizing missed portions of storylines | |
US10019623B2 (en) | Systems and methods for providing timely and relevant social media updates from persons related to a person of interest in a video simultaneously with the video | |
US20220394328A1 (en) | Consolidated Watch Parties | |
US20230254543A1 (en) | Coordinated primary media stream with aggregate supplemental media stream | |
CN111246269B (en) | Display, playing content monitoring method and playing content monitoring system | |
US20120185890A1 (en) | Synchronized video presentation | |
US9420338B1 (en) | Apparatus and method for providing power on and power off for a display device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BROADCOM CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RANGO, ROBERT AMERICO;SOTO, WALTER GLENN;SIGNING DATES FROM 20131219 TO 20131220;REEL/FRAME:031967/0954 |
|
AS | Assignment |
Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH CAROLINA Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:BROADCOM CORPORATION;REEL/FRAME:037806/0001 Effective date: 20160201 Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:BROADCOM CORPORATION;REEL/FRAME:037806/0001 Effective date: 20160201 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD., SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BROADCOM CORPORATION;REEL/FRAME:041706/0001 Effective date: 20170120 Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BROADCOM CORPORATION;REEL/FRAME:041706/0001 Effective date: 20170120 |
|
AS | Assignment |
Owner name: BROADCOM CORPORATION, CALIFORNIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS COLLATERAL AGENT;REEL/FRAME:041712/0001 Effective date: 20170119 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITE Free format text: MERGER;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:047422/0464 Effective date: 20180509 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITE Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE EXECUTION DATE PREVIOUSLY RECORDED AT REEL: 047422 FRAME: 0464. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:048883/0702 Effective date: 20180905 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |