METHOD AND DEVICE FOR PROCESSING MULTIMEDIA DATA
The invention relates to a method and a device for processing multimedia data.
Devices for processing multimedia data streams are required to process streams of differing nature demanding short processing times as well as noteworthy performance.
The expression "multimedia data processing" should be understood to mean, non-exhaustively, coding, decoding in different formats according to the types of application for which the data streams are intended as well as the manipulation of multimedia data consisting in customizing the data according to a user's desires.
The expression "multimedia data" is understood, non-exhaustively, to mean data of audio/video type, stationary pictures, sound, graphical data.
Moreover, the destination devices for these streams are often devices having characteristics and requirements that are often incompatible, these devices possibly being, non-exhaustively, display devices, storage devices or data networks.
The processing to be performed on the incoming streams is therefore often different, for example as regards coding or decoding, and sometimes even incompatible when a stream is intended both for a storage means and for a display means for example. In this case, the data streams must very often be manipulated several times, this requiring considerable resources in the known devices.
Specifically, the known devices for processing multimedia data implement bus-based architectures interconnecting the various components for carrying out the various processings to be performed on the video. If one and the same stream is intended for various destinations of differing nature, these streams are often conveyed on the bus so as to undergo the various processings. The data therefore pass over the bus several times so as to undergo the various processings performed by the various modules and when the data are intended for several modules, they
very often monopolize the bus several times for the same type of processing, thereby giving rise to considerable processing times and requiring numerous resources.
The invention therefore proposes a device whose architecture is adapted to the processing of multimedia data streams and makes it possible to improve the performance of current devices.
For this purpose, the invention proposes a device for processing multimedia data comprising:-
- means for receiving incoming streams of multimedia data of differing nature,
- stream management means,
- means for routing the incoming streams as a function of their nature to the stream management means,
- a plurality of means for processing streams as a function of the origin or of the destination of the streams, connected to the stream management means,
The management means (4), the reception means (1 , 2, 10), the means (3) for routing the incoming streams and the stream processing means (5, 6, 7) cooperating so as to ensure at least one path for each incoming stream of differing nature so as to allow parallel processing of the streams of differing natures and their forwarding to at least one stream destination means (8, 9), characterized in that the incoming streams may be streams originating from the plurality of stream processing means.
Thus, the various streams processed are not in competition and may be processed independently of one another in such a way as not to slow down the processing time and avoid conflicts which may occur in known architectures of the bus type where a single bus links the various components involved in the processing of the streams.
The architecture proposed may for example be regarded as an architecture of "pipeline" type as opposed to an architecture of single-bus type, commonly employed for this kind of processing.
Each incoming stream entering the processing device is forwarded directly from the input to the output, passing through the various modules but not making several return trips between the various modules.
The streams may be re-edited or reformatted in such a way as to store them or transmit them to other nearby or distant devices. Thus, it is possible to code the data streams into a format which would be more suitable for storage, using for example higher compression rates or to transmit them to other networks or devices capable of recognizing a different data format.
According to a preferred embodiment, the stream destination means may be chosen from - a permanent storage means,
- a removable storage means,
- a data network,
- a display means, and any combination of these entities.
The invention is particularly advantageous, but not limited to this embodiment, when the various streams are intended for storage means, networks or display devices whose requirements differ in terms of stream format, bit rate and data processing.
According to a preferred embodiment, when the destination module is a display means, the processing means comprise
- stream decoding means,
- means for presenting the streams of audio/video type, - means for coding a video stream.
According to this embodiment, the data streams may undergo processing making it possible to carry out functions of stream decoding, audio/video conversion, video coding according to various standards and thus
to accommodate the various destination devices while not monopolizing the device for this particular processing. All of this processing is performed using different paths allowing it to be performed concurrently.
According to a preferred embodiment, the means for decoding the stream comprise at least two decoding means adapted for decoding at least two streams simultaneously.
This may possibly allow video inlaying to be carried out. According to a preferred embodiment, the means for presenting the streams of audio/video type are intended for manipulating the data streams by adding thereto means chosen from
- text inlaying
- graphical synthesis,
- picture in picture insertion, - video effects,
- real-time editing on the video stream, and any combination of these entities.
In accordance with this latter embodiment, the device is particularly beneficial for manipulating various types of stream by supplementing same with possibilities of graphical manipulation and various special effects making it possible to customize the programmes received.
According to a preferred embodiment, the incoming streams of multimedia data are chosen from
- streams of digital nature,
- streams of analogue nature,
- streams coming from the means for presenting the streams of audio/video type - streams coming from the stream decoding means, and any combination of these entities.
According to a preferred embodiment, the device comprises
- means for converting the streams of analogue nature into streams of digital nature,
- means for coding the digitized analogue streams, the streams coming from the stream decoding means and the outgoing streams from the means for presenting the streams of audio/video type.
According to a preferred embodiment, the device comprises content protection means making it possible to protect the content of the outgoing streams from the means of presenting the streams of audio-video type and the streams coming from the stream decoding means and intended to be stored or broadcast to an external network.
Content protection is an important plus for content creators and in particular when the audiovideo data processing devices are inserted into communication networks since it is then difficult to oversee the use of the content by the various peripherals connected to the network. It may therefore be particularly advantageous, in certain embodiments, to provide content protection in such a way as to limit the use of the data and the rights of certain users.
The invention also relates to a method of processing multimedia data comprising the steps of:
- receiving the incoming streams of multimedia data of differing nature, - routing the streams as a function of their nature to stream management means,
- stream processing as a function of the origin or of the destination of the stream, connected to the stream management means, characterized in that it comprises a step allowing the management means, the stream routing means and the processing means to cooperate so as to ensure at least one direct and distinct path for each incoming stream of differing nature so as to ensure the
simultaneous processing of the streams of differing nature and their forwarding to a content destination module.
The invention will be better understood and illustrated by means of wholly nonlimiting, advantageous exemplary embodiments and modes of implementation, with reference to the appended figures in which: s Figure 1 shows a global representation of an embodiment of the invention,
Figure 2 shows a detailed representation of an embodiment of the invention.
In Figure 1 , the processing device 100 comprises inputs of digital nature D1 , D2,...,Dn and inputs of analogue nature Ax conveying multimedia or analogue data.
The analogue streams are converted into digital streams by an analogue/digital converter 10 and sent to a multiplexer 1 also multiplexing the inputs fed back from the device 100. The output of the coder 2 is connected to the device 100 and more particularly to the module 3 for routing the streams.
The module 3 receives as input all the incoming streams mentioned and routes the streams as a function of their nature and of their destination. The inputs of the module 3 are dedicated to the incoming streams, the outputs are dedicated to the inputs of a stream manager 4. The module 3 is an adaptation layer which makes it possible to route an incoming signal to a process (which has a dedicated input) of the stream manager 4. The routing is dependent on the application. For example, the direct recording of a stream on a storage means of hard disk type, not represented, connected to an input/output 8 of the device 100, is carried out by setting up a path between an input D1 , D2,...,Dn or an output of the coder 2 and the input of the process which manages the audio/video recording of the hard disk.
The routing module 3 is connected to the data stream manager 4. The module 4 performs, among other things, operations of demultiplexing, of access control with regard to the audio/video stream, of descrambling, of time stamping, of storage, of reading, of unbundling of two or more incoming streams. It also performs readouts of data stored in storage means connected to the input/output 8. This module is capable of processing at least two streams simultaneously. All the operations are performed in real time by fast processors.
At output the stream manager manages streams of differing nature depending on the destination of the streams.
The device 100 comprises processing means consisting of decoders 5, of module for presenting the video 6 and of video encoder 7.
The streams may be intended for storage devices, for display devices, or for other terminal across the networks.
The streams intended for display devices are transmitted to decoders 5 operating according to well known standards depending on the type of destination device. The streams input to the decoder are in accordance with the following standards: MPEG-2 (standing for "Motion Picture Expert Group"), MPEG-4, DV (standing for "Digital video") or JPEG (standing for "Joint Picture Expert Group"), TIFF (standing for "Tagged Image File Format") for the coding of stationary pictures, PNG (standing for "Portable Network Graphics") or MP3 (standing for "MPEG audio layer"), MP3Pro, WMA (standing for "Windows Media Audio") in respect of sound. The decoders 5 are for example decoders in accordance with the CCIR 656 standard, also known by the name 4:2:2.
The decoders are, for example, coders based on the discrete cosine transform (DCT). The decoders 5 comprise fast processors driven by real-time software which allows the real-time decoding of the video streams without the assistance of outside processors.
The audio streams of MPEG-2, MP3, WMA type are processed by audio decoders which are not represented in this figure.
The output of the video decoder is connected to a video presentation module 6.
This module comprises means allowing it to perform special effects modes, text inlaying ("OSD"), picture-in-picture ("PIP"), graphical synthesis, real-time editing on the video streams.
This module also makes it possible to perform advanced presentations by adding special modes such as pictures in two or three dimensions. In such modules such as the module 6, the video is regarded as a texture and is overlaid onto two-dimensional shapes such as rectangles or any other shapes. When three-dimensional effects are produced, three-dimensional shapes such as cubes for example are used to overlay the video.
Calculations are performed at the pixel level in order to apply illumination, shadows and other effects.
The video presentation module 6 gives the means of local production of moving pictures, completely synthesized or created from original pictures or from scenes. Such moving pictures may for example be EPGs (standing for "electronic programme guide") or games. The presentation module makes it possible to present pictures either by object-based scene composition, or by synthesis, or both. The video presentation module also allows a user to carry out real-time video editing and therefore to produce his personal films or his picture libraries.
The video presentation module 6 is connected at output to a video encoding module 7 which produces a signal in accordance with the format required by the display device. This output signal may be a known signal of digital or analogue type, for example of SDTV type (standing for "Standard Definition Television") or of HDTV type (standing for "High Definition Television").
In a first embodiment, when the output format is in accordance with the SDTV standard, the analogue encoder is embodied as an encoder of digital type to which a digital/analogue converter is added. The digital encoder converts a 4:2:2 signal directly into an NTSC signal (standing for "National Television System Committee"), in accordance with the CCIR601 standard, or a PAL (standing for "Phase Alternating Line"), SECAM (standing for Systeme
Electronique Couleur Avec Memoire"), Y/C or RGB (standing for "Red, Green, Blue"), signal.
In a variant, when the output format is in accordance with the HDTV standard, the signals are formatted to the 4:2:2 signal and transmitted using for example the DVI standard (standing "Digital Visual Interface").
The output of the decoders 5 is also connected to the multiplexer 1.
This advantageously makes it possible to record audio/video data in any format whatsoever before or after transcoding. This advantageously makes it possible to record data coded initially in the DV format (at 30 Mbit/s) in the
MPEG-2 format with a rate of 4 Mbits/s, the loss of quality being very small.
The output of the video presentation module 6 is also connected to the multiplexer 1. This advantageously makes it possible to record video after it has been edited, modified, processed, etc. For example, this can allow a user receiving video from a digital camcorder by way of a network based on the IEEE-1394 standard to edit this video by adding special effects, comments to it and to record it subsequently on a storage means with a lower bit rate so as not to use too much memory space on a storage means.
Content protection mechanisms, not represented, are used in order to preserve the rights associated with the data received; content protection is an important objective in audio/video data processing and content creators do not want unlimited use of the data.
Figure 2 represents a more detailed view of a device 200 in accordance with an embodiment of the invention.
The device 200 in accordance with an embodiment according to the invention comprises two analogue inputs A1 , A2, two digital inputs D1 , D2, two inputs 35, 36 connected to an interface 34 in accordance with the IEEE- 1394 standard. The two analogue inputs are connected to a multiplexer 38, whose output is connected to an encoding module 11 making it possible to encode the analogue streams in the CCIR656 format.
The encoded stream in the CCIR656 format leaving the encoding module 11 is multiplexed by a multiplexer 1 with streams fed back from the device 200. At the output of the multiplexer 1 , the streams are transmitted to a coder 2 coding the data in the MPEG-2 format. A memory 37 of SDRAM type (standing for "Synchronous Dynamic Random Access Memory) is connected to the coder 2.
The digital inputs D1, D2, the outputs of the IEEE-1394 interface and the outputs of the coder 2 are all linked to the data routing module 3 that routes the streams as a function of their nature and of their destination, as described previously.
The module 3 is connected to the streams management module 4 described previously. This module 4 is connected to permanent 12 and removable 13 storage means. These storage means advantageously make it possible to record video data coded according to various formats. The stream manager 4 is also connected to a microprocessor 14 itself connected to an Ethernet link 15, to a chip card reader 16 and to an SDRAM type memory 17.
The microprocessor 14 houses the system, the applications and controls the various system-related peripherals, such as a PROM memory (standing for "Programmable Read Only Memory) 18, a real-time clock 19, a graphics generator 26, the slow peripherals such as the Ethernet interface 15, the chip card reader 16, the asynchronous mode of the IEEE-1394 interface 34, configures all the system modules.
The stream manager 4 is a collection of processors dedicated to the processing of the streams, that is controlled and monitored by the microprocessor but possesses the necessary resources for performing the tasks requested by the various applications.
The stream manager 4 is connected to video decoders 20 and 21 and to audio decoders 22 and 23. The video decoders 20 and 21 are connected respectively to memories 24 and 25 of SDRAM type.
The video decoders 20 and 21 are connected to the video presentation module 6.
The audio decoders 22 and 23 are connected to an audio module 29. This audio module 29 allows a direct connection to an audio playback device 32.
The video presentation module 6 is connected to a graphics accelerator 26 itself linked to a memory 27 of the SDRAM type.
The module 6 is also connected to a CVBS (standing for "Composite Video Baseband Signal") coding device 28 itself connected to a display device 31.
A connection bus B links the graphics accelerator 26, the microprocessor 14, the chip card reader 16 and the memory 18 of PROM type, the real-time clock 19 and the IEEE-1394 interface 34.