DE102004047068A1 - Apparatus and method for grouping temporal segments of a piece of music - Google Patents

Abstract

For grouping temporal segments of an audio piece, which is divided into different segment classes in main sections repeatedly occurring in the audio track, first a similarity representation is provided for the segments, the similarity representation for each segment having an associated plurality of similarity values, the similarity values indicating how similar the segment is to each other segment of the piece of music. Then, using the similarity values associated with the segment, a similarity threshold is calculated for a segment to then associate a segment with a segment class if the similarity value of the segment meets a predetermined relationship with respect to the similarity threshold. This achieves clustering that works efficiently and correctly even where there are segments with very different or nearly identical combined similarity values.

Description

The The present invention relates to audio segmentation and in particular to the analysis of pieces of music contained in the pieces of music individual main parts that may occur repeatedly in the piece of music.

music from the rock and pop area usually consists of more or less unique segments, such as intro, stanza, chorus, Bridge, Outro, etc. The beginning and To detect end times of such segments and the segments after their affiliation to group to the main classes (stanza and chorus), is the target of audio segmentation. A correct segmentation and Also marking the calculated segments can be different Areas are used meaningfully. For example, like this music be smartly "played" by online providers, such as Amazon, Musicline, etc.

The Most providers on the Internet are limited to their audio samples on a short excerpt from the offered pieces of music. In this Case would be it of course also useful, the prospective not only the first 30 seconds or any 30 seconds, but as representative as possible To offer an excerpt from the song. This could be z. B. be the refrain, or a summary of the song, consisting of segments, which belong to the different main classes (verse, chorus, ...).

One further application example for The technique of audio segmentation is to integrate the segmentation / grouping / marking algorithm into a music player. The information about segment beginnings and Enable segment ends Targeted navigation through a piece of music. By class membership the segments, ie whether a segment is a verse, a chorus, etc. is, z. B. also directly to the next chorus or the next stanza be jumped. Such an application is for large music markets of Interested in giving their customers the opportunity to complete Listen to albums. This saves the customer the annoying, searching forward to characteristic Places in the song that might motivate him to do so piece of music in the end actually to buy.

On In the field of audio segmentation there are different approaches. following is the approach of Jonathan Foote and Matthew Cooper exemplary shown. This method is described in FOOTE, J.T./Cooper, M.L .: Summarizing Popular Music via Structural Similarity Analysis. Proceedings of the IEEE Workshop on Signal Processing to Audio and Acoustics 2003. FOOTE, J.T./COOPER, M.L .: Media Segmentation using Self-Similar Decomposition. Proceedings of SPIE Storage and Retrieval for Multimedia Databases, Vol. 5021, pp. 167-75, January 2003.

The known method of Foote is based on the block diagram of 5 exemplified. First, a WAV file 500 provided. In a downstream extraction block 502 then a feature extraction takes place, as a feature the spectral coefficients per se or alternatively the mel frequency cepstral coefficients (MFCCs) are extracted. Prior to this extraction, a short-term Fourier transform (STFT) is performed with 0.05 second wide non-overlapping windows with the WAV file. The MFCC features are then extracted in the spectral range. It should be noted that the parameterization is not optimized for compression, transmission or reconstruction, but for audio analysis. The requirement is that similar audio pieces produce similar features.

The extracted features are then stored in memory 504 stored.

The feature extraction algorithm now has a segmentation algorithm that ends in a similarity matrix, as in a block 506 is shown. First, however, the feature matrix is read in ( 508 ), then to group feature vectors ( 510 ), then based on the grouped feature vectors to build a similarity matrix, which consists of a distance measurement between each of all features. In particular, all pairs of audio window pairs are compared using a quantitative similarity measure, distance.

The structure of the similarity matrix is in 8th shown. So is in 8th the piece of music as stream or stream 800 represented by audio samples. The audio piece is windowed as it has been executed, with a first window labeled i and a second window labeled j. Overall, the audio piece has z. B. K windows. This means that the similarity matrix has K rows and K columns. Then, for each window i and for each window j, a similarity measure to each other is calculated, and the calculated similarity measure or distance measure D (i, j) is input to the row or column designated by i and j in the similarity matrix. One column therefore shows the similarity of the window designated by j to all other audio windows in the piece of music. The similarity of the window j to the very first window of the piece of music would then be in column j and row 1. The similarity of the window j to the second window of the piece of music would then be in the column j, but now in the line 2. By contrast, the similarity of the second window to the first window would be in the second column of the matrix and in the first row of the matrix.

It It can be seen that the matrix is redundant in that it to the diagonal is symmetrical, and that on the diagonal the similarity of a window to itself, which is the trivial case of a 100% similarity represents.

An example of a similarity matrix of a piece is in 6 to see. Here again the completely symmetrical structure of the matrix with respect to the main diagonal is recognizable, whereby the main diagonal is visible as a light stripe. It should also be noted that due to the small window length compared to the relatively coarse time resolution in 6 the main diagonal is not to be seen as a lighter solid line, but off 6 only approximately recognizable.

This is done using the similarity matrix as described, for. In 6 is shown, a kernel correlation 512 with a kernel matrix 514 to obtain a measure of novelty, also known as the Novelty Score, which could be averaged and smoothed in 9 is shown. The smoothing of this Novelty Score is in 5 schematically by a block 516 shown.

This will be in a block 518 the segment boundaries are read out using the smoothed novelty value curve, for which purpose the local maxima in the smoothed novelty curve have to be determined and possibly even shifted by a constant number of samples caused by the smoothing in order to actually obtain the correct segment boundaries of the audio piece as absolute or relative time information.

This is done, as it already is in a block called clustering 5 can be seen, creates a so-called segment similarity representation or segment similarity matrix. An example of a segment similarity matrix is in 7 shown. The similarity matrix in 7 is basically similar to the feature similarity matrix of 6 , but now not more, as in 6 Features are used in windows, but features from a whole segment. The segment similarity matrix has a similar meaning to the feature similarity matrix, but with a much coarser resolution, which of course is desired when considering that window lengths are in the order of 0.05 seconds, while reasonably long segments are in the range of perhaps 10 seconds of a piece ,

This is then in a block 522 a clustering is carried out, ie a classification of the segments into segment classes (an arrangement of similar segments in the same segment class), and then in a block 524 To mark the found segment classes, which is also called "labeling", it is determined in the labeling which segment class contains segments which are stanzas, which are refrains, which are intros, outros, bridges, etc.

Finally, in one with 526 in 5 designated block creates a music grammar, the z. B. can be provided to a user, without redundancy of a piece only z. B. a verse, a chorus and the intro to hear.

following will be on the individual blocks yet detailed.

As already stated, the actual segmentation of the piece of music does not take place until the feature matrices have been generated and stored (Block 504 ).

ever according to which feature the piece of music examines its structure is to be read, the corresponding feature matrix is read out and loaded into a working memory for further processing. The feature matrix has the dimension number of analysis windows times number of feature coefficients.

The similarity matrix brings the feature course of a piece into a two-dimensional representation. For each pairwise combination of feature vectors, the distance measure is computed, which is recorded in the similarity matrix. There are various possibilities for calculating the distance measure between two vectors, namely, for example, the Euclidean distance measurement and the cosine distance measurement. A result D (i, j) between the two feature vectors is stored in the i, jth element of the window similarity matrix (block 506 ) saved. The main diagonal of the similarity matrix represents the course over the entire piece. Accordingly, the elements of the main diagonal result from the respective comparison of a window with itself and always have the value of the greatest similarity. For the cosine distance measurement this is the value 1, for the simple scalar difference and the Euclidean distance this value is 0.

To visualize a similarity matrix, as in 6 is shown, each element i, j is assigned a gray value. The gray values are graduated in proportion to the similarity values, so that the maximum similarity (the main diagonal) corresponds to the maximum similarity. Through this representation, one can already recognize the structure of a song visually on the basis of the matrix. Areas of similar feature expression correspond to quadrants of similar brightness along the main diagonal. Finding the boundaries between the areas is the task of the actual segmentation.

The structure of the similarity matrix is important for kernel correlation 512 calculated novelty measure. The novelty measure arises from the correlation of a particular kernel along the main diagonal of the similarity matrix. An exemplary kernel K is in 5 shown. If one correlates this kernel matrix along the main diagonal of the similarity matrix S, and thereby sums all the products of the superimposed matrix elements for each time point i of the piece, one obtains the measure of novelty, which in a smoothed form exemplifies 9 is shown. Preferably, the kernel K is not in 5 but an enlarged kernel, which is also superimposed with a Gaussian distribution, so that the edges of the matrix go to zero.

The Selection of the distinctive maxima in the novelty course is important for the segmentation. The Selection of all maxima of unsmoothed Novelty course would to a strong over-segmentation of the audio signal.

Therefore should the novelty measure be smoothed out, and with different filters, such as IIR filters or FIR filters.

are extracts the segment boundaries of a piece of music, so must similar now Segments are marked as such and grouped into classes.

Foote and Cooper describe the calculation of a segment-based similarity matrix using a Cullback-Leibler distance. For this purpose, individual segment feature matrices are extracted from the entire feature matrix on the basis of the segment boundaries obtained from the novelty process, ie each of these matrices is a submatrix of the entire feature matrix. The resulting segment similarity matrix 520 is now subjected to a singular value decomposition (SVD = SVD = Singular Value Decomposition).

hereupon receives singular values in descending order.

In the block 526 Then an automatic summary of a piece based on the segments and clusters of a piece of music is performed. For this purpose, first the two clusters with the largest singular values are selected. Then, the segment with the maximum value of the corresponding cluster indicator is added to this summary. This means that the summary includes a stanza and a chorus. Alternatively, all repeated segments can also be removed to ensure that all piece information is provided, but always exactly once.

Regarding further Techniques for segmentation / music analysis will be on CHU, s./LOGAN B .: Music Summary using Key Phrases. Technical Report, Cambridge Research Laboratory 2000, BARTSCH, M.A./WAKEFIELD, g. H .: To Catch a Chorus: Using Chroma-Based Representation for Audio Thumbnailing. Proceedings of the IEEE Workshop on Signal Processing to Audio and Acoustics 2001. http://musen.engin.umich.edu/papers/bartsch wakefield waspaa01 final.pdf, referenced

adversely in the known method is the fact that the singular value decomposition (SVD) for segment class formation, that is for assigning segments to clusters on the one hand is very compute-consuming and on the other problematic in the assessment of the results. So then, if the singular values are almost the same size, one maybe wrong decision made that the two similar Singular values are actually the same segment class and not two represent different segment classes.

Further It was found that the results obtained by the singular value decomposition then get more and more problematic when it differs strong similarity value gives, so if a piece very similar Contains shares, like stanza and chorus, but also relatively dissimilar parts, like intro, Outro or Bridge.

Further The problem with the known method is that always assumed is that the cluster among the two clusters with the highest singular values, which has the first segment in the song, the cluster is "stanza", and that the other cluster is the Cluster "refrain" is this procedure is based on the fact that in the known method it is assumed that a song always starts with a stanza. Experience has shown that significant labeling errors are obtained. This is insofar problematic, since labeling is effectively the "harvest" of the entire process, that is, what the user experiences directly. Were the previous steps still so precise and elaborate, so Everything is relativized, if at the end wrongly labeled, because then At the user's total loss of confidence in the whole concept could take.

At It should also be noted that, in particular, needs according to automatic music analysis methods, without that Result always checked and if necessary can be corrected. Instead, a procedure is only on Market usable if it automatically without human correction can expire.

The Object of the present invention is to provide an improved and at the same time efficient concept for grouping temporal Segments of a piece of music to accomplish.

These The object is achieved by a device for grouping according to claim 1, a method of grouping according to claim 16 or a Computer program according to claim 17 solved.

Of the The present invention is based on the finding that the assignment a segment to a segment class based on an adaptive similarity mean for a Segment to perform is such that by the similarity mean considered which overall similarity score has a segment throughout the piece. After for a segment such a mean of similarity has been calculated, for the calculation of which the number of segments and the similarity values the plurality of similarity values associated with the segment are needed, then becomes the actual Assignment of a segment to a segment class, ie to a cluster, on the basis of this similarity mean carried out. Is a similarity value a segment to the currently considered segment, for example above the similarity mean, thus, the segment is considered to be the segment class currently considered assigned. Is the similarity value a segment to the currently considered segment, however, below this similarity mean, so it will not be assigned to the segment class.

In in other words This means that the assignment is no longer dependent on the absolute size of the similarity values carried out but relative to the mean of similarity. This means that for a segment that has a relatively low similarity score, ie z. For example Segment having an intro or outro, the similarity mean lower will be as for a segment that is a stanza or chorus. With that the strong deviations of the similarities of Segments in pieces or the frequency the occurrence of certain segments in pieces, where z. Numeric Problems and thus also ambiguities and associated wrong assignments can be avoided.

The inventive concept is particularly suitable for Pieces of music, not just stanzas and choruses, so the segments that belong to segment class, the same similarity values have, but also for Pieces that besides stanza and chorus also have other parts, namely one introduction (Intro), an interlude (Bridge) or a finale (Outro).

at a preferred embodiment of present invention, the calculation of the adaptive similarity mean and assigning a segment iteratively, assigning segments at the next Iterationsdurchgang be ignored. This changes for the next iteration again the similarity absolute value, so the sum of the similarity values in a column of the similarity matrix, since already assigned segments have been set to 0.

at a preferred embodiment of present invention, a segmentation post-correction is performed, and Although to the effect that after the segmentation z. B. due to the Novelty value (the local maxima of novelty value) and after a subsequent Assignment to segment classes relatively short segments are examined to see if they are the predecessor segment or the successor segment can be assigned as segments below a minimum segment length highly likely to indicate over-segmentation.

at an alternative preferred embodiment of the present invention Invention will after the final segmentation and assignment in the segment classes are labeling using a special selection algorithm to ensure that the labeling is as correct as possible the segment classes as a stanza or chorus.

preferred embodiments The present invention will be described below with reference to FIG the attached drawing explained in detail. Show it:

1 a block diagram of the device according to the invention for grouping according to a preferred embodiment of the present invention;

2 a flow chart for illustrating a preferred embodiment of the invention for iterative assignment;

3 a block diagram of Funktionswei se the segmentation correction device;

4a and 4b a preferred embodiment of the segment class designator;

5 an overall block diagram of an audio analysis tool;

6 a representation of an exemplary feature similarity matrix;

7 an exemplary representation of a segment similarity matrix;

8th a schematic representation for illustrating the elements in a similarity matrix S; and

9 a schematic representation of a smoothed novelty value.

1 shows a device for grouping temporal segments of a piece of music, which is divided into main parts repeatedly occurring in the piece of music, into different segment classes, one segment class being assigned to a main part. The present invention thus relates particularly to pieces of music which are subject to a certain structure in which similar sections appear several times and alternate with other sections. Most rock and pop songs have a clear structure in terms of their main parts.

The Literature deals mainly with the topic of music analysis classical music, but there is also a lot for rock and pop music. The main parts of a piece of music are also called "large moldings." Under one Large molding of a piece is understood a section which, in terms of various characteristics, z. As melody, rhythm, texture, etc., a relatively uniform Texture has. This definition applies generally in music theory.

Large moldings in rock and pop music are z. Eg verse, chorus, bridge and solo. In classical music, an interplay of chorus and other parts (couplets) of a composition is also called rondo. In general, the couplets contrast with the chorus, for example in terms of melody, rhythm, harmony, key or instrumentation. This can also be transferred to modern light music. Just as there are different forms in the rondo (chain rondo, bowed rondo, sonata rondo), there are also proven patterns in rock and pop music for setting up a song. Of course these are just a few options out of many. Ultimately, of course, the composer decides how his piece is constructed. An example of a typical structure of a rock song is the pattern.
ABABCDAB,
where A equals strophe, B equals refrain, C equals bridge, and D equals solo. Often a piece of music is introduced with a prelude. Intros often consist of the same chord progression as the stanza, but with different instrumentation, eg. B. without drums, without bass or distortion of the guitar in rock songs etc.

The device according to the invention initially comprises a device 10 for providing a similarity representation for the segments, wherein the similarity representation for each segment has an associated plurality of similarity values, the similarity values indicating how similar the segment is to each other segment. The similarity representation is preferably that in 7 shown segment similarity matrix. It has for each segment (in 7 Segments 1-10) has its own column, which has the index "j." Furthermore, the similarity representation has a separate row for each segment, one row being denoted by a row index i, hereinafter referred to as the exemplary segment 5. The element (5,5) in the main diagonal of the matrix of 7 is the similarity value of the segment 5 with itself, ie the maximum similarity value. Furthermore, the segment 5 is still medium-like to the segment No. 6, as it is by the element (6,5) or by the element (5,6) of the matrix in 7 is designated. In addition, the segment has 5 similarities to the segments 2 and 3, as represented by the elements (2,5) or (3,5) or (5,2) or (5,3) in 7 is shown. As for the other segments 1, 4, 7, 8, 9, 10, the segment No. 5 has a similarity as described in FIG 7 is no longer visible.

A plurality of similarity values associated with the segment is, for example, a column or a row of the segment similarity matrix in FIG 7 , which column or row indicates by its column / row index which segment it refers to, namely for example the fifth segment, and which row / column comprises the similarities of the fifth segment to each other segment in the piece , Thus, the plurality of similarity values is, for example, a row of the similarity matrix or, alternatively, a column of the similarity matrix of 7 ,

The device for grouping temporal segments of the piece of music further comprises a device 12 for calculating a similarity mean for a segment, using the segments and the similarity values of the plurality of similarity values associated with the segment. The device 12 is designed to z. For column 5 in 7 a similarity agent to calculate value. In a preferred embodiment, when the arithmetic mean is used, the device becomes 12 add the similarity values in the column and divide by the total number of segments. In order to eliminate self-similarity, the similarity of the segment to itself could also be deducted from the result of the addition, whereby, of course, a division should no longer be performed by all elements, but by all elements less 1.

The device 12 Alternatively, to calculate, one could also calculate the geometric mean, that is, square each similarity value of a column to sum the squared results, and then calculate a root from the summation result represented by the number of elements in the column (or number the elements in the column less 1) is to divide. Any other mean values, such as the median value, etc., are usable as long as the mean value for each column of the similarity matrix is adaptively calculated, that is, a value calculated using the similarity values of the plurality of similarity values associated with the segment.

The adaptively calculated similarity threshold then becomes a device 14 to assign a segment to a segment class. The device 14 The assignment is adapted to assign a segment to a segment class if the similarity value of the segment satisfies a predetermined condition with respect to the mean of similarity. For example, if the similarity mean is such that a larger value indicates greater similarity and a smaller value indicates lesser similarity, then the predetermined relationship will be that the similarity value of a segment must be equal to or above the similarity mean, thus the segment is assigned to a segment class.

In a preferred embodiment of the present invention, there are still other devices to realize specific embodiments, which will be discussed later. These devices are a segment selector 16 , a segment allocation conflicting device 18 , a segmentation correction device 20 and a segment class designator 22 ,

The segment selection device 16 in 1 is trained to look for each column in the matrix first 7 to calculate an overall similarity value V (j), which is determined as follows:

P is the number of segments. SÄ is the value of self-similarity of a segment to itself. Depending on the technique used, the value z. B. zero or one. The segment selection device 16 will first calculate the value V (j) for each segment to then find the vector element i of the maximum value vector V. In other words, this means that the column in 7 is chosen, which achieves the greatest value or score when adding up the individual similarity values in the column. For example, this segment could be segment no. 5 or column 5 of the matrix in 7 be, since this segment has at least a certain similarity with three other segments. Another candidate in the example of 7 could also be the segment with the number 7, since this segment also has a certain similarity to three other segments, which is even greater than the similarity of the segment 5 to the segments 2 and 3 (higher shade of gray in FIG 7 ).

For the following example, it is now assumed that the segment selector 16 select segment # 7 because it has the highest similarity score due to matrix elements (1,7), (4,7) and (10,7). In other words, this means that V (7) is the component of the vector V which has the maximum value among all the components of V.

Now, the similarity score of column 7, that is, for segment # 7, is still divided by the number "9" to get out of the facility 12 get the similarity threshold for the segment.

hereupon becomes in the segment similarity matrix for the seventh row or column checks which segment similarities above the calculated threshold value, d. H. with which segments the i-th Segment has an above-average similarity. All these segments will now be the same as the seventh segment assigned to a first segment class.

For the present For example, it is assumed that the similarity of the segment 10 to the segment 7 is below average, but that the similarities of segment 4 and segment 1 to segment 7 are above average. Therefore, be in the first segment class in addition to segment no. 7 also the segment No. 4 and segment No. 1. In contrast, the segment No. 10 due to the below-average similarity to segment no. 7 not classified in the first segment class.

After the assignment, the corresponding vector elements V (j) of all the segments which in this threshold value investigation become a cluster were set to 0. In the example these are beside V (7) also the components V (4) and V (1). This immediately implies that the 7th, 4th, and 1st columns of the matrix will no longer be available for a later maximum search, that they are zero, so they can not be a maximum.

This is roughly equivalent to the entries (1,7), (4,7), (7,7) and (10,7) of the segment similarity matrix are set to 0 become. The same procedure is used for column 1 (elements (1, 1), (4, 1) and (7, 1)) and the column 4 (elements (1,4), (4,4), (7,4) and (10, 4)). Due to the easier handling, however, the matrix becomes not changed, but become the components of V that are assigned to one Belonging to the segment, at the next Maximum search in a later Iteration step ignored.

In one next Iteration step will now be a new maximum among the still remaining elements of V, thus under V (2), V (3), V (5), V (6,), V (8), V (9) and V (10) are searched. Probably the segment will be # 5, which is V (5), the largest similarity score result. The second segment class then receives the segments 5 and 6. Due to the fact that the similarities to the segments 2 and 3 are below average, segments 2 and 3 become not placed in the second-order cluster. This will be the Elements V (6) and V (5) of vector V due to the assignment made set to 0 while nor the components V (2), V (3), V (8), V (9) and V (10) of the vector for the Selection of third order cluster remains.

Then again a new maximum among the mentioned remaining elements of V is searched for. The new maximum could be V (10), ie the component of V for segment 10. Segment 10 thus comes in the segment class of third order. Thus, it could also be found that the segment 7 also has an above-average similarity to the segment 10, although the segment 7 is already identified as belonging to the first segment class. Thus, an allocation conflict arises that is caused by the segment assignment conflicting mechanism 18 from 1 is resolved.

A simple way of dissolution could be just to not assign segment 7 into the third segment class and Z. For example, instead assign the segment 4 if for the segment 4 would not also conflict.

Preferably However, the similarity between the segment 7 and the segment 10 is not taken into account to let the similarity between 7 and 10 in the following algorithm.

Generally, the invention is designed not to discount the similarity between i and k. Therefore, the similarity values S _s (i, k) of segment i and k are compared with the similarity value S _s (i *, k), where i * is the first segment assigned to the cluster C *. The cluster or the segment class C * is the cluster to which segment k is already assigned on the basis of a previous examination. The similarity value S _s (i *, k) is decisive for the fact that the segment k belongs to the cluster C *. If S _s (i *, k) is greater than S _s (i, k), segment k remains in cluster C *. If S _s (i *, k) is smaller than S _s (i, k), the segment k is taken out of the cluster C * and assigned to the cluster C. For the first case, ie if the segment k does not change the cluster membership, a tendency towards the cluster i is noted for the cluster C *. Preferably, however, this tendency is also noted when segment k changes cluster membership. In this case, a tendency of this segment to the cluster in which it was originally recorded is noted. These tendencies can advantageously be achieved with a segmentation correction performed by the segmentation correction device 20 is executed, used.

The similarity value check is due to the fact that segment 7 is the "originating segment" in the first segment class is in favor of the first segment class. Segment 7 becomes its cluster affiliation (segment affiliation) do not change, but it will remain in the first segment class. this fact but it is taken into account that segment no. 10 in the third segment class is a trend towards first segment class is attested.

According to the invention is thus takes into account that especially for the segments, their segmental similarities to two different segment classes exist, these similarities nevertheless not be ignored, but possibly later the trend or tendency is still taken into account.

The Procedure will continue for so long until all segments in the segment similarity matrix are assigned are what is the case when all elements from vector V to zero are set.

This would be for the in 7 shown example mean that next, in the fourth segment class, the maximum of V (2), V (3), V (8), V (9), so the segment 2 and 3 are classified, then, in a fifth segment class, segments 8 and 9, respectively, until all segments have been assigned. This is the in 2 shown iterative algorithm finished.

The following details the preferred implementation of the segmentation correction device 20 based on 3 received.

Thus, when calculating the segment boundaries by means of the kernel correlation, but also when calculating segment boundaries by means of other measures, an over-segmentation of a piece often occurs, ie too many segment boundaries or generally too short segments are calculated. An over-segmentation, z. B. caused by an incorrect subdivision of the stanza is inventively corrected by the fact that due to the segment length and the information in which class segment a predecessor or successor segment has been sorted, is corrected. In other words, the correction serves to completely eliminate segments that are too short, ie to merge with adjacent segments, and segments that are short but are not too short, ie that are short in length but longer than the minimum length to undergo a special investigation into whether they may not yet be merged with a predecessor segment or a successor segment. In principle, successive segments belonging to the same segment class are always fused together. Results in 7 shown scenario z. B. that the segments 2 and 3 come in the same segment class, they are automatically merged together, while the segments in the first segment class, ie the segments 7, 4, 1 are spaced apart and therefore (at least initially) are not merged. This will be in 3 through a block 30 indicated. Now it is in a block 31 examines whether segments have a segment length smaller than a minimum length. Thus, there are preferably different minimum lengths.

It be relatively short segments shorter than 11 seconds (a first threshold) are, at all only examined while later even shorter segments (a second threshold that is smaller than the first one), which is shorter than 9 seconds are to be examined, and later remaining segments, the shorter than 6 seconds (a third threshold, which is shorter than the second threshold ) are treated again alternatively.

In the preferred embodiment of the present invention where this staggered length check occurs, the segment length check is in the block 31 initially aimed at finding the segments shorter than 11 seconds. For the segments that are longer than 11 seconds, no post processing is done, as indicated by a "No" on the block 31 is recognizable. For segments shorter than 11 seconds, a trend check (Block 32 ) carried out. Thus, it is first examined whether a segment is due to the functionality of the segment allocation conflicting device 18 from 1 has an associated trend or an associated trend. In the example of 7 this would be the segment 10 that has a trend towards the segment 7 or a trend towards the first segment class. If the tenth segment is shorter than 11 seconds, the in 7 Nevertheless, due to the trend check, nothing is done, since a fusion of the considered segment takes place only if it has a tendency not to any cluster, ie to any segment class, but a tendency to a cluster of an adjacent (before or after) segment , However, this is for the segment 10 in the in 7 example not shown.

In order to avoid too short segments which have no tendency to cluster an adjacent segment, the procedure is as in the blocks 33a . 33b . 33c and 33d in 3 is shown. Thus, nothing is done on segments longer than 9 seconds, but shorter than 11 seconds. They are left. In a block 33a however, for segments from cluster X that are shorter than 9 seconds, and where both the predecessor segment and the successor segment belong to cluster Y, an assignment to cluster Y is made, which automatically means that one Segment is merged with both the predecessor and the successor segment, so that an overall longer segment is formed, which consists of the considered segment as well as the predecessor and the successor segment. Thus, a subsequent merger may result in a combination of initially separate segments over an intervening segment to be merged.

In a block 33b is also what is done with a segment that is shorter than 9 seconds and that is the only segment in a segment group. For example, segment no. 10 is the only segment in the third segment class. If it was shorter than 9 seconds, it will automatically be assigned to the segment class to which segment no. 9 belongs. This automatically leads to a fusion of the segment 10 with the segment 9. If the segment 10 is longer than 9 seconds, then this merger is not performed.

In a block 33c then an examination is made for segments shorter than 9 seconds, which are not the only segment in a corresponding cluster X, than in a corresponding segment group. They are subjected to a closer examination, in which a Re regularity in the clustering sequence. First, all segments from segment group X shorter than the minimum length are searched for. Subsequently, it is checked for each of these segments whether the predecessor and successor segments each belong to one uniform cluster. If all predecessor segments are from a uniform cluster, all segments that are too short from cluster X are assigned to the predecessor cluster. If, on the other hand, all successor segments are from a uniform cluster, the segments too short from cluster X are each assigned to the successor cluster.

In a block 33d is executed, what happens, even if this condition is not fulfilled for segments that are shorter than 9 seconds. In this case, a novelty value check is performed by resorting to the novelty value curve, which in 9 is shown. In particular, the novelty curve, which resulted from the kernel correlation, is read out at the locations of the affected segment boundaries, and the maximum of these values is determined. If the maximum occurs at the beginning of a segment, the segments that are too short are assigned to the cluster of the successor segment. If the maximum occurs at a segment end, the segments that are too short are assigned to the cluster of the predecessor segment. Would that be in 9 With 90 If the segment has a segment that is shorter than 9 seconds, then the novelty check would be at the beginning of the segment 90 a higher novelty value 91 given as at the end of the segment, with the novelty value at the end of the segment with 92 is designated. This would mean that the segment 90 would be assigned to the successor segment because the novelty value to the successor segment is less than the novelty value to the predecessor segment.

Remain now still segments, the shorter than 9 seconds are and yet could not be merged so a staggered selection is made among them again. Especially now all the segments will be among the remaining segments, the shorter ones than 6 seconds are selected. The segments whose length is between 6 and 9 seconds out of this group are allowed "untouched".

The segments that are shorter than 6 seconds, but now all of them are based on the elements 90 . 91 . 92 subjected to novelty examination and assigned to either the predecessor or the successor segment, so that at the end of the in 3 Correction algorithm shown all too short segments, namely all segments below a length of 6 seconds, have been merged intelligently with predecessor and successor segments.

This inventive procedure has the advantage that no elimination of parts of the piece has been done So that's not a simple elimination of too short segments performed by zeroing but that is still the entire complete piece of music through represents the entirety of the segments is. Due to the segmentation is therefore no loss of information occurred, which would be, however, if one z. In response to the over-segmentation Simply eliminate all too short segments "regardless of losses" would.

Subsequently, reference will be made to 4a and 4b a preferred implementation of the segment class designator 22 from 1 shown. According to the invention, two clusters are assigned the labels "stanza" and "refrain" during labeling.

According to the invention will now not a biggest singular value a singular value decomposition and the associated cluster as refrain and the cluster for the second largest singular value used as a stanza. Furthermore, it is not assumed in principle that every song begins with a stanza, that is, the cluster with the first segment of the stanza cluster is and the other cluster is the refrain cluster. Instead According to the invention, the cluster in the candidate selection, which has the last segment, as a refrain and the other cluster is called a stanza.

Thus, for the two clusters finally available for the stanza / chorus selection ( 40 ), which cluster has the segment that occurs as the last segment of the segments of the two segment groups in the song history, to call the same as a refrain.

The last segment can actually be in the song last segment or a segment that occurs later in the song as all segments of the other segment class. Is this segment not that actually last segment in the song, this means that there is still an outro.

These Decision based on the realization that the refrain in the most cases comes in a song behind the last stanza, so directly as last segment of the song, if a piece z. B. is hidden with the chorus, or as a segment in front of an outro that follows a chorus and with that the piece is ended.

If the last segment is from the first segment group, then all segments of that first (most significant) segment class are referred to as a refrain, as it is by a block 41 in 4b is shown. In addition, in this case, all seg Elements of the other segment class to be selected are marked as "stanza", since typically of the two candidate segment classes one class will have the chorus and thus the other class will have the stanzas.

On the other hand results in the investigation in the block 40 That is, which segment class in the selection has the last segment in the music track progression, that this is the second, that is rather low-valued segment class, is written in a block 42 examines whether the second segment class has the first segment in the piece of music. This investigation is based on the knowledge that the likelihood is very high that a song starts with a verse, not a chorus.

Will the question in the block 42 answered "No", so if the second segment class does not have the first segment in the piece of music, the second segment class is called a chorus, and the first segment class is called a stanza, as in a block 43 is indicated. If, on the other hand, the query is in the block 42 answered "Yes", contrary to the rule, the second segment group is called a stanza and the first segment group as a refrain, as it is in a block 44 is indicated. The name in the block 44 This happens because the probability that the second segment class corresponds to the refrain is already quite low. If the improbability that a piece of music is introduced with a refrain comes to the fore, there are some signs of an error in clustering. B. that the last considered segment was erroneously assigned to the second segment class.

In 4b It was shown how the stanza / chorus determination was performed on the basis of two available segment classes. After this stanza / refrain determination then the remaining segment classes in a block 45 Where appropriate, an outro will be the segment class having the last segment of the piece per se, while an intro will be the segment class having the first segment of a piece per se.

The following is based on 4a It shows how to determine the two segment classes that represent the candidates for the in 4b given algorithm.

Generally Labeling is used to assign the label "stanza" and "refrain", with a segment group is marked as a stanza segment group, while the other segment group marked as refrain segment group. Basically, this concept is based on the assumption (A1) that the two clusters (segment groups) with the highest similarity values, So cluster 1 and cluster 2, the chorus and stanza clusters correspond. The last of these two clusters is the refrain cluster, assuming that a stanza follows a chorus.

The Experience from numerous tests has shown that cluster 1 in the most cases corresponds to the refrain. For Cluster 2, the assumption (A1) is often not fulfilled. These The situation usually occurs when there is still a third one, either often repeating part in the piece there, z. B. a bridge, with a high similarity of intro and outro, or for the not uncommon case that a segment in the piece one high similarity to the chorus, thus a high overall similarity has, the similarity to the chorus but just not big enough to make it to the cluster 1 belong to.

Research has shown that this situation often occurs for variations of the refrain at the end of the piece. In order to mark (label) chorus and stanza correctly with as high a certainty as possible, the in 4b described segment selection in that, as shown in 4a is shown, the two candidates for the verse-chorus selection is determined depending on the existing in the same segments.

First, in one step 46 the cluster or the segment group with the highest similarity value (value of the component of V which is once a maximum for the segment class determined first, ie segment 7 in the example of FIG 7 , was), that is, the segment group that at the first pass of 1 has been identified as the first candidate included in the stanza refrain selection.

It is now questionable which other segment group will be the second participant in the verse-chorus selection. The most likely candidate is the second highest segment class, that is, the segment class that passes through the in 1 described concept is found. However, this does not always have to be this way. Therefore, for the second highest segment class (segment 5 in 7 ), so cluster 2 checks if this class has only a single segment or exactly two segments, one of the two segments being the first segment and the other segment of the two being the last segment in the song (block 47 ).

If, on the other hand, the answer to the question is "no", then the second-highest segment class, for example, has at least three segments, or two segments, one of which is within the piece and not at the "edge" of the piece, so the second segment class remains in the selection and will henceforth be called "Se cond cluster ".

Will the question in the block 47 on the other hand answered with "yes", the second highest class is eliminated (Block 48a ), it is replaced by the segment class that occurs most frequently in the entire song (in other words, that contains the most segments) and does not correspond to the highest segment class (cluster 1). This segment class will henceforth be referred to as "Second Cluster".

"Second Cluster" still has to compete with a third segment class, as explained below. 48b ), which is referred to as a "third cluster" to end up as a candidate to end the selection process.

The Segment class "Third Cluster "corresponds to the Cluster, which is the most common occurs in the entire song, but neither the highest segment class (cluster 1) nor the segment class "Second Cluster "corresponds the next most common, so to speak (often the same) occurring clusters after cluster 1 and "second clusters".

With regard to the so-called bridge problem, it is now checked for "third cluster" whether it belongs more in the stanza-refrain selection than "second cluster" or not. This happens because "Second Cluster" and "Third Cluster" often occur the same number of times, so one of them may represent a bridge or another recurring intermediate part. To ensure that the segment class is chosen by the two most likely to match the stanza or chorus, not a bridge or other intermediate, those in the blocks 49a . 49b . 49c performed investigations.

The first investigation in block 49a is to examine whether each segment of ThirdCluster ei ne ne certain minimum length has, as a threshold z. B. 4% of the total song length is preferred. Other values between 2% and 10% can also lead to meaningful results.

In a block 49b then it is examined if ThirdCluster has a bigger total part of the song than SecondCluster. To do this, the total time of all segments in ThirdCluster is added up and compared to the corresponding total of all Segments in SecondCluster, where ThirdCluster has a larger overall song share than SecondCluster if the accumulated segments in ThirdCluster are greater than the accumulated SecondCluster segments.

In the block 49c Finally, it is checked whether the distance of the segments from third cluster to the segments from cluster 1, ie the most frequent cluster, is constant, ie whether a regularity is evident in the sequence.

If all these three conditions are answered with "yes", ThirdCluster enters the stanza-refrain selection, but if at least one of these conditions is not met, ThirdCluster does not enter the stanza-refrain selection, instead SecondCluster is included in the stanza -Refrain selection as it is by a block 50 in 4a is shown. This completes the "Candidate search" for the verse-chorus selection, and the in 4b algorithm which finally determines which segment class comprises the stanzas and which segment class comprises the chorus.

At this point it should be noted that the three conditions in the blocks 49a . 49b . 49c alternatively could be weighted so that z. Eg a no answer in the block 49a then "overruled" when both the query in the block 49b as well as the query in the block 49c Alternatively, a condition of the three conditions could be emphasized so that, for example, only the regularity of the sequence between the third segment class and the first segment class is checked, while the queries in the blocks 49a and 49b can not be performed or performed only if the query is in the block 49c is answered with "No", however, eg a relatively large total proportion in the block 49b and relatively large minimum quantities in the block 49a be determined.

Alternative combinations are also possible, and for a low-level investigation only the query of one of the blocks 49a . 49b . 49c will be sufficient for certain implementations.

The following are exemplary implementations of the block 526 set out for performing a music summary. So there are different possibilities, which can be stored as music summary. Two of them are described below, namely the option entitled "Refrain" and the option entitled "Medley".

The refrain option is to choose a version of the chorus as a summary. This will attempt to choose a chorus version that lasts between 20 and 30 seconds if possible. If a segment with such a length is not contained in the refrain cluster, a version is chosen which has the smallest possible deviation to a length of 25 seconds. If the selected chorus is longer than 30 seconds, it will be blanked out over 30 seconds in this embodiment If it is shorter than 20 seconds, it will be extended to 30 seconds with the following segment.

The Storing a medley for the second possibility is more like an actual one Summary of a piece of music. Here are a section of the stanza, a section of the chorus and a section of a third segment in their actual chronological order constructed as a medley. The third segment is selected from a cluster, that's the biggest share has on the song and is not stanza or chorus.

• – „drittes Segment"-Strophe-Refrain;
• – Strophe-Refrain-„drittes Segment"; oder
• – Strophe-„drittes Segment"-Refrain.

The following priority is used to search for the most appropriate sequence of segments:

• - "third segment" stanza refrain;
• - stanza-chorus- "third segment"; or
• - stanza "third segment" refrain.

The selected Segments are not installed in their full length in the medley. The length is preferably set to fixed 10 seconds per segment so that Total again a summary of 30 seconds arises. alternative However, values are also readily feasible.

Preferably, the computation time savings after the feature extraction in the block 502 or after the block 508 a grouping of several feature vectors in the block 510 performed by averaging over the grouped feature vectors. In the next processing step, the calculation of the similarity matrix, the grouping can save computing time. To calculate the similarity matrix, a distance is determined between all possible combinations of two feature vectors. This results in n vectors over the entire piece of nxn calculations. A grouping factor g indicates how many consecutive feature vectors are grouped into a vector by averaging. This can reduce the number of calculations.

The Grouping is also a kind of noise reduction, with small changes in the characteristic expression of successive vectors compensated on average who the. This property has a positive effect on finding huge Song structures.

The inventive concept allows, using a special music player through the calculated segments to navigate and selectively select individual segments, so that a consumer in a music store by, for example, pressing a specific key or by activating a specific software command readily jump to the chorus of a piece immediately to determine if he likes the chorus, and then, maybe listen to a stanza, so that the consumer finally a Can make a purchase decision. So it is a prospective buyer comfortably possible, from one piece to hear exactly that what interests him particularly, while he z. B. the solo or the bridge then actually for the listening save at home.

alternative is the concept according to the invention also for one Music shop of great Advantage, because the customer targeted and thus listen in quickly and ultimately can buy, so that customers do not wait long for a listen have to, but also quickly come to the series. This is because that a user is not constantly but must roll back and forth, but targeted and quickly all Information of the piece gets he also wants to have.

Further be a significant advantage of the inventive concept pointed out, namely that especially due to the post-correction of the segmentation no Information of the piece get lost. Thus, although all segments, which are preferably shorter than 6 seconds are, with the predecessor or successor segment merged. However, no segments, as short as they are, eliminated. This has the advantage that the Users basically everything in the piece listen to can, so a short but a user but very good striking piece, which would have been omitted in a segmentation post-correction, which is actually a Section of the piece completely eliminated, nevertheless available to the user, perhaps he just because of the short distinctive piece one well considered Can make a purchase decision.

However, the present invention is also applicable in other application scenarios, for example in advertising monitoring, ie where an advertiser wants to check whether the audio piece for which he has bought advertising time, has actually been played over the entire length. An audio piece may include, for example, music segments, speaker segments, and noise segments. The segmentation algorithm, ie the segmentation and subsequent classification into segment groups, then makes it possible to carry out a quick and considerably less complicated check than a complete sample-wise comparison. The efficient checking would simply consist of segment class statistics, that is, a comparison of how many segment classes have been found and how many segments are in each segment class, with a default given the ideal ad item. Thus it is easily possible for an advertiser to recognize whether a broadcaster or television station actually has all the main parts (Ab sections) of the commercial signal has broadcast or not.

The The present invention is further advantageous in that it for research in large music databases can be used, for example, only the refrains of many Listen through music to then perform a music program selection. In this case, only would individual segments from the segment class marked "Refrain" to be selected from many different pieces and provided by a program provider. alternative could Also, there is an interest to all of one performer z. B. to compare guitar solo with each other. According to the invention, these also readily be provided by always one or multiple segments (if any) in the segment class called "Solo" from a big one Number of pieces of music z. B. joined together and provided as a file.

Again other applications exist in it, stanzas from various audio pieces and choruses, which will be particularly interesting for DJs and completely New opportunities the creative music synthesis opens, which simple and, most of all, automatically accurate can. This is the concept according to the invention easily automatable, as there is nowhere a user intervention requires. This means that users of the inventive concept by no means need a special education, except z. B. a common skill in dealing with normal software user interfaces.

ever according to the practical conditions, the inventive concept be implemented in hardware or in software. The implementation can be on a digital storage medium, especially a floppy disk or CD with electronically readable control signals, the so can interact with a programmable computer system that the corresponding procedure is carried out. Generally exists The invention thus also in a computer program product with a stored on a machine-readable carrier Program code for execution of the method according to the invention, when the computer program product runs on a computer. In other In words Thus, the invention provides a computer program with a program code to perform of the procedure when the computer program is on a computer expires.

Claims (21)

Device for grouping temporal segments of an audio piece, which is subdivided into main parts which repeatedly occur in the audio piece, into different segment classes, one segment class being assigned to a main part, having the following features: 10 ) for providing a similarity representation for the segments, the similarity representation having for each segment an associated plurality of similarity values, the similarity values indicating how similar the segment is to each other segment of the audio piece; a facility ( 12 ) for calculating a similarity threshold for a segment using the plurality of similarity values associated with the segment; and a facility ( 14 ) for assigning a segment to a segment class if the similarity value of the segment satisfies a predetermined condition regarding the similarity threshold. Apparatus according to claim 1, further comprising: a segment selector ( 16 ) for determining an extreme segment whose associated plurality of similarity values taken together has an extremum, the device ( 12 ) for calculating to calculate the similarity threshold for the extreme segment, and wherein the device ( 14 ) is adapted to assign to mark the segment class with an indication of the extreme segment. Device according to Claim 1 or 2, in which the device ( 14 ) is arranged to allocate a segment which does not satisfy the predetermined condition with respect to the similarity threshold, not to allocate the segment class but to leave for an assignment to another segment class, and wherein the device ( 14 ) is adapted to assign to no longer consider the similarity value of the associated segment in an assignment to another segment class for an associated segment. Device according to one of the preceding claims, in which the device ( 12 ) for calculating the similarity threshold in a later pass to ignore similarity values for earlier assigned segments in the plurality of similarity values after an earlier assignment of a segment class, and wherein the device ( 14 ) is arranged to assign to perform an assignment to a segment class other than the segment class in a later pass in a later pass. Device according to one of the preceding Claim further comprising: a segment allocation conflicting device ( 18 ), which is designed to, in the case in which by the device ( 14 ) for assigning a conflict segment to two different segment classes, determining a first similarity value of the conflict segment with a segment of a first segment class, and determining a second similarity value of the conflict segment with a segment of a second segment class, and wherein the device ( 14 ) is arranged to assign, in the case where the second similarity value indicates a greater similarity of the conflict segment with the segment of the second segment class, to remove the conflict segment from the first segment class and assign it to the second segment class. Apparatus according to claim 5, wherein the segment allocation conflict means ( 18 ) is adapted, in the case of removal of the segment from the first segment class, to assign the segment a tendency to the first segment class or, in the case of an unsuccessful removal of the segment, to assign the segment a tendency to the second segment class. Device according to one of the preceding claims, further comprising: a segmentation correction device ( 20 ) which is adapted to correct a segmentation of the audio piece, wherein the segmentation correction device ( 20 ) is adapted to merge segments with a preceding segment or segment depending on segment class information for the segments. Device according to Claim 7, in which the segmentation correction device ( 20 ) is adapted to determine, for a segment shorter than a predetermined minimum length, whether a trend of the segment coincides with a segment class belonging to an immediately preceding temporal segment, and in this case the segment with the immediately preceding segment or for a segment shorter than a predetermined minimum length, determining whether a trend of the segment is indicative of a segment class belonging to a temporally succeeding segment, and in this case the segment having the segment temporally immediately following segment merge. Device according to one of the preceding claims, comprising a segmentation correction device ( 20 ), which is designed to fuse temporally successive segments belonging to the same segment class. Device according to one of Claims 6 to 9, in which the segmentation correction device ( 20 ) is adapted to select only segments having a temporal segment length shorter than a predetermined minimum length for correcting the segments. Device according to Claim 10, in which the segmentation correction device ( 20 ) is adapted to merge a selected segment of a second segment class, whose temporal precursor segment, and its temporal successor segment of a first segment class, with the predecessor segment and the successor segment. Device according to Claim 10 or 11, in which the segmentation correction device ( 20 ) is adapted to merge a segment, which is in a segment class comprising only a single segment, with the preceding segment or segment. Device according to Claim 10, 11 or 12, in which the segmentation correction device ( 20 ) is adapted to merge a plurality of selected segments that are in the same segment class, each with a temporally preceding segment or a temporally subsequent segment, if all selected segments of the segment class precursor segments from the same segment class or successor segments from and the same segment class. Device according to one of claims 7 to 13, wherein the Segmentierungskorrektureinrichtung formed is to go for a segment having a shorter time length than a predetermined minimum length, determine a first novelty value at a beginning of the segment and determine a second novelty value at one end of the segment and to merge the segment with a subsequent segment, if the first novelty value is greater than is the second novelty value, or around the segment with a time merging previous segment when the first novelty value less than the second novelty value. Device according to one of Claims 7 to 14, in which the segmentation correction device ( 20 ) is designed to perform various corrective actions depending on different predetermined segment lengths. Device according to one of the preceding claims, further comprising a segment class designating means, which is designed to differentiate a designation of segment classes depending on a temporal position of segments in different segment classes To carry out major parts. Device according to Claim 16, in which the segment class designator ( 22 ) is designed to select two segment class candidates before a segment class designation into a main part "stanza" and into a main part "chorus" for consideration of the segments in the segment classes. Device according to Claim 16 or 17, in which the segment class designation device ( 22 ) is adapted to designate a candidate segment class as a refrain class if the candidate segment class comprises the segment occurring in the audio piece after all other segments of the other candidate segment class in time. Device according to one of the claims 16 to 18, wherein the segment class designation means ( 22 ) is adapted to designate a candidate segment class as a stanza class if the candidate segment class does not include the segment occurring in the audio piece after all other segments of the other candidate segment class in time. A method for grouping temporal segments of an audio piece, which is subdivided into main parts repeatedly occurring in the audio piece, into different segment classes, wherein a segment class is assigned to a main part, comprising the following steps: 10 ) a similarity representation for the segments, the similarity representation for each segment having an associated plurality of similarity values, the similarity values indicating how similar the segment is to each other segment of the audio piece; To calculate ( 12 ) a similarity threshold for a segment using the plurality of similarity values associated with the segment; and Assign ( 14 ) of a segment to a segment class when the similarity value of the segment satisfies a predetermined condition regarding the similarity threshold. Computer program with a program code to run the Method according to claim 20, when the computer program up a computer is running.