US20110035035A1 - Method and system for analyzing digital audio files - Google Patents

Method and system for analyzing digital audio files Download PDF

Info

Publication number
US20110035035A1
US20110035035A1 US12/906,584 US90658410A US2011035035A1 US 20110035035 A1 US20110035035 A1 US 20110035035A1 US 90658410 A US90658410 A US 90658410A US 2011035035 A1 US2011035035 A1 US 2011035035A1
Authority
US
United States
Prior art keywords
audio
cluster centers
bin
bins
match
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/906,584
Inventor
Rehan M. Khan
George Tzanetakis
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Adeia Technologies Inc
Original Assignee
Rovi Technologies Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Rovi Technologies Corp filed Critical Rovi Technologies Corp
Priority to US12/906,584 priority Critical patent/US20110035035A1/en
Publication of US20110035035A1 publication Critical patent/US20110035035A1/en
Assigned to JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT reassignment JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: APTIV DIGITAL, INC., A DELAWARE CORPORATION, GEMSTAR DEVELOPMENT CORPORATION, A CALIFORNIA CORPORATION, INDEX SYSTEMS INC, A BRITISH VIRGIN ISLANDS COMPANY, ROVI CORPORATION, A DELAWARE CORPORATION, ROVI GUIDES, INC., A DELAWARE CORPORATION, ROVI SOLUTIONS CORPORATION, A DELAWARE CORPORATION, ROVI TECHNOLOGIES CORPORATION, A DELAWARE CORPORATION, STARSIGHT TELECAST, INC., A CALIFORNIA CORPORATION, UNITED VIDEO PROPERTIES, INC., A DELAWARE CORPORATION
Assigned to UNITED VIDEO PROPERTIES, INC., GEMSTAR DEVELOPMENT CORPORATION, STARSIGHT TELECAST, INC., INDEX SYSTEMS INC., TV GUIDE INTERNATIONAL, INC., ALL MEDIA GUIDE, LLC, APTIV DIGITAL, INC., ROVI CORPORATION, ROVI TECHNOLOGIES CORPORATION, ROVI SOLUTIONS CORPORATION, ROVI GUIDES, INC. reassignment UNITED VIDEO PROPERTIES, INC. PATENT RELEASE Assignors: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/683Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63Querying
    • G06F16/632Query formulation
    • G06F16/634Query by example, e.g. query by humming
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63Querying
    • G06F16/635Filtering based on additional data, e.g. user or group profiles
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63Querying
    • G06F16/638Presentation of query results
    • G06F16/639Presentation of query results using playlists
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99944Object-oriented database structure
    • Y10S707/99945Object-oriented database structure processing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99948Application of database or data structure, e.g. distributed, multimedia, or image

Definitions

  • the present invention relates to analyzing audio files and more particularly to presenting a playlist based upon listener preferences and audio file content.
  • the Internet connects thousands of computers world wide into a vast network using well-known protocols, for example, Transmission Control Protocol (TCP)/Internet Protocol (IP).
  • TCP Transmission Control Protocol
  • IP Internet Protocol
  • Information on the Internet is stored world wide as computer files, mostly written in the Hypertext Mark Up Language (“HTML”).
  • HTML Hypertext Mark Up Language
  • the collection of all such publicly available computer files is known as the World Wide Web (WWW).
  • the WWW is a multimedia-enabled hypertext system used for navigating the Internet and is made up of hundreds of thousands of web pages with audio, images, text and video files. Each web page can have connections to other pages, which may be located on any computer connected to the Internet.
  • a typical Internet user uses a client program called a “Web Browser” to connect to the Internet.
  • a user can connect to the Internet via a proprietary network, such as America Online or CompuServe, or via an Internet Service Provider, e.g., Earthlink.
  • a Web Browser may run on any computer connected to the Internet. Currently, various browsers are available of which two prominent browsers are Netscape Navigator and Microsoft Internet Explorer. The Web Browser receives and sends requests to a web server and acquires information from the WWW.
  • a web server is a program that, upon receipt of a request, sends the requested data to the requesting user.
  • URL Uniform Resource Locator
  • HTTP Hypertext Transport Protocol
  • WAIS Wide Area Information Service
  • FTP File Transport Protocol
  • HTTP Hypertext Transfer Protocol
  • the header of self-describing formats contain parameters of a sampling device and may also include other information (e.g. a human-readable description of sound, or a copyright notice etc.).
  • Other information e.g. a human-readable description of sound, or a copyright notice etc.
  • Headerless formats define single encoding and usually allow no variation in device parameters (except sometimes for sampling rates). The following are a few examples of Headerless formats:
  • music listeners can store audio files
  • conventional music search techniques do not allow a music listener to search for music based upon audio file content.
  • Conventional systems also do not allow a music listener to generate play lists based upon music listener preferences and/or audio file content.
  • the present invention solves the foregoing drawbacks by providing a method and system for analyzing audio files.
  • Plural audio file feature vector values based on an audio file's content are determined and the audio file feature vectors are stored in a database, that also stores other pre-computed audio file features.
  • the process determines if the audio file's feature vectors match the stored audio file vectors.
  • the process also associates a plurality of known attributes to the audio file.
  • the present invention includes a system for analyzing audio files that includes a playlist generator that determines a plurality of audio file vectors based upon an audio file's content; and a signature comparator between the playlist generator and a database, wherein the database stores a plurality of audio file vector values of plural music samples.
  • the signature comparator compares input audio samples with previously stored audio samples in the database.
  • a user interface that allows a music listener to input search request for searching music based upon attributes that define music content.
  • the present invention includes a method for determining audio signatures for input audio samples.
  • the process extracts plural features representing the input audio samples, wherein the features are extracted by Fourier transform analysis.
  • the process also identifies a set of representative points based upon the plural features, and determines a codebook of plural elements for mapping the representative points to the elements of the codebook.
  • the present invention includes a method for comparing input audio signatures with pre-computed stored audio signatures.
  • the process determines a query signature based upon the input audio signature and divides the query signature into a string of characters; and compares the string of characters to stored pre-computed audio signatures.
  • the present invention divides an input audio sample into bins and determines a plurality of features describing the bins. Thereafter, the process determines a univariate signal based upon the plural features and computes an audio signature based upon the univariate signal.
  • One advantage of the foregoing aspects of the present invention is that unique audio signatures may be assigned to audio files. Also various attributes may be tagged to audio files.
  • the present invention can generate a customized playlist for a user based upon audio file content and the attached attributes, hence making the music searching experience easy and customized.
  • FIG. 1 illustrates a computing system to carry out the inventive technique.
  • FIG. 2 is a block diagram of the architecture of the computing system of FIG. 1 .
  • FIG. 3 is a block diagram of the Internet Topology.
  • FIG. 4 is a block diagram of the architecture of the present system.
  • FIG. 5 is a block diagram showing the architecture of a playlist generator.
  • FIG. 6 is a flow diagram of computer executable process steps for analyzing an audio file.
  • FIG. 6A is a graphical illustration of an audio file's content.
  • FIG. 7 is a flow diagram of computer executable process steps for comparing input audio files with stored audio data.
  • FIG. 8 is a flow diagram of computer executable process steps of generating a playlist, according to the present invention.
  • FIG. 9 is a flow diagram of computer executable process steps for determining audio signatures based upon one aspect of the present invention.
  • FIG. 10A shows a set of representative points randomly scattered and used for vector quantization, according to another aspect of the present invention.
  • FIG. 10B is a flow diagram of computer executable process steps for performing vector quantization, according to another aspect of the present invention.
  • FIG. 10C is a flow diagram of computer executable process steps for determining audio signatures, according to yet another aspect of the present invention.
  • FIG. 1 is a block diagram of a computing system for executing computer executable process steps according to one embodiment of the present invention.
  • FIG. 1 includes a host computer 10 and a monitor 11 .
  • Monitor 11 may be a CRT type, a LCD type, or any other type of color or monochrome display.
  • Also provided with computer 10 is a keyboard 13 for entering text data and user commands, and a pointing device 14 for processing objects displayed on monitor 11 .
  • Computer 10 includes a computer-readable memory medium such as a rotating disk 15 for storing readable data.
  • disk 15 can store application programs including web browsers by which computer 10 connects to the Internet and the systems according to the present invention as described below.
  • Computer 10 can also access a computer-readable floppy disk storing data files, application program files, and computer executable process steps embodying the present invention or the like via a floppy disk drive 16 .
  • a CD-ROM interface (not shown) may also be provided with computer 10 to access application program files, audio files and data files stored on a CD-ROM.
  • a modem, an integrated services digital network (ISDN) connection, or the like also provides computer 10 with an Internet connection 12 to the World Wide Web (WWW).
  • the Internet connection 12 allows computer 10 to download data files, audio files, application program files and computer-executable process steps embodying the present invention.
  • Computer 10 is also provided with external audio speakers 17 A and 17 B to assist a listener to listen to music either on-line, downloaded from the Internet or off-line using a CD (not shown). It is noteworthy that a listener may use headphones instead of audio speakers 17 A and 17 B to listen to music.
  • FIG. 2 is a block diagram showing the internal functional architecture of computer 10 .
  • Computer 10 includes a CPU 201 for executing computer-executable process steps and interfaces with a computer bus 208 .
  • Also shown in FIG. 2 are a WWW interface 202 , a display device interface 203 , a keyboard interface 204 , a pointing device interface 205 , an audio interface 209 , and a rotating disk 15 .
  • Audio Interface 209 allows a listener to listen to music, On-line (downloaded using the Internet or a private network) or off-line (using a CD, not shown).
  • disk 15 stores operating system program files, application program files, web browsers, and other files. Some of these files are stored on disk 15 using an installation program. For example, CPU 201 executes computer-executable process steps of an installation program so that CPU 201 can properly execute application programs.
  • a random access main memory (“RAM”) 206 also interfaces to computer bus 208 to provide CPU 201 with access to memory storage.
  • CPU 201 stores and executes the process steps out of RAM 206 .
  • ROM 207 is provided to store invariant instruction sequences such as start-up instruction sequences or basic input/output operating system (BIOS) sequences for operation of keyboard 13 .
  • invariant instruction sequences such as start-up instruction sequences or basic input/output operating system (BIOS) sequences for operation of keyboard 13 .
  • BIOS basic input/output operating system
  • the present invention is not limited to the computer architecture described above.
  • Systems comparable to Computer 10 for example, portable devices or hand held computing devices that can be connected to the Internet may also be used to implement the present inventive techniques.
  • FIG. 3 shows a typical topology of a computer network with computers similar to computer 10 , connected to the Internet.
  • three computers X, Y and Z are shown connected to the Internet 302 via Web interface 202 , through gateway 301 , where gateway 301 can interface numerous computers.
  • Web interface 202 may be a modem, network interface card or a unit for providing connectivity to other computer systems over a network using protocols such as X.25, Ethernet or TCP/IP, or to any device that allows direct or indirect computer-to-computer communications.
  • the invention is not limited to a particular number of computers. Any number of computers that can be connected to the Internet 302 or to any other computer network may be used to implement the present inventive techniques.
  • FIG. 3 further also shows a second gateway 303 that connects a network of web servers 304 and 305 to the Internet 302 .
  • Web servers 304 and 305 may be connected with each other over a computer network.
  • Web servers 304 and 305 can provide content including music samples and audio clips to a user from database 306 and/or 307 .
  • Web servers 304 and 305 can also host the system according to the present invention.
  • a client side web server 308 that can be provided by an Internet service provider.
  • FIG. 4 shows a block diagram of a system used for analyzing audio files, according to the present invention.
  • Audio files may be stored on rotating disk 15 in a music listener's computer 10 , or at any remote computer 10 connected to the Internet 302 .
  • a playlist generator 400 accesses audio files stored on rotating disk 15 .
  • Playlist generator 400 is an application program that can be located on remote server 304 or on a music listener's computer 10 .
  • Playlist generator 400 scans and analyzes audio files stored on rotating disk 15 and computes audio signatures that uniquely and compactly identify the content of the audio file.
  • An audio signature is computed only once for each file and stored in a local database.
  • the computed audio signature that compares the analyzed audio files with previously analyzed audio data is stored in a central database 401 .
  • Central database 401 stores a plurality of feature vector values as discussed below.
  • Central database 401 also includes data similar to the data stored in a production database that is described in U.S.
  • Customized play lists are generated after audio file content is compared to the data stored in central database 401 , as described below.
  • a user interface 400 A is also provided that allows a music listener to input preferences for generating play lists.
  • User interface 400 A may be a separate component or integrated with playlist generator 400 .
  • One such user interface is described in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, incorporated herein by reference in its entirety.
  • FIG. 5 is a block diagram showing various components of playlist generator 400 .
  • Playlist generator 400 includes an audio file analyzer 400 B and a Signature Comparator 400 C.
  • Audio file analyzer 400 B receives and analyzes audio files as described below. Audio files may be stored on rotating disk 15 or may be acquired from a remote computer.
  • the Signature Comparator 400 C receives user requests from UI 400 A and obtains music or a list of music based upon user preferences, as described below. It is noteworthy that audio file analyzer 400 B and signature comparator 400 C may be integrated into a single module to determine audio signature and implement the various aspects of the present invention.
  • FIG. 6 is a process flow diagram of computer executable process steps for analyzing audio files according to the present invention.
  • audio analyzer 400 B receives audio files (“input audio files”). Audio files may be stored on a user's disk 15 or at webserver 304 connected to the Internet or on any private network. Audio analyzer 400 B may seek audio files based upon user input in UI 400 A or may receive audio files from a designated source.
  • audio analyzer 400 B analyzes input audio file content and computes a set of parameters (“audio file vectors”). Audio file vectors can uniquely identify audio files and can also be used to assign a unique “audio signature” for an input audio file. Details of determining audio signatures are described below. These audio signatures need be computed only once for a given audio file and can thereafter be stored either locally or remotely and referred to as necessary.
  • step S 603 input audio file vector values for specific bins and the audio signature are stored in a database.
  • a database is a central database 401 .
  • Audio file signatures are sets or vectors of values based upon the audio segments as shown in FIG. 6A .
  • FIG. 7 is a flowchart of computer executable process steps that allows playlist generator 400 to compare audio file content with stored and analyzed audio file data.
  • Such audio files may be stored on rotating disk 15 or on a remote computer.
  • playlist generator 400 scans an audio file(s) stored on disk 15 . Audio files may be pulled by playlist generator 400 from disk 15 , or a music listener may send audio files to playlist generator 400 via the Internet 302 or a private network.
  • playlist generator 400 determines the audio signature of input audio file(s).
  • An audio signature may be determined by the process described below.
  • step S 703 playlist generator 400 transfers the audio signature and audio file vector values (V t1 . . . V tk ) to Signature Comparator 400 C.
  • Playlist generator 400 also commands the Signature Comparator 400 C to compare the audio signature and feature vectors values determined in step S 702 with historical audio signatures and audio file vector values stored in central database 401 . Thereafter, the Signature Comparator 400 C compares input audio signature and audio file vector values with historical audio signatures and audio file vector values, stored in database 401 .
  • Step S 704 Signature Comparator 400 C determines whether the input audio signature and audio file vectors values match stored audio signatures and vector values. If the input audio file vector values do not match with any stored audio data (audio signature and feature vector), then in step S 705 input audio file vector values are stored in central database 401 .
  • step S 706 the process determines if the stored entries are confirmed or provisional.
  • a confirmed entry in database 401 is an entry that has been ratified by multiple sources.
  • a provisional entry in database 401 is one that is not confirmed.
  • step S 707 other feature values stored in database 401 are associated with the audio file.
  • Examples of such feature vectors are provided in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, filed on Mar. 22, 2000, assigned to the assignee herein, and incorporated by reference in its entirety.
  • Associating a plurality of feature vectors allows a listener to search for music based upon content. For example, feature values associated with a given audio file may indicate that it is Rock music with a slow tempo and a smooth, female singer accompanied by a band featuring a prominent saxophone.
  • Some of the features that can be associated with the audio files are:
  • step S 708 the process converts the provisional rating to a confirmed rating.
  • FIG. 8 is a flow diagram of computer executable process steps for generating a customized music list (“playlist”) based upon user defined parameters.
  • a music listener inputs a request for a playlist.
  • UI 400 A ( FIG. 4 ) may be used by a music listener to input user preferences.
  • UI 400 A is described in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, filed on Mar. 22, 2000, and incorporated herein by reference.
  • a user may request a playlist by specifying emotional quality, voice quality, instrument or genre vector, tempo, artist, album title and year of release, etc.
  • playlist generator 400 scans audio files stored on user's disk 15 .
  • One methodology of analyzing audio files is provided above in FIG. 7 . Audio files may also be acquired form a remote computer connected to the Internet and analyzed as shown above.
  • Step S 803 Signature Comparator 400 C searches for music based upon analyzed audio file data.
  • step S 804 playlist generator 400 compiles a playlist based upon user preferences and the compared data.
  • Audio signatures identify audio files and are based upon the signal characteristics of a particular audio file sample. If the audio signatures of a group of audio files is known then a new audio file may be compared with the known group of audio signatures. Audio signature is a representation of an audio file that assists in comparing audio samples. Audio signature may be developed for any audio file and whenever two audio samples overlap, the audio signatures of the samples will also overlap. It is this property that assists comparison of audio samples.
  • FIG. 9 is a flow diagram showing process steps for determining audio signatures according to one aspect of the present invention.
  • step S 901 an audio sample is represented as a signal.
  • the signal is then used to determine a set of parameters or features that describe the signal.
  • FIG. 6A shows an example of representing an audio file over time.
  • step S 902 the determined features from step S 901 are used to compute an audio signature for the audio sample.
  • step S 903 the audio signature determined in step S 902 is compared to pre-computed audio signatures stored in database 401 .
  • an audio file is transformed into a raw intensity signal and the transformed signal is used to compute a set of features or parameters.
  • the features are computed from sequential (possibly overlapping) sets of raw intensity signal and transforms the intensity signal into a time series of feature values.
  • This multivariate time series of feature vectors is then compressed into a 1-dimensional vector of elements. This compression is achieved using vector quantization of the multivariate features, so that each point in the feature space can be mapped onto an element in a finite codebook. Thereafter, the computed 1-dimensional vector (1-d string) is matched against known values stored in a database using fast string matching algorithms to retrieve a match.
  • pre-processing is used to convert an audio file whose signature is to be calculated, to a standard reference format. More specifically mono audio files of 16-bit samples at 22050 Hz sampling rate are used as a reference format. Standard commercially available software converters and decoders may be used for this purpose. More specifically Sox sound converting utility that can be downloaded from the Internet address www.spies.com/Sox/ and Xaudio mp3 decoder downloaded from a website located at www.xaudio.com may be used.
  • step S 901 certain features or parameters are extracted from an audio file signal.
  • the features of this methodology are based on Short Time Fourier Transform (STFT) analysis.
  • STFT is a signal processing analysis technique that processes audio file signals in samples designated as bins or windows ( FIG. 6A ) and then a Discrete Fourier Transform (DFT) is determined for the bins or windows.
  • DFT Discrete Fourier Transform
  • step S 901 The following STFT-based features may be extracted in step S 901 :
  • Spectral Centroid is the balancing point of the spectrum (The spectrum is the representation of audio intensity over time. ( FIG. 6A )) magnitude and is defined as:
  • N is the FFT window size and the Mi's are the frequency bins of the spectrum magnitude.
  • Spectral Rolloff like the spectral centroid is another measure of the shape of the spectrum. It is defined as:
  • the Mi's are the frequency bins of the spectrum magnitude.
  • Spectral Flux is the 2-norm of the difference between the magnitude of the short time Fourier transform (STFT) spectrum evaluated at two successive analysis windows. Note that the signal is first normalized for energy, i.e. all segments on which the STFT is calculated are constrained to have equal energy (the same dc value of the STFT).
  • Peak ratio is the ratio of the magnitude of the highest peak in the magnitude spectrum to the average magnitude.
  • Subband energy vector is a vector of sub-band energies calculated by grouping STFT frequency bins to logarithmically spaced sub-bands.
  • the STFT is normalized in energy.
  • Subband flux is the 2-norm of the difference between the subband energy vectors evaluated at two successively analyzed windows.
  • Subband Energy Ratios are formed by analyzing the energy component of the signal in each of a defined set of frequency bands. The ratios of the energy in these sub-bands provide a set of measures of the spectral composition of the signal. For example, if energies in 5 sub-bands are determined then there are 10 (5 choose 2) distinct ratios between the 5 sub-bands. These 10 numbers define a vector, which characterizes the spectral energy distribution. A related set of statistics that may be used is the logarithmic ratios of the sub-band energies. The logarithmic ratios make the numbers more stable and robust.
  • Feature extraction in step S 901 provides a time series of feature vectors.
  • Vector Quantization is a technique that maps a large set of vectors to a smaller representative indexed set of vectors (code words) designated as the code-book. VQ is used to compress vector information. VQ reduces multidimensional vectors comprising of the computed features onto a single value.
  • FIG. 10A is a flow diagram showing computer executable process steps to perform VQ, according to one aspect of the present invention.
  • step S 101 identify a set of representative points.
  • FIG. 10B shows a distribution of a set of points.
  • Each point may include a plurality of features defined above.
  • the representative points are based upon music samples across many genres of music.
  • the representative points specify location in a multidimensional feature space.
  • a c-means clustering algorithm is applied to the data set.
  • the c-means clustering algorithm determines a set of cluster centers (A 1 , A 2 , A 3 and A 4 in FIG. 10B ). 20 and 40 cluster centers may be used for a 5-10 dimensional feature space.
  • the invention is not limited to any particular numbers of cluster centers.
  • the iterative c-means algorithm based upon relative Mahalonobis distance determines the cluster centers. Details of the foregoing techniques are provided in “Multivariate Observations” by G. A. F. Seber, published by John Wiley & Sons (1984), incorporated herein by reference in its entirety.
  • step S 102 the process defines a code-book and each of the representative points derived in step S 101 above is mapped to an element in the codebook, where each element corresponds to a unique ASCII character.
  • step S 103 the process defines a rule that can map any point in a feature space onto an element of the codebook.
  • Cluster centers are assigned to the elements of the codebook. Points that are not cluster centers are mapped to a cluster center, and then assigned to an element of the codebook corresponding to that cluster center.
  • Mahalonobis distance metric and a nearest neighbor algorithm is used to map an arbitrary point onto a cluster center. Every point is mapped onto the closest cluster center, using a Mahalonobis distance or a similar metric.
  • the compressed string provides compact audio signatures that can be efficiently transmitted and compared. Also, logarithmic run length compression highlights the regions when the signal is changing.
  • the process described above depends upon STFT which in turn depends on the bin or window size of the signal.
  • the raw audio signal is partitioned into bins of a certain size (for example bins size 512 of sample points).
  • bins should start and end affects the accuracy of the computed audio signature. For example, if a raw signal has 10,000 points and the signal is divided into bins of length 512 , the computation could start at the beginning of every 512 points or the computation could start at the second, or tenth, or hundredth point etc. Since not every audio sample starts at the “beginning” of some canonical version of the exact choice of bins or windows the signature extraction procedure should be robust to average out the arbitrariness in the computation.
  • the present solution provides a solution to the arbitrariness of bin selection by choosing features that are intrinsically robust to shifts in binning
  • multiple signatures may be computed for each audio sample based on different bin shifts. For example, one signature is computed based on starting the first bin at the beginning of the audio file. A second signature is then computed based on bins starting at the Nth data point. A third signature might be computed based on the first bin starting at the N+Kth data point and so on. Plural number of shifted audio signatures is computed for each audio sample.
  • Each audio signature is a short list of strings (typically of length 8 or 16 ) where each string corresponds to a slightly shifted bin analysis.
  • Audio signatures determined by the foregoing process allow signature comparison of new audio samples (query signature) with stored pre-computed audio signatures (base signature)( FIG. 7 , step S 704 ).
  • query signature is divided into overlapping sub-strings.
  • the sub-strings are then compared to the base signature.
  • the following example illustrates the matching process.
  • a query signature is denoted as AABBCADAABC . . . .
  • the above signature is broken into sub-strings of defined length (e.g. 11 characters).
  • the sub-string of 11 characters is then compared to the stored base signature stored in central database 401 .
  • Hierarchical matching may also be used to obtain the best match.
  • a sub-string of X characters is used and thereafter, a subset of the sub-string comprised of X′ characters where X′ is less than X is used to match a query signature with the base algorithm.
  • a specific confidence level or value may be assigned to a match for each sub-string.
  • a match index value may be used to enhance the accuracy of matching where match index value is given by:
  • W i is a weight (real number) for matches of length i and N i is the number of substring matches of length i.
  • Match Index is a weighted sum of the number of substring matches of different length substrings. Matches for longer substrings are less likely by chance and should thus be assigned a higher weight than matches of shorter length.
  • the magnitude of the weight W i is a measure of the confidence assigned to matches of a given substring length.
  • the foregoing process can be iterated to improve performance. For example, after a particular code-book is defined to match a small set of characters out of a large set, the process may define a second code-book for the subset of songs matched with the first codebook to expedite matching. Alternatively, many independent codebooks may be defined and used in conjunction. The best matching songs will match across the most codebooks. The advantage of this approach is that errors in matching in one codebook will be compensated for by matches in another codebook.
  • Another method for determining audio signatures involves the following: 1) an audio sample is segmented into disjoint, contiguous regions in a robust and deterministic manner, 2) a robust description of the audio characteristics of each segment is determined. This description may be any parameterization of the audio file, or may consist of nonparametric samples, and 3) an audio signature is developed which reflects the segmentation structure of the original audio sample.
  • One method to determine audio for segmenting audio files is described in an article “Multifeature Audio Segmentation for Browsing and Annotation”, by George Tzanetakis, one of the present inventors, and Perry Cook (“Tzanetakis and Cook”), published Oct. 17-20, 1999, in “IEEE Workshop on Applications of Signal Processing to Audio and Acoustics”, incorporated herein by reference in its entirety:
  • FIG. 10C is a flow diagram of process steps to implement the Segmentation techniques.
  • step S 1001 an input audio file's is plotted over time.
  • FIG. 6A shows an audio sample where audio intensity (I A ) is plotted against time.
  • step S 1002 the process determines audio sample feature vectors or parameters.
  • a set of parameters for I A is computed for a set of short contiguous bins (or short overlapping bins).
  • I A is transformed into a set of parametric descriptions (over time), designated for the purpose of illustration as Pa 1 , Pa 2 , . . . , PaN, for N parameterizations.
  • Pa1, Pa2, . . . , PaN is a time series of feature vectors that describe the audio file intensity I A . Examples of such parameters are described above under vector quantization methodology and also include the following:
  • step S 1003 the process determines a univariate signal based upon the foregoing audio file parameters.
  • Pa1, Pa2, . . . , PaN of I A are combined into a single, univariate signal d t .
  • One method to compute d t is to compute the distance between successive time frames of the feature vector of parameterizations.
  • Various distance metrics may be used as discussed in Tzanetakis and Cook. Mahalonobis distance is one such metric and is used between successive values of feature vectors Pa1, Pa2, . . . , PaN.
  • Other distance metrics, such as Minkowski metrics, may also be used.
  • V 1 [Pa1 1 Pa2 1 . . . PaN 1 ], and (4)
  • V 2 [Pa1 2 Pa2 2 . . . PaN 2 ]. (5)
  • ⁇ ⁇ indicates computation of the Mahalonobis distance between V 1 and V 2 .
  • ⁇ t i is determined by ⁇ V t ⁇ V t-1 ⁇ for all adjacent bins.
  • step S 1004 After determining the univariate distance signal ⁇ t, in step S 1004 , the process segments the audio sample. Regions of high change in the signal define segmentation. A derivative,
  • Points at which the derivative value is greater than the threshold value indicate sudden transitions in the audio signal and define segmentation points.
  • step S 1005 the audio signature for a sample audio file is determined, as illustrated by the following example.
  • segment points are determined at t 1 , t 2 , . . . , t N using the procedure described above.
  • L k t k+1 ⁇ t k and t k is the time point (in the sample) of the kth segment.
  • L k t k+1 ⁇ t k
  • t k is the time point (in the sample) of the kth segment.
  • a set of robust statistics are computed. Numerous statistical values may be used to define and describe specific segments for example, spectral centroid, spectral rolloff; intensity variance, skewness, kurtosis, and sub-band energy ratios etc.
  • an audio signature, (“AS”) may be determined as follows:
  • AS is a vector or string which concatenates the segment length and robust summary for each of the segments.
  • AS is definable for any audio sample of sufficient length to contain at least 1 segment, and if two different audio samples contain overlapping audio segments, then the audio signatures contain overlapping elements.
  • the audio file is first segmented into fixed-length segments, and then robust statistics are calculated within each segment.
  • One advantage of the foregoing aspects of the present invention is that unique audio signatures may be assigned to audio files. Also various attributes may be tagged to audio files.
  • the present invention can generate a customized playlist for a user based upon audio file content and the attached attributes, hence making the music listening experiences easy and customized.

Abstract

A fingerprint is generated from an unknown audio signal by dividing the unknown audio signal into bins, where each bin includes points representing a feature space. Each of the points is mapped to one of a plurality of predetermined cluster centers based on the distance between each point and the plurality of cluster centers, each cluster center being associated with an element of a codebook. A string of elements is generated based on the mapping and compressed.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • The present application is a continuation of U.S. application Ser. No. 11/839,768, filed on Aug. 16, 2007, now U.S. Pat. No. ______, entitled “METHOD AND SYSTEM FOR ANALYZING DIGITAL AUDIO FILES”, issued on ______, which is a continuation of U.S. patent application Ser. No. 09/695,457, filed on Oct. 24, 2000, now U.S. Pat. No. 7,277,766, entitled “METHOD AND SYSTEM FOR ANALYZING DIGITAL AUDIO FILES”, issued on Oct. 2, 2007, both of which are incorporated herein by reference in their entirety.
  • BACKGROUND OF THE INVENTION
  • 1. Field of the Invention
  • The present invention relates to analyzing audio files and more particularly to presenting a playlist based upon listener preferences and audio file content.
  • 2. Background
  • The Internet connects thousands of computers world wide into a vast network using well-known protocols, for example, Transmission Control Protocol (TCP)/Internet Protocol (IP). Information on the Internet is stored world wide as computer files, mostly written in the Hypertext Mark Up Language (“HTML”). The collection of all such publicly available computer files is known as the World Wide Web (WWW).
  • The WWW is a multimedia-enabled hypertext system used for navigating the Internet and is made up of hundreds of thousands of web pages with audio, images, text and video files. Each web page can have connections to other pages, which may be located on any computer connected to the Internet.
  • A typical Internet user uses a client program called a “Web Browser” to connect to the Internet. A user can connect to the Internet via a proprietary network, such as America Online or CompuServe, or via an Internet Service Provider, e.g., Earthlink.
  • A Web Browser may run on any computer connected to the Internet. Currently, various browsers are available of which two prominent browsers are Netscape Navigator and Microsoft Internet Explorer. The Web Browser receives and sends requests to a web server and acquires information from the WWW. A web server is a program that, upon receipt of a request, sends the requested data to the requesting user.
  • A standard naming convention known as Uniform Resource Locator (“URL”) has been adopted to represent hypermedia links and links to network services. Most files or services can be represented with a URL. URLs enable Web Browsers to go directly to any file held on any WWW server.
  • Information from the WWW is accessed using well-known protocols, including the Hypertext Transport Protocol (“HTTP”), the Wide Area Information Service (“WAIS”) and the File Transport Protocol (“FTP”), over TCP/IP protocol. The transfer format for standard WWW pages is Hypertext Transfer Protocol (HTTP).
  • The advent and progress of the Internet has changed the way consumers buy or listen to music. Consumers today can download digital music via the Internet using MP3 or SDMI technology, with a click of a mouse. Audio delivery techniques have also made it easy to stream audio from a website to a consumer, upon demand. A typical music listener can download audio files from the WWW, store the audio files, and listen to music.
  • Currently music can be stored in various file formats. Generally there are two types of file formats: (1) self-describing formats, where device parameters and encoding are made explicit in a header, and (2) headerless formats, where device parameters and encoding are fixed.
  • The header of self-describing formats contain parameters of a sampling device and may also include other information (e.g. a human-readable description of sound, or a copyright notice etc.). Some examples of popular self describing formats are provided below:
  • File Extension Variable Parameters (Fixed; Comments)
  • au or .snd rate, #channels, encoding, info string
    aif(f), AIFF rate, #channels, sample width, lots of info
    aif(f), AIFC same (extension of AIFF with compression)
    iff, IFF/8SVX rate, #channels, instrument info (8 bits)
    mp2, .mp3 rate, #channels, sample quality
    .ra rate, #channels, sample quality
    .sf rate, #channels, encoding, info
    .smp loops, cues, (16 bits/1 ch)
    .voc rate (8 bits/1 ch; can use silence deletion)
    .wav, WAVE rate, #channels, sample width, lots of info
  • Headerless formats define single encoding and usually allow no variation in device parameters (except sometimes for sampling rates). The following are a few examples of Headerless formats:
  • Extension Parameters or name
    .snd, .fssd Variable rate, 1 channel, 8 bits unsigned
    .ul
    8 k, 1 channel, 8 bit “u-law” encoding
    .snd Variable rate, 1 channel, 8 bits signed
  • Although music listeners can store audio files, conventional music search techniques do not allow a music listener to search for music based upon audio file content. Conventional systems also do not allow a music listener to generate play lists based upon music listener preferences and/or audio file content.
  • Hence what is needed is a method and system that can analyze audio file content and produce a play list based upon preferences defined by a music listener.
  • SUMMARY
  • The present invention solves the foregoing drawbacks by providing a method and system for analyzing audio files. Plural audio file feature vector values based on an audio file's content are determined and the audio file feature vectors are stored in a database, that also stores other pre-computed audio file features. The process determines if the audio file's feature vectors match the stored audio file vectors. The process also associates a plurality of known attributes to the audio file.
  • The present invention includes a system for analyzing audio files that includes a playlist generator that determines a plurality of audio file vectors based upon an audio file's content; and a signature comparator between the playlist generator and a database, wherein the database stores a plurality of audio file vector values of plural music samples. The signature comparator compares input audio samples with previously stored audio samples in the database. Also provided is a user interface that allows a music listener to input search request for searching music based upon attributes that define music content.
  • In another aspect, the present invention includes a method for determining audio signatures for input audio samples. The process extracts plural features representing the input audio samples, wherein the features are extracted by Fourier transform analysis. The process also identifies a set of representative points based upon the plural features, and determines a codebook of plural elements for mapping the representative points to the elements of the codebook.
  • In yet another aspect, the present invention includes a method for comparing input audio signatures with pre-computed stored audio signatures. The process determines a query signature based upon the input audio signature and divides the query signature into a string of characters; and compares the string of characters to stored pre-computed audio signatures.
  • In yet another aspect, the present invention divides an input audio sample into bins and determines a plurality of features describing the bins. Thereafter, the process determines a univariate signal based upon the plural features and computes an audio signature based upon the univariate signal.
  • One advantage of the foregoing aspects of the present invention is that unique audio signatures may be assigned to audio files. Also various attributes may be tagged to audio files. The present invention can generate a customized playlist for a user based upon audio file content and the attached attributes, hence making the music searching experience easy and customized.
  • This brief summary has been provided so that the nature of the invention may be understood quickly. A more complete understanding of the invention can be obtained by reference to the following detailed description of the preferred embodiments thereof in connection with the attached drawings.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a computing system to carry out the inventive technique.
  • FIG. 2 is a block diagram of the architecture of the computing system of FIG. 1.
  • FIG. 3 is a block diagram of the Internet Topology.
  • FIG. 4 is a block diagram of the architecture of the present system.
  • FIG. 5 is a block diagram showing the architecture of a playlist generator.
  • FIG. 6 is a flow diagram of computer executable process steps for analyzing an audio file.
  • FIG. 6A is a graphical illustration of an audio file's content.
  • FIG. 7 is a flow diagram of computer executable process steps for comparing input audio files with stored audio data.
  • FIG. 8 is a flow diagram of computer executable process steps of generating a playlist, according to the present invention.
  • FIG. 9 is a flow diagram of computer executable process steps for determining audio signatures based upon one aspect of the present invention.
  • FIG. 10A shows a set of representative points randomly scattered and used for vector quantization, according to another aspect of the present invention.
  • FIG. 10B is a flow diagram of computer executable process steps for performing vector quantization, according to another aspect of the present invention.
  • FIG. 10C is a flow diagram of computer executable process steps for determining audio signatures, according to yet another aspect of the present invention.
  • The use of similar reference numerals in different figures indicates similar or identical items.
  • DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • FIG. 1 is a block diagram of a computing system for executing computer executable process steps according to one embodiment of the present invention. FIG. 1 includes a host computer 10 and a monitor 11. Monitor 11 may be a CRT type, a LCD type, or any other type of color or monochrome display. Also provided with computer 10 is a keyboard 13 for entering text data and user commands, and a pointing device 14 for processing objects displayed on monitor 11.
  • Computer 10 includes a computer-readable memory medium such as a rotating disk 15 for storing readable data. Besides other programs, disk 15 can store application programs including web browsers by which computer 10 connects to the Internet and the systems according to the present invention as described below.
  • Computer 10 can also access a computer-readable floppy disk storing data files, application program files, and computer executable process steps embodying the present invention or the like via a floppy disk drive 16. A CD-ROM interface (not shown) may also be provided with computer 10 to access application program files, audio files and data files stored on a CD-ROM.
  • A modem, an integrated services digital network (ISDN) connection, or the like also provides computer 10 with an Internet connection 12 to the World Wide Web (WWW). The Internet connection 12 allows computer 10 to download data files, audio files, application program files and computer-executable process steps embodying the present invention.
  • Computer 10 is also provided with external audio speakers 17A and 17B to assist a listener to listen to music either on-line, downloaded from the Internet or off-line using a CD (not shown). It is noteworthy that a listener may use headphones instead of audio speakers 17A and 17B to listen to music.
  • FIG. 2 is a block diagram showing the internal functional architecture of computer 10. Computer 10 includes a CPU 201 for executing computer-executable process steps and interfaces with a computer bus 208. Also shown in FIG. 2 are a WWW interface 202, a display device interface 203, a keyboard interface 204, a pointing device interface 205, an audio interface 209, and a rotating disk 15. Audio Interface 209 allows a listener to listen to music, On-line (downloaded using the Internet or a private network) or off-line (using a CD, not shown).
  • As described above, disk 15 stores operating system program files, application program files, web browsers, and other files. Some of these files are stored on disk 15 using an installation program. For example, CPU 201 executes computer-executable process steps of an installation program so that CPU 201 can properly execute application programs.
  • A random access main memory (“RAM”) 206 also interfaces to computer bus 208 to provide CPU 201 with access to memory storage. When executing stored computer-executable process steps from disk 15 (or other storage media such as floppy disk 16 or WWW connection 12), CPU 201 stores and executes the process steps out of RAM 206.
  • Read only memory (“ROM”) 207 is provided to store invariant instruction sequences such as start-up instruction sequences or basic input/output operating system (BIOS) sequences for operation of keyboard 13.
  • The present invention is not limited to the computer architecture described above. Systems comparable to Computer 10, for example, portable devices or hand held computing devices that can be connected to the Internet may also be used to implement the present inventive techniques.
  • FIG. 3 shows a typical topology of a computer network with computers similar to computer 10, connected to the Internet. For illustration purposes, three computers X, Y and Z are shown connected to the Internet 302 via Web interface 202, through gateway 301, where gateway 301 can interface numerous computers. Web interface 202 may be a modem, network interface card or a unit for providing connectivity to other computer systems over a network using protocols such as X.25, Ethernet or TCP/IP, or to any device that allows direct or indirect computer-to-computer communications.
  • It is noteworthy that the invention is not limited to a particular number of computers. Any number of computers that can be connected to the Internet 302 or to any other computer network may be used to implement the present inventive techniques.
  • FIG. 3 further also shows a second gateway 303 that connects a network of web servers 304 and 305 to the Internet 302. Web servers 304 and 305 may be connected with each other over a computer network. Web servers 304 and 305 can provide content including music samples and audio clips to a user from database 306 and/or 307. Web servers 304 and 305 can also host the system according to the present invention. Also shown in FIG. 3 is a client side web server 308 that can be provided by an Internet service provider.
  • FIG. 4 shows a block diagram of a system used for analyzing audio files, according to the present invention. Audio files may be stored on rotating disk 15 in a music listener's computer 10, or at any remote computer 10 connected to the Internet 302.
  • A playlist generator 400 accesses audio files stored on rotating disk 15. Playlist generator 400 is an application program that can be located on remote server 304 or on a music listener's computer 10. Playlist generator 400 scans and analyzes audio files stored on rotating disk 15 and computes audio signatures that uniquely and compactly identify the content of the audio file. An audio signature is computed only once for each file and stored in a local database. The computed audio signature that compares the analyzed audio files with previously analyzed audio data is stored in a central database 401. Central database 401 stores a plurality of feature vector values as discussed below. Central database 401 also includes data similar to the data stored in a production database that is described in U.S. patent application Ser. No. 09/533,045, entitled, “Method for Creating a Database for Comparing Music Attributes”, incorporated herein by reference in its entirety. Customized play lists are generated after audio file content is compared to the data stored in central database 401, as described below.
  • A user interface 400A is also provided that allows a music listener to input preferences for generating play lists. User interface 400A may be a separate component or integrated with playlist generator 400. One such user interface is described in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, incorporated herein by reference in its entirety.
  • FIG. 5 is a block diagram showing various components of playlist generator 400. Playlist generator 400 includes an audio file analyzer 400B and a Signature Comparator 400C. Audio file analyzer 400B receives and analyzes audio files as described below. Audio files may be stored on rotating disk 15 or may be acquired from a remote computer.
  • The Signature Comparator 400C receives user requests from UI 400A and obtains music or a list of music based upon user preferences, as described below. It is noteworthy that audio file analyzer 400B and signature comparator 400C may be integrated into a single module to determine audio signature and implement the various aspects of the present invention.
  • FIG. 6 is a process flow diagram of computer executable process steps for analyzing audio files according to the present invention.
  • In step S601 audio analyzer 400B receives audio files (“input audio files”). Audio files may be stored on a user's disk 15 or at webserver 304 connected to the Internet or on any private network. Audio analyzer 400B may seek audio files based upon user input in UI 400A or may receive audio files from a designated source.
  • In step S602, audio analyzer 400B analyzes input audio file content and computes a set of parameters (“audio file vectors”). Audio file vectors can uniquely identify audio files and can also be used to assign a unique “audio signature” for an input audio file. Details of determining audio signatures are described below. These audio signatures need be computed only once for a given audio file and can thereafter be stored either locally or remotely and referred to as necessary.
  • In step S603, input audio file vector values for specific bins and the audio signature are stored in a database. One such database is a central database 401. Audio file signatures are sets or vectors of values based upon the audio segments as shown in FIG. 6A.
  • FIG. 7 is a flowchart of computer executable process steps that allows playlist generator 400 to compare audio file content with stored and analyzed audio file data. Such audio files may be stored on rotating disk 15 or on a remote computer.
  • In step S701, playlist generator 400 scans an audio file(s) stored on disk 15. Audio files may be pulled by playlist generator 400 from disk 15, or a music listener may send audio files to playlist generator 400 via the Internet 302 or a private network.
  • In step S702, playlist generator 400 determines the audio signature of input audio file(s). An audio signature may be determined by the process described below.
  • In step S703, playlist generator 400 transfers the audio signature and audio file vector values (Vt1 . . . Vtk) to Signature Comparator 400C. Playlist generator 400 also commands the Signature Comparator 400C to compare the audio signature and feature vectors values determined in step S702 with historical audio signatures and audio file vector values stored in central database 401. Thereafter, the Signature Comparator 400C compares input audio signature and audio file vector values with historical audio signatures and audio file vector values, stored in database 401.
  • In step S704, Signature Comparator 400C determines whether the input audio signature and audio file vectors values match stored audio signatures and vector values. If the input audio file vector values do not match with any stored audio data (audio signature and feature vector), then in step S705 input audio file vector values are stored in central database 401.
  • If the input audio file signature matches with stored audio signatures and vector values, then in step S706, the process determines if the stored entries are confirmed or provisional. A confirmed entry in database 401 is an entry that has been ratified by multiple sources.
  • A provisional entry in database 401 is one that is not confirmed.
  • If the input audio file matches a confirmed entry, then in step S707 other feature values stored in database 401 are associated with the audio file. Examples of such feature vectors are provided in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, filed on Mar. 22, 2000, assigned to the assignee herein, and incorporated by reference in its entirety. Associating a plurality of feature vectors allows a listener to search for music based upon content. For example, feature values associated with a given audio file may indicate that it is Rock music with a slow tempo and a smooth, female singer accompanied by a band featuring a prominent saxophone.
  • Some of the features that can be associated with the audio files are:
  • (a) Emotional quality vector values that indicate whether an audio file content is Intense, Happy, Sad Mellow, Romantic, Heartbreaking, Aggressive or Upbeat.
    (b) Vocal vector values that indicates whether the audio file content includes a Sexy voice, a Smooth voice, a Powerful voice, a Great voice, or a Soulful voice.
    (c) Sound quality vector values that indicate whether the audio file content includes a strong beat, is simple, has a good groove, is fast, is speech like or emphasizes a melody.
    (d) Situational quality vector values that indicate whether the audio file content is good for a workout, a shopping mall, a dinner party, a dance party, slow dancing or studying.
    (e) Ensemble vector values indicating whether the audio file includes a female solo, male solo, female duet, male duet, mined duet, female group, male group or instrumental.
    (f) Genre vector values that indicate whether the audio file content belongs to a plurality of genres including Alternative, Blues, County, Electronics/Dance, Folk, Gospel, Jazz, Latin, New Age, Rhythm and Blues (R and B), Soul, Rap, Hip-Hop, Reggae, Rock and others.
    (g) Instrument vectors that indicate whether the audio file content includes an acoustic guitar, electric guitar, bass, drum, harmonica, organ, piano, synthesizer, horn or saxophone.
  • If the input audio file matches a provisional rating, then in step S708, the process converts the provisional rating to a confirmed rating.
  • FIG. 8 is a flow diagram of computer executable process steps for generating a customized music list (“playlist”) based upon user defined parameters.
  • In step S801, a music listener inputs a request for a playlist. UI 400A (FIG. 4) may be used by a music listener to input user preferences. UI 400A is described in U.S. patent application Ser. No. 09/533,045, entitled “Method for Creating a Database for Comparing Music Attributes”, filed on Mar. 22, 2000, and incorporated herein by reference. A user may request a playlist by specifying emotional quality, voice quality, instrument or genre vector, tempo, artist, album title and year of release, etc.
  • In step S802, playlist generator 400 scans audio files stored on user's disk 15. One methodology of analyzing audio files is provided above in FIG. 7. Audio files may also be acquired form a remote computer connected to the Internet and analyzed as shown above.
  • In step S803, Signature Comparator 400C searches for music based upon analyzed audio file data.
  • In step S804, playlist generator 400 compiles a playlist based upon user preferences and the compared data.
  • Determining Audio Signatures
  • Audio signatures identify audio files and are based upon the signal characteristics of a particular audio file sample. If the audio signatures of a group of audio files is known then a new audio file may be compared with the known group of audio signatures. Audio signature is a representation of an audio file that assists in comparing audio samples. Audio signature may be developed for any audio file and whenever two audio samples overlap, the audio signatures of the samples will also overlap. It is this property that assists comparison of audio samples.
  • FIG. 9 is a flow diagram showing process steps for determining audio signatures according to one aspect of the present invention.
  • In step S901, an audio sample is represented as a signal. The signal is then used to determine a set of parameters or features that describe the signal. FIG. 6A shows an example of representing an audio file over time.
  • In step S902, the determined features from step S901 are used to compute an audio signature for the audio sample.
  • In step S903, the audio signature determined in step S902 is compared to pre-computed audio signatures stored in database 401.
  • The following describes different techniques under the present invention that may be used to implement the process steps of FIG. 9.
  • Vector Quantization Methodology
  • Under this methodology, an audio file is transformed into a raw intensity signal and the transformed signal is used to compute a set of features or parameters. The features are computed from sequential (possibly overlapping) sets of raw intensity signal and transforms the intensity signal into a time series of feature values. This multivariate time series of feature vectors is then compressed into a 1-dimensional vector of elements. This compression is achieved using vector quantization of the multivariate features, so that each point in the feature space can be mapped onto an element in a finite codebook. Thereafter, the computed 1-dimensional vector (1-d string) is matched against known values stored in a database using fast string matching algorithms to retrieve a match.
  • Pre-Processing
  • As discussed in step S901, pre-processing is used to convert an audio file whose signature is to be calculated, to a standard reference format. More specifically mono audio files of 16-bit samples at 22050 Hz sampling rate are used as a reference format. Standard commercially available software converters and decoders may be used for this purpose. More specifically Sox sound converting utility that can be downloaded from the Internet address www.spies.com/Sox/ and Xaudio mp3 decoder downloaded from a website located at www.xaudio.com may be used.
  • Feature Extraction
  • As discussed in step S901, certain features or parameters are extracted from an audio file signal. The features of this methodology are based on Short Time Fourier Transform (STFT) analysis. The STFT is a signal processing analysis technique that processes audio file signals in samples designated as bins or windows (FIG. 6A) and then a Discrete Fourier Transform (DFT) is determined for the bins or windows. This technique is disclosed in A. V. Oppenheim and A. S. Wilsky, “Signals and Systems” (2nd Ed.) Prentice-Hall (1997), which is herein incorporated by reference in its entirety for all purposes. The signal consists of a time series of values. The time series is divided into smaller units or bins (that may be overlapping). The STFT and related features are computed on each of these bins. Thus the original signal time series is transformed into a vector time series of features.
  • It is noteworthy that because overall level (volume) effects vary across different encodings of the same audio file and can not be assumed to be uniform, STFT features are normalized to eliminate level (volume) effects by dividing out the dc component of the STFT at each sample point before further processing.
  • The following STFT-based features may be extracted in step S901:
  • Spectral Centroid is the balancing point of the spectrum (The spectrum is the representation of audio intensity over time. (FIG. 6A)) magnitude and is defined as:
  • C = i * M i M i ( 1 ) where i = 1 , 2 , , N
  • where N is the FFT window size and the Mi's are the frequency bins of the spectrum magnitude.
  • Spectral Rolloff like the spectral centroid is another measure of the shape of the spectrum. It is defined as:

  • R=r such that Σ1 r M i=0.8*Σr N M i,  (2)
  • where N is the FFT window size, the Mi's are the frequency bins of the spectrum magnitude.
  • Spectral Flux is the 2-norm of the difference between the magnitude of the short time Fourier transform (STFT) spectrum evaluated at two successive analysis windows. Note that the signal is first normalized for energy, i.e. all segments on which the STFT is calculated are constrained to have equal energy (the same dc value of the STFT).
  • Peak ratio is the ratio of the magnitude of the highest peak in the magnitude spectrum to the average magnitude.
  • Subband energy vector is a vector of sub-band energies calculated by grouping STFT frequency bins to logarithmically spaced sub-bands. The STFT is normalized in energy.
  • Subband flux is the 2-norm of the difference between the subband energy vectors evaluated at two successively analyzed windows.
  • Subband Energy Ratios are formed by analyzing the energy component of the signal in each of a defined set of frequency bands. The ratios of the energy in these sub-bands provide a set of measures of the spectral composition of the signal. For example, if energies in 5 sub-bands are determined then there are 10 (5 choose 2) distinct ratios between the 5 sub-bands. These 10 numbers define a vector, which characterizes the spectral energy distribution. A related set of statistics that may be used is the logarithmic ratios of the sub-band energies. The logarithmic ratios make the numbers more stable and robust.
  • Signature Calculation (Step S902, FIG. 9) Vector Quantization
  • Feature extraction in step S901 provides a time series of feature vectors. Vector Quantization (VQ) is a technique that maps a large set of vectors to a smaller representative indexed set of vectors (code words) designated as the code-book. VQ is used to compress vector information. VQ reduces multidimensional vectors comprising of the computed features onto a single value. FIG. 10A is a flow diagram showing computer executable process steps to perform VQ, according to one aspect of the present invention.
  • In step S101, identify a set of representative points. FIG. 10B shows a distribution of a set of points. Each point may include a plurality of features defined above. Generally, the representative points are based upon music samples across many genres of music. The representative points specify location in a multidimensional feature space. After identifying a set of representative points, a c-means clustering algorithm is applied to the data set. The c-means clustering algorithm determines a set of cluster centers (A1, A2, A3 and A4 in FIG. 10B). 20 and 40 cluster centers may be used for a 5-10 dimensional feature space. The invention is not limited to any particular numbers of cluster centers. The iterative c-means algorithm based upon relative Mahalonobis distance determines the cluster centers. Details of the foregoing techniques are provided in “Multivariate Observations” by G. A. F. Seber, published by John Wiley & Sons (1984), incorporated herein by reference in its entirety.
  • In step S102, the process defines a code-book and each of the representative points derived in step S101 above is mapped to an element in the codebook, where each element corresponds to a unique ASCII character.
  • In step S103, the process defines a rule that can map any point in a feature space onto an element of the codebook. Cluster centers are assigned to the elements of the codebook. Points that are not cluster centers are mapped to a cluster center, and then assigned to an element of the codebook corresponding to that cluster center. To map an arbitrary point onto a cluster center Mahalonobis distance metric and a nearest neighbor algorithm is used. Every point is mapped onto the closest cluster center, using a Mahalonobis distance or a similar metric.
  • VQ provides a string of characters. These characters may be compressed by using a logarithmic run length compression scheme. In this scheme a run of the same character is compressed to a shorter run with length equal to the logarithm of the original run length. For example, the string of characters aaaabcccccccc will be compressed to aabccc (Log28=3 so the 8 c's are compressed to 3). The compressed string provides compact audio signatures that can be efficiently transmitted and compared. Also, logarithmic run length compression highlights the regions when the signal is changing.
  • Base Overshifting
  • The process described above depends upon STFT which in turn depends on the bin or window size of the signal. In particular, to compute STFT efficiently, the raw audio signal is partitioned into bins of a certain size (for example bins size 512 of sample points). However, where the bins should start and end affects the accuracy of the computed audio signature. For example, if a raw signal has 10,000 points and the signal is divided into bins of length 512, the computation could start at the beginning of every 512 points or the computation could start at the second, or tenth, or hundredth point etc. Since not every audio sample starts at the “beginning” of some canonical version of the exact choice of bins or windows the signature extraction procedure should be robust to average out the arbitrariness in the computation.
  • The present solution provides a solution to the arbitrariness of bin selection by choosing features that are intrinsically robust to shifts in binning Also, multiple signatures may be computed for each audio sample based on different bin shifts. For example, one signature is computed based on starting the first bin at the beginning of the audio file. A second signature is then computed based on bins starting at the Nth data point. A third signature might be computed based on the first bin starting at the N+Kth data point and so on. Plural number of shifted audio signatures is computed for each audio sample. Each audio signature is a short list of strings (typically of length 8 or 16) where each string corresponds to a slightly shifted bin analysis.
  • Audio signatures determined by the foregoing process allow signature comparison of new audio samples (query signature) with stored pre-computed audio signatures (base signature)(FIG. 7, step S704). In order to compare a query signature with a base signature, the query signature is divided into overlapping sub-strings. The sub-strings are then compared to the base signature. The following example illustrates the matching process.
  • Assume that a query signature is denoted as AABBCADAABC . . . .
  • The above signature is broken into sub-strings of defined length (e.g. 11 characters). The sub-string of 11 characters is then compared to the stored base signature stored in central database 401.
  • Hierarchical matching may also be used to obtain the best match. In this process a sub-string of X characters is used and thereafter, a subset of the sub-string comprised of X′ characters where X′ is less than X is used to match a query signature with the base algorithm. A specific confidence level or value may be assigned to a match for each sub-string. A match index value may be used to enhance the accuracy of matching where match index value is given by:

  • MI=ΣW i *N i,  (3)
  • where Wi is a weight (real number) for matches of length i and Ni is the number of substring matches of length i. Thus, Match Index (MI) is a weighted sum of the number of substring matches of different length substrings. Matches for longer substrings are less likely by chance and should thus be assigned a higher weight than matches of shorter length. Thus, the magnitude of the weight Wi is a measure of the confidence assigned to matches of a given substring length.
  • The foregoing process can be iterated to improve performance. For example, after a particular code-book is defined to match a small set of characters out of a large set, the process may define a second code-book for the subset of songs matched with the first codebook to expedite matching. Alternatively, many independent codebooks may be defined and used in conjunction. The best matching songs will match across the most codebooks. The advantage of this approach is that errors in matching in one codebook will be compensated for by matches in another codebook.
  • The following specific values may be used for the foregoing process:
  • sampling_rate=22050 Hz
    sample size (bits)=16
    fft_size=512
    window_size=512
    code_book_size=14
    overshifting=8
    query substring length=14
    query overlap=13
  • It is noteworthy that the foregoing values are merely to illustrate the foregoing process and is not to limit the invention.
  • Segmentation Technique
  • Another method for determining audio signatures according to the present invention involves the following: 1) an audio sample is segmented into disjoint, contiguous regions in a robust and deterministic manner, 2) a robust description of the audio characteristics of each segment is determined. This description may be any parameterization of the audio file, or may consist of nonparametric samples, and 3) an audio signature is developed which reflects the segmentation structure of the original audio sample. One method to determine audio for segmenting audio files is described in an article “Multifeature Audio Segmentation for Browsing and Annotation”, by George Tzanetakis, one of the present inventors, and Perry Cook (“Tzanetakis and Cook”), published Oct. 17-20, 1999, in “IEEE Workshop on Applications of Signal Processing to Audio and Acoustics”, incorporated herein by reference in its entirety:
  • FIG. 10C is a flow diagram of process steps to implement the Segmentation techniques.
  • In step S1001, an input audio file's is plotted over time. FIG. 6A shows an audio sample where audio intensity (IA) is plotted against time.
  • In step S1002, the process determines audio sample feature vectors or parameters. A set of parameters for IA is computed for a set of short contiguous bins (or short overlapping bins). Hence, IA is transformed into a set of parametric descriptions (over time), designated for the purpose of illustration as Pa1, Pa2, . . . , PaN, for N parameterizations. Pa1, Pa2, . . . , PaN, is a time series of feature vectors that describe the audio file intensity IA. Examples of such parameters are described above under vector quantization methodology and also include the following:
  • (a) Zero Crossings: Number of time domain zero crossings, i.e., the number of times the signal intensity changes from positive to negative or negative to positive in the sample file.
    (b) Root Mean Square (RMS) energy (intensity) within a given bin.
    (c) Nth percentile frequency of the spectrum (e.g., N=25, 50, 75, etc.)
  • The Nth percentile frequency of the spectrum of a specific bin is the largest frequency fk such that Sum(Ai)/Sum(Aj)<=N*100 where i=1, 2, . . . , fk and j=1, 2, . . . , fM, where fM is the highest sample frequency.
  • (h) Spectral energy correlation of contiguous bins (the correlation between bins i and i+N (where N is a positive integer representing the correlation lag computed for the spectral energy across frequencies).
  • The foregoing list of parameters is not exhaustive. Other mathematical variables may also be used to determine audio file vectors.
  • In step S1003, the process determines a univariate signal based upon the foregoing audio file parameters. Pa1, Pa2, . . . , PaN of IA, are combined into a single, univariate signal dt. One method to compute dt is to compute the distance between successive time frames of the feature vector of parameterizations. Various distance metrics may be used as discussed in Tzanetakis and Cook. Mahalonobis distance is one such metric and is used between successive values of feature vectors Pa1, Pa2, . . . , PaN. Other distance metrics, such as Minkowski metrics, may also be used.
  • An example of determining dt is provided below. Assume that t1 and t2 are adjacent bins of IA which have parameterization feature vectors V1 and V2 respectively.
  • Hence,

  • V1=[Pa11Pa21 . . . PaN1], and  (4)

  • V2=[Pa12Pa22 . . . PaN2].  (5)

  • d t =∥V 2 −V 1∥∥,  (6)
  • where ∥ ∥ indicates computation of the Mahalonobis distance between V1 and V2.
  • Therafter Δti is determined by ∥Vt−Vt-1∥ for all adjacent bins.
  • After determining the univariate distance signal Δt, in step S1004, the process segments the audio sample. Regions of high change in the signal define segmentation. A derivative,
  • Δ t t ( 7 )
  • is computed and compared to a predefined threshold value. Points at which the derivative value is greater than the threshold value indicate sudden transitions in the audio signal and define segmentation points.
  • In step S1005, the audio signature for a sample audio file is determined, as illustrated by the following example.
  • For a given audio sample S, segment points are determined at t1, t2, . . . , tN using the procedure described above. A set of N−1 segments may then be constructed, having lengths L1, L2, . . . , LN-1 where Lk=tk+1−tk and tk is the time point (in the sample) of the kth segment. For each audio segment a set of robust statistics are computed. Numerous statistical values may be used to define and describe specific segments for example, spectral centroid, spectral rolloff; intensity variance, skewness, kurtosis, and sub-band energy ratios etc.
  • Hence, having determined a set of audio segments L1, L2, . . . , LM and a set of k robust statistics for each segment R11, R12, . . . , R1k, an audio signature, (“AS”) may be determined as follows:

  • AS=[L1R11R12 . . . R1kL2R21R22 . . . R2k . . . LMRM1RM2 . . . RMk]  (8)
  • AS is a vector or string which concatenates the segment length and robust summary for each of the segments.
  • It is noteworthy that AS is definable for any audio sample of sufficient length to contain at least 1 segment, and if two different audio samples contain overlapping audio segments, then the audio signatures contain overlapping elements.
  • It is noteworthy that instead of segmenting an audio file into variable length segments as described above, the audio file is first segmented into fixed-length segments, and then robust statistics are calculated within each segment.
  • It is also noteworthy that any or all of these techniques described above may be combined in a multistage procedure.
  • One advantage of the foregoing aspects of the present invention is that unique audio signatures may be assigned to audio files. Also various attributes may be tagged to audio files. The present invention can generate a customized playlist for a user based upon audio file content and the attached attributes, hence making the music listening experiences easy and customized.
  • Although the present invention has been described with reference to specific embodiments, these embodiments are illustrative only and not limiting. Many other applications and embodiments of the present invention will be apparent in light of this disclosure and the following claims.

Claims (24)

1. A method of generating a fingerprint from an unknown audio signal, the method comprising:
dividing the unknown audio signal into bins, each bin including a plurality of points representing a feature space;
mapping each of the plurality of points to one of a plurality of predetermined cluster centers based on the distance between each point and the plurality of cluster centers, each cluster center being associated with an element of a codebook;
generating a string of elements based on the mapping; and
compressing the string of elements.
2. The method according to claim 1, further comprising:
performing a Fourier transform on the plurality of bins to generate a frequency representation of each bin; and
computing at least one feature from the frequency representation of each bin.
3. The method according to claim 2, wherein the at least one feature includes at least one of:
(a) a spectral centroid;
(b) a spectral rolloff;
(c) a spectral flux;
(d) a peak ratio;
(e) a subband energy vector;
(f) a subband flux; and
(g) a subband energy ratio.
4. The method according to claim 1, wherein each point representing the feature space specifies a location in a multidimensional feature space.
5. The method according to claim 1, wherein the plurality of cluster centers are determined by using a c-means clustering algorithm.
6. The method according to claim 1, wherein the distance between each point and the plurality of cluster centers is based on a Mahalonobis distance.
7. The method according to claim 1, wherein each element of the codebook corresponds to a unique ASCII character.
8. The method according to claim 1, further comprising:
parsing the compressed set of elements into a plurality of sub-strings of a predefined length of characters.
matching each substring to a database of precomputed signatures to obtain at least one base signature;
assigning a match index value (MI) to each match obtained by the matching given by:

MI=ΣW i *N i,
where Wi is a weight for matches of length i and Ni is a number of substring matches of length i; and
selecting a best match based on the match index value (MI).
9. A non-transitory computer-readable medium storing a computer instructions, which when executed by one or more processors, causes the one or more processors to execute the steps of:
dividing the unknown audio signal into bins, each bin including a plurality of points representing a feature space;
mapping each of the plurality of points to one of a plurality of predetermined cluster centers based on the distance between each point and the plurality of cluster centers, each cluster center being associated with an element of a codebook;
generating a string of elements based on the mapping; and
compressing the string of elements.
10. The non-transitory computer-readable medium according to claim 9, further comprising the steps of:
performing a Fourier transform on the plurality of bins to generate a frequency representation of each bin; and
computing at least one feature from the frequency representation of each bin.
11. The non-transitory computer-readable medium according to claim 10, wherein the at least one feature includes at least one of:
(a) a spectral centroid;
(b) a spectral rolloff;
(c) a spectral flux;
(d) a peak ratio;
(e) a subband energy vector;
(f) a subband flux; and
(g) a subband energy ratio.
12. The non-transitory computer-readable medium according to claim 9, wherein each point representing the feature space specifies a location in a multidimensional feature space.
13. The non-transitory computer-readable medium according to according to claim 9, wherein the plurality of cluster centers are determined by using a c-means clustering algorithm.
14. The non-transitory computer-readable medium according to according to claim 9, wherein the distance between each point and the plurality of cluster centers is based on a Mahalonobis distance.
15. The non-transitory computer-readable medium according to according to claim 9, wherein each element of the codebook corresponds to a unique ASCII character.
16. The non-transitory computer-readable medium according to according to claim 9, further comprising the steps of:
parsing the compressed set of elements into a plurality of sub-strings of a predefined length of characters.
matching each substring to a database of precomputed signatures to obtain at least one base signature;
assigning a match index value (MI) to each match obtained by the matching given by:

MI=ΣW i *N i,
where Wi is a weight for matches of length i and Ni is a number of substring matches of length i; and
selecting a best match based on the match index value (MI).
17. An apparatus for generating a fingerprint from an unknown audio signal, comprising:
at least one processor operable to perform:
dividing the unknown audio signal into bins, each bin including a plurality of points representing a feature space;
mapping each of the plurality of points to one of a plurality of predetermined cluster centers based on the distance between each point and the plurality of cluster centers, each cluster center being associated with an element of a codebook stored in a memory;
generating a string of elements based on the mapping; and
compressing the string of elements.
18. The apparatus according to claim 17, the processor further operable to perform a Fourier transform on the plurality of bins to generate a frequency representation of each bin and compute at least one feature from the frequency representation of each bin.
19. The apparatus according to claim 18, wherein the at least one feature includes at least one of:
(a) a spectral centroid;
(b) a spectral rolloff;
(c) a spectral flux;
(d) a peak ratio;
(e) a subband energy vector;
(f) a subband flux; and
(g) a subband energy ratio.
20. The apparatus according to claim 17, wherein each point representing the feature space specifies a location in a multidimensional feature space.
21. The apparatus according to claim 17, wherein the plurality of cluster centers are determined by using a c-means clustering algorithm.
22. The apparatus according to claim 17, wherein the distance between each point and the plurality of cluster centers is based on a Mahalonobis distance.
23. The method according to claim 17, wherein each element of the codebook corresponds to a unique ASCII character.
24. The apparatus according to claim 17, wherein the at one processor is further operable to perform the following steps:
parsing the compressed set of elements into a plurality of sub-strings of a predefined length of characters.
matching each substring to a database of precomputed signatures to obtain at least one base signature;
assigning a match index value (MI) to each match obtained by the matching given by:

MI=ΣW i *N i,
where Wi is a weight for matches of length i and Ni is a number of substring matches of length i; and
selecting a best match based on the match index value (MI).
US12/906,584 2000-10-24 2010-10-18 Method and system for analyzing digital audio files Abandoned US20110035035A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/906,584 US20110035035A1 (en) 2000-10-24 2010-10-18 Method and system for analyzing digital audio files

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US09/695,457 US7277766B1 (en) 2000-10-24 2000-10-24 Method and system for analyzing digital audio files
US11/839,768 US7853344B2 (en) 2000-10-24 2007-08-16 Method and system for analyzing ditigal audio files
US12/906,584 US20110035035A1 (en) 2000-10-24 2010-10-18 Method and system for analyzing digital audio files

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/839,768 Continuation US7853344B2 (en) 2000-10-24 2007-08-16 Method and system for analyzing ditigal audio files

Publications (1)

Publication Number Publication Date
US20110035035A1 true US20110035035A1 (en) 2011-02-10

Family

ID=38535878

Family Applications (3)

Application Number Title Priority Date Filing Date
US09/695,457 Expired - Lifetime US7277766B1 (en) 2000-10-24 2000-10-24 Method and system for analyzing digital audio files
US11/839,768 Expired - Fee Related US7853344B2 (en) 2000-10-24 2007-08-16 Method and system for analyzing ditigal audio files
US12/906,584 Abandoned US20110035035A1 (en) 2000-10-24 2010-10-18 Method and system for analyzing digital audio files

Family Applications Before (2)

Application Number Title Priority Date Filing Date
US09/695,457 Expired - Lifetime US7277766B1 (en) 2000-10-24 2000-10-24 Method and system for analyzing digital audio files
US11/839,768 Expired - Fee Related US7853344B2 (en) 2000-10-24 2007-08-16 Method and system for analyzing ditigal audio files

Country Status (1)

Country Link
US (3) US7277766B1 (en)

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120254806A1 (en) * 2011-03-30 2012-10-04 Google Inc. System and method for dynamic, feature-based playlist generation
US8385662B1 (en) * 2009-04-30 2013-02-26 Google Inc. Principal component analysis based seed generation for clustering analysis
US8391634B1 (en) 2009-04-28 2013-03-05 Google Inc. Illumination estimation for images
US8396325B1 (en) 2009-04-27 2013-03-12 Google Inc. Image enhancement through discrete patch optimization
US8583656B1 (en) 2009-04-23 2013-11-12 Google Inc. Fast covariance matrix generation
US8611695B1 (en) 2009-04-27 2013-12-17 Google Inc. Large scale patch search
US8732739B2 (en) 2011-07-18 2014-05-20 Viggle Inc. System and method for tracking and rewarding media and entertainment usage including substantially real time rewards
CN103870476A (en) * 2012-12-12 2014-06-18 百度在线网络技术(北京)有限公司 Retrieval method and device
US8798393B2 (en) 2010-12-01 2014-08-05 Google Inc. Removing illumination variation from images
US8938119B1 (en) 2012-05-01 2015-01-20 Google Inc. Facade illumination removal
US9020415B2 (en) 2010-05-04 2015-04-28 Project Oda, Inc. Bonus and experience enhancement system for receivers of broadcast media
WO2013106343A3 (en) * 2012-01-09 2015-05-14 Viggle, Inc. Method and system for identifying a media program from an audio signal associated with the media program
US20170092089A1 (en) * 2015-09-30 2017-03-30 Tianjin Hualai Technology Co., Ltd. Security monitoring apparatus, camera having the same and security monitoring method
US9922118B2 (en) 2015-04-28 2018-03-20 International Business Machines Corporation Creating an audio file sample based upon user preferences
CN110072183A (en) * 2019-03-14 2019-07-30 天津大学 Passive type location fingerprint base construction method based on intelligent perception
WO2022186952A1 (en) * 2021-03-04 2022-09-09 Gracenote Inc. Methods and apparatus to fingerprint an audio signal

Families Citing this family (115)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7444353B1 (en) 2000-01-31 2008-10-28 Chen Alexander C Apparatus for delivering music and information
US20020072982A1 (en) 2000-12-12 2002-06-13 Shazam Entertainment Ltd. Method and system for interacting with a user in an experiential environment
DE10134471C2 (en) * 2001-02-28 2003-05-22 Fraunhofer Ges Forschung Method and device for characterizing a signal and method and device for generating an indexed signal
US7421376B1 (en) * 2001-04-24 2008-09-02 Auditude, Inc. Comparison of data signals using characteristic electronic thumbprints
US7962482B2 (en) * 2001-05-16 2011-06-14 Pandora Media, Inc. Methods and systems for utilizing contextual feedback to generate and modify playlists
DE10133333C1 (en) * 2001-07-10 2002-12-05 Fraunhofer Ges Forschung Producing fingerprint of audio signal involves setting first predefined fingerprint mode from number of modes and computing a fingerprint in accordance with set predefined mode
US7707221B1 (en) * 2002-04-03 2010-04-27 Yahoo! Inc. Associating and linking compact disc metadata
US6987221B2 (en) * 2002-05-30 2006-01-17 Microsoft Corporation Auto playlist generation with multiple seed songs
US7222071B2 (en) * 2002-09-27 2007-05-22 Arbitron Inc. Audio data receipt/exposure measurement with code monitoring and signature extraction
EP1435620A1 (en) * 2003-01-06 2004-07-07 Thomson Licensing S.A. Method for creating and accessing a menu for audio content without using a display
KR20050113614A (en) * 2003-02-26 2005-12-02 코닌클리케 필립스 일렉트로닉스 엔.브이. Handling of digital silence in audio fingerprinting
US7882139B2 (en) 2003-09-29 2011-02-01 Xunlei Networking Technologies, Ltd Content oriented index and search method and system
US20060010167A1 (en) * 2004-01-21 2006-01-12 Grace James R Apparatus for navigation of multimedia content in a vehicle multimedia system
US7885926B2 (en) * 2004-01-21 2011-02-08 GM Global Technology Operations LLC System and apparatus for wireless synchronization of multimedia content
US20100318529A1 (en) * 2004-03-26 2010-12-16 Rahav Dor Method of accessing a work of art, a product, or other tangible or intangible objects without knowing the title or name thereof using fractional sampling of the work of art or object
EP1745463B1 (en) * 2004-04-29 2008-07-09 Koninklijke Philips Electronics N.V. Method of and system for classification of an audio signal
US20060005226A1 (en) * 2004-05-17 2006-01-05 Lee Peter S System and method for synchronization of a portable media player to a user's profile
US7626110B2 (en) * 2004-06-02 2009-12-01 Stmicroelectronics Asia Pacific Pte. Ltd. Energy-based audio pattern recognition
US7563971B2 (en) * 2004-06-02 2009-07-21 Stmicroelectronics Asia Pacific Pte. Ltd. Energy-based audio pattern recognition with weighting of energy matches
US7574742B2 (en) * 2004-11-15 2009-08-11 Industrial Technology Research Institute System and method of string matching for uniform data classification
ITMI20050907A1 (en) * 2005-05-18 2006-11-20 Euriski Nop World S R L METHOD AND SYSTEM FOR THE COMPARISON OF AUDIO SIGNALS AND THE IDENTIFICATION OF A SOUND SOURCE
JP4559943B2 (en) * 2005-06-24 2010-10-13 株式会社東芝 Playlist creation device, playlist creation method, playlist creation program, copyright information management device, and viewing terminal device
US7765192B2 (en) 2006-03-29 2010-07-27 Abo Enterprises, Llc System and method for archiving a media collection
US7772478B2 (en) * 2006-04-12 2010-08-10 Massachusetts Institute Of Technology Understanding music
US7444388B1 (en) * 2006-04-13 2008-10-28 Concert Technology Corporation System and method for obtaining media content for a portable media player
WO2007133754A2 (en) * 2006-05-12 2007-11-22 Owl Multimedia, Inc. Method and system for music information retrieval
US8620699B2 (en) 2006-08-08 2013-12-31 Napo Enterprises, Llc Heavy influencer media recommendations
JP4239109B2 (en) * 2006-10-20 2009-03-18 ソニー株式会社 Information processing apparatus and method, program, and recording medium
US8307092B2 (en) 2007-02-21 2012-11-06 Napo Enterprises, Llc Method and system for collecting information about a user's media collections from multiple login points
WO2008110002A1 (en) * 2007-03-12 2008-09-18 Webhitcontest Inc. A method and a system for automatic evaluation of digital files
US7873069B2 (en) * 2007-03-12 2011-01-18 Avaya Inc. Methods and apparatus for controlling audio characteristics of networked voice communications devices
US7873634B2 (en) * 2007-03-12 2011-01-18 Hitlab Ulc. Method and a system for automatic evaluation of digital files
US9425993B2 (en) * 2007-03-28 2016-08-23 Avaya Inc. Methods and apparatus for customizing the audio characteristics of networked voice communications devices
US8280539B2 (en) * 2007-04-06 2012-10-02 The Echo Nest Corporation Method and apparatus for automatically segueing between audio tracks
US7949649B2 (en) * 2007-04-10 2011-05-24 The Echo Nest Corporation Automatically acquiring acoustic and cultural information about music
US8073854B2 (en) * 2007-04-10 2011-12-06 The Echo Nest Corporation Determining the similarity of music using cultural and acoustic information
JP5090523B2 (en) * 2007-06-06 2012-12-05 ドルビー ラボラトリーズ ライセンシング コーポレイション Method and apparatus for improving audio / video fingerprint search accuracy using a combination of multiple searches
US8140331B2 (en) * 2007-07-06 2012-03-20 Xia Lou Feature extraction for identification and classification of audio signals
US8122032B2 (en) * 2007-07-20 2012-02-21 Google Inc. Identifying and linking similar passages in a digital text corpus
US9323827B2 (en) * 2007-07-20 2016-04-26 Google Inc. Identifying key terms related to similar passages
US20090150445A1 (en) * 2007-12-07 2009-06-11 Tilman Herberger System and method for efficient generation and management of similarity playlists on portable devices
US8352479B2 (en) 2007-12-10 2013-01-08 At&T Intellectual Property I, L.P. Systems,methods and computer products for content-derived metadata
US8285718B1 (en) * 2007-12-21 2012-10-09 CastTV Inc. Clustering multimedia search
US7958130B2 (en) * 2008-05-26 2011-06-07 Microsoft Corporation Similarity-based content sampling and relevance feedback
KR20100057307A (en) * 2008-11-21 2010-05-31 삼성전자주식회사 Singing score evaluation method and karaoke apparatus using the same
US8751494B2 (en) * 2008-12-15 2014-06-10 Rovi Technologies Corporation Constructing album data using discrete track data from multiple sources
US8209313B2 (en) 2009-01-28 2012-06-26 Rovi Technologies Corporation Structuring and searching data in a hierarchical confidence-based configuration
US20100205222A1 (en) * 2009-02-10 2010-08-12 Tom Gajdos Music profiling
JP2010257162A (en) * 2009-04-23 2010-11-11 Brother Ind Ltd Install program
US8718805B2 (en) * 2009-05-27 2014-05-06 Spot411 Technologies, Inc. Audio-based synchronization to media
US8620967B2 (en) * 2009-06-11 2013-12-31 Rovi Technologies Corporation Managing metadata for occurrences of a recording
US8359315B2 (en) * 2009-06-11 2013-01-22 Rovi Technologies Corporation Generating a representative sub-signature of a cluster of signatures by using weighted sampling
US20110041154A1 (en) 2009-08-14 2011-02-17 All Media Guide, Llc Content Recognition and Synchronization on a Television or Consumer Electronics Device
US8239443B2 (en) * 2009-09-01 2012-08-07 Rovi Technologies Corporation Method and system for tunable distribution of content
US20110066700A1 (en) * 2009-09-15 2011-03-17 Jackson Chao Behavior monitoring system
US20110072117A1 (en) * 2009-09-23 2011-03-24 Rovi Technologies Corporation Generating a Synthetic Table of Contents for a Volume by Using Statistical Analysis
US8161071B2 (en) 2009-09-30 2012-04-17 United Video Properties, Inc. Systems and methods for audio asset storage and management
US8677400B2 (en) 2009-09-30 2014-03-18 United Video Properties, Inc. Systems and methods for identifying audio content using an interactive media guidance application
US8706276B2 (en) 2009-10-09 2014-04-22 The Trustees Of Columbia University In The City Of New York Systems, methods, and media for identifying matching audio
US8428955B2 (en) * 2009-10-13 2013-04-23 Rovi Technologies Corporation Adjusting recorder timing
US20110085781A1 (en) * 2009-10-13 2011-04-14 Rovi Technologies Corporation Content recorder timing alignment
WO2011046719A1 (en) 2009-10-13 2011-04-21 Rovi Technologies Corporation Adjusting recorder timing
US8321394B2 (en) * 2009-11-10 2012-11-27 Rovi Technologies Corporation Matching a fingerprint
EP2519910A4 (en) * 2009-12-31 2016-10-19 Redigi Inc Methods and apparatus for sharing, transferring and removing previously owned digital media
US8886531B2 (en) 2010-01-13 2014-11-11 Rovi Technologies Corporation Apparatus and method for generating an audio fingerprint and using a two-stage query
US20110173185A1 (en) * 2010-01-13 2011-07-14 Rovi Technologies Corporation Multi-stage lookup for rolling audio recognition
US20110238679A1 (en) * 2010-03-24 2011-09-29 Rovi Technologies Corporation Representing text and other types of content by using a frequency domain
US8725766B2 (en) * 2010-03-25 2014-05-13 Rovi Technologies Corporation Searching text and other types of content by using a frequency domain
US9159338B2 (en) 2010-05-04 2015-10-13 Shazam Entertainment Ltd. Systems and methods of rendering a textual animation
EP2567332A1 (en) 2010-05-04 2013-03-13 Shazam Entertainment Ltd. Methods and systems for processing a sample of a media stream
WO2011140221A1 (en) 2010-05-04 2011-11-10 Shazam Entertainment Ltd. Methods and systems for synchronizing media
US8239412B2 (en) 2010-05-05 2012-08-07 Rovi Technologies Corporation Recommending a media item by using audio content from a seed media item
US20110289121A1 (en) 2010-05-18 2011-11-24 Rovi Technologies Corporation Metadata modifier and manager
US8527268B2 (en) 2010-06-30 2013-09-03 Rovi Technologies Corporation Method and apparatus for improving speech recognition and identifying video program material or content
US20120020647A1 (en) 2010-07-21 2012-01-26 Rovi Technologies Corporation Filtering repeated content
US20120023403A1 (en) * 2010-07-21 2012-01-26 Tilman Herberger System and method for dynamic generation of individualized playlists according to user selection of musical features
WO2012015846A1 (en) 2010-07-26 2012-02-02 Rovi Technologies Corporation Delivering regional content information from a content information sources to a user device
US8761545B2 (en) 2010-11-19 2014-06-24 Rovi Technologies Corporation Method and apparatus for identifying video program material or content via differential signals
JP5750167B2 (en) 2010-12-07 2015-07-15 エンパイア テクノロジー ディベロップメント エルエルシー Audio fingerprint difference for measuring quality of experience between devices
US8676794B2 (en) 2011-02-09 2014-03-18 Bellmar Communications Llc Method and system for online searching of physical objects
EP2678798A1 (en) 2011-02-27 2014-01-01 Redigi Inc. Methods and apparatus for sharing, transferring and removing previously owned digital media
US8478719B2 (en) 2011-03-17 2013-07-02 Remote Media LLC System and method for media file synchronization
US8589171B2 (en) 2011-03-17 2013-11-19 Remote Media, Llc System and method for custom marking a media file for file matching
US8688631B2 (en) 2011-03-17 2014-04-01 Alexander Savenok System and method for media file synchronization
EP2718849A1 (en) 2011-06-10 2014-04-16 Shazam Entertainment Ltd. Methods and systems for identifying content in a data stream
US9384272B2 (en) * 2011-10-05 2016-07-05 The Trustees Of Columbia University In The City Of New York Methods, systems, and media for identifying similar songs using jumpcodes
US9054876B1 (en) * 2011-11-04 2015-06-09 Google Inc. Fast efficient vocabulary computation with hashed vocabularies applying hash functions to cluster centroids that determines most frequently used cluster centroid IDs
US8586847B2 (en) * 2011-12-02 2013-11-19 The Echo Nest Corporation Musical fingerprinting based on onset intervals
US8492633B2 (en) 2011-12-02 2013-07-23 The Echo Nest Corporation Musical fingerprinting
US20140040201A1 (en) * 2012-08-01 2014-02-06 Redigi, Inc. Transfer of Digital Media Objects Via Migration
US8798438B1 (en) * 2012-12-07 2014-08-05 Google Inc. Automatic video generation for music playlists
US9159327B1 (en) * 2012-12-20 2015-10-13 Google Inc. System and method for adding pitch shift resistance to an audio fingerprint
CN103971689B (en) * 2013-02-04 2016-01-27 腾讯科技(深圳)有限公司 A kind of audio identification methods and device
US10122983B1 (en) * 2013-03-05 2018-11-06 Google Llc Creating a video for an audio file
US9451048B2 (en) 2013-03-12 2016-09-20 Shazam Investments Ltd. Methods and systems for identifying information of a broadcast station and information of broadcasted content
US9773058B2 (en) 2013-03-15 2017-09-26 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US9390170B2 (en) 2013-03-15 2016-07-12 Shazam Investments Ltd. Methods and systems for arranging and searching a database of media content recordings
US9654898B2 (en) * 2013-10-21 2017-05-16 Amazon Technologies, Inc. Managing media content, federated player
US9639607B2 (en) * 2013-10-21 2017-05-02 Amazon Technologies Inc. Managing media content, playlist sharing
US9094453B2 (en) 2013-11-06 2015-07-28 Google Technology Holdings LLC Method and apparatus for associating mobile devices using audio signature detection
US9300991B2 (en) 2013-11-13 2016-03-29 International Business Machines Corporation Use of simultaneously received videos by a system to generate a quality of experience value
CN104093079B (en) 2014-05-29 2015-10-07 腾讯科技(深圳)有限公司 Based on the exchange method of multimedia programming, terminal, server and system
US9905233B1 (en) 2014-08-07 2018-02-27 Digimarc Corporation Methods and apparatus for facilitating ambient content recognition using digital watermarks, and related arrangements
CN105589970A (en) * 2015-12-25 2016-05-18 小米科技有限责任公司 Music searching method and device
JP6658181B2 (en) 2016-03-24 2020-03-04 ティアック株式会社 Audio device and audio system
JP6672934B2 (en) 2016-03-24 2020-03-25 ティアック株式会社 Audio signal processing device and program
JP6617632B2 (en) * 2016-03-24 2019-12-11 ティアック株式会社 Audio / video signal processing apparatus and program
WO2018046088A1 (en) 2016-09-09 2018-03-15 Huawei Technologies Co., Ltd. A device and method for classifying an acoustic environment
US9934785B1 (en) 2016-11-30 2018-04-03 Spotify Ab Identification of taste attributes from an audio signal
US11354510B2 (en) 2016-12-01 2022-06-07 Spotify Ab System and method for semantic analysis of song lyrics in a media content environment
US10360260B2 (en) * 2016-12-01 2019-07-23 Spotify Ab System and method for semantic analysis of song lyrics in a media content environment
US10762347B1 (en) 2017-05-25 2020-09-01 David Andrew Caulkins Waveform generation and recognition system
US11238287B2 (en) * 2020-04-02 2022-02-01 Rovi Guides, Inc. Systems and methods for automated content curation using signature analysis
US11574248B2 (en) 2020-04-02 2023-02-07 Rovi Guides, Inc. Systems and methods for automated content curation using signature analysis
US11516347B2 (en) 2020-06-30 2022-11-29 ROVl GUIDES, INC. Systems and methods to automatically join conference

Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4677466A (en) * 1985-07-29 1987-06-30 A. C. Nielsen Company Broadcast program identification method and apparatus
US4843562A (en) * 1987-06-24 1989-06-27 Broadcast Data Systems Limited Partnership Broadcast information classification system and method
US5199077A (en) * 1991-09-19 1993-03-30 Xerox Corporation Wordspotting for voice editing and indexing
US5210820A (en) * 1990-05-02 1993-05-11 Broadcast Data Systems Limited Partnership Signal recognition system and method
US5432852A (en) * 1993-09-29 1995-07-11 Leighton; Frank T. Large provably fast and secure digital signature schemes based on secure hash functions
US5612729A (en) * 1992-04-30 1997-03-18 The Arbitron Company Method and system for producing a signature characterizing an audio broadcast signal
US5862260A (en) * 1993-11-18 1999-01-19 Digimarc Corporation Methods for surveying dissemination of proprietary empirical data
US5918223A (en) * 1996-07-22 1999-06-29 Muscle Fish Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information
US5960388A (en) * 1992-03-18 1999-09-28 Sony Corporation Voiced/unvoiced decision based on frequency band ratio
US5987525A (en) * 1997-04-15 1999-11-16 Cddb, Inc. Network delivery of interactive entertainment synchronized to playback of audio recordings
US6304523B1 (en) * 1999-01-05 2001-10-16 Openglobe, Inc. Playback device having text display and communication with remote database of titles
US20020023020A1 (en) * 1999-09-21 2002-02-21 Kenyon Stephen C. Audio identification system and method
US20020087565A1 (en) * 2000-07-06 2002-07-04 Hoekman Jeffrey S. System and methods for providing automatic classification of media entities according to consonance properties
US20020101989A1 (en) * 1997-06-02 2002-08-01 Vishal Markandey Data protection system
US6453252B1 (en) * 2000-05-15 2002-09-17 Creative Technology Ltd. Process for identifying audio content
US6505160B1 (en) * 1995-07-27 2003-01-07 Digimarc Corporation Connected audio and other media objects
US6512796B1 (en) * 1996-03-04 2003-01-28 Douglas Sherwood Method and system for inserting and retrieving data in an audio signal
US20030028796A1 (en) * 2001-07-31 2003-02-06 Gracenote, Inc. Multiple step identification of recordings
US6539395B1 (en) * 2000-03-22 2003-03-25 Mood Logic, Inc. Method for creating a database for comparing music
US20030086341A1 (en) * 2001-07-20 2003-05-08 Gracenote, Inc. Automatic identification of sound recordings
US6571144B1 (en) * 1999-10-20 2003-05-27 Intel Corporation System for providing a digital watermark in an audio signal
US6570991B1 (en) * 1996-12-18 2003-05-27 Interval Research Corporation Multi-feature speech/music discrimination system
US20030135513A1 (en) * 2001-08-27 2003-07-17 Gracenote, Inc. Playlist generation, delivery and navigation
US6675174B1 (en) * 2000-02-02 2004-01-06 International Business Machines Corp. System and method for measuring similarity between a set of known temporal media segments and a one or more temporal media streams
US6829368B2 (en) * 2000-01-26 2004-12-07 Digimarc Corporation Establishing and interacting with on-line media collections using identifiers in media signals

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5473759A (en) * 1993-02-22 1995-12-05 Apple Computer, Inc. Sound analysis and resynthesis using correlograms
US5825830A (en) * 1995-08-17 1998-10-20 Kopf; David A. Method and apparatus for the compression of audio, video or other data
US7167857B2 (en) 1997-04-15 2007-01-23 Gracenote, Inc. Method and system for finding approximate matches in database
IL122498A0 (en) 1997-12-07 1998-06-15 Contentwise Ltd Apparatus and methods for manipulating sequences of images
US6201176B1 (en) * 1998-05-07 2001-03-13 Canon Kabushiki Kaisha System and method for querying a music database
US6826350B1 (en) * 1998-06-01 2004-11-30 Nippon Telegraph And Telephone Corporation High-speed signal search method device and recording medium for the same
AU5460299A (en) * 1998-07-24 2000-02-14 Jarg Corporation Distributed computer database system and method for performing object search
US6434520B1 (en) * 1999-04-16 2002-08-13 International Business Machines Corporation System and method for indexing and querying audio archives
US7185201B2 (en) 1999-05-19 2007-02-27 Digimarc Corporation Content identifiers triggering corresponding responses
US6321200B1 (en) * 1999-07-02 2001-11-20 Mitsubish Electric Research Laboratories, Inc Method for extracting features from a mixture of signals
US8326584B1 (en) 1999-09-14 2012-12-04 Gracenote, Inc. Music searching methods based on human perception
US8528019B1 (en) 1999-11-18 2013-09-03 Koninklijke Philips N.V. Method and apparatus for audio/data/visual information
US6963975B1 (en) * 2000-08-11 2005-11-08 Microsoft Corporation System and method for audio fingerprinting
US6657117B2 (en) * 2000-07-14 2003-12-02 Microsoft Corporation System and methods for providing automatic classification of media entities according to tempo properties
WO2002065782A1 (en) 2001-02-12 2002-08-22 Koninklijke Philips Electronics N.V. Generating and matching hashes of multimedia content
DE10109648C2 (en) 2001-02-28 2003-01-30 Fraunhofer Ges Forschung Method and device for characterizing a signal and method and device for generating an indexed signal
DE10134471C2 (en) 2001-02-28 2003-05-22 Fraunhofer Ges Forschung Method and device for characterizing a signal and method and device for generating an indexed signal
US7058889B2 (en) 2001-03-23 2006-06-06 Koninklijke Philips Electronics N.V. Synchronizing text/visual information with audio playback
DE10133333C1 (en) 2001-07-10 2002-12-05 Fraunhofer Ges Forschung Producing fingerprint of audio signal involves setting first predefined fingerprint mode from number of modes and computing a fingerprint in accordance with set predefined mode
DE10200653B4 (en) 2002-01-10 2004-05-27 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Scalable encoder, encoding method, decoder and decoding method for a scaled data stream
KR20040086350A (en) 2002-02-05 2004-10-08 코닌클리케 필립스 일렉트로닉스 엔.브이. Efficient storage of fingerprints
CN100353767C (en) 2002-05-10 2007-12-05 皇家飞利浦电子股份有限公司 Watermark embedding and retrieval
WO2004040416A2 (en) 2002-10-28 2004-05-13 Gracenote, Inc. Personal audio recording system
AU2003274545A1 (en) 2002-11-12 2004-06-03 Koninklijke Philips Electronics N.V. Fingerprinting multimedia contents
KR20050113614A (en) 2003-02-26 2005-12-02 코닌클리케 필립스 일렉트로닉스 엔.브이. Handling of digital silence in audio fingerprinting
EP1457889A1 (en) 2003-03-13 2004-09-15 Koninklijke Philips Electronics N.V. Improved fingerprint matching method and system

Patent Citations (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4677466A (en) * 1985-07-29 1987-06-30 A. C. Nielsen Company Broadcast program identification method and apparatus
US4843562A (en) * 1987-06-24 1989-06-27 Broadcast Data Systems Limited Partnership Broadcast information classification system and method
US5210820A (en) * 1990-05-02 1993-05-11 Broadcast Data Systems Limited Partnership Signal recognition system and method
US5199077A (en) * 1991-09-19 1993-03-30 Xerox Corporation Wordspotting for voice editing and indexing
US5960388A (en) * 1992-03-18 1999-09-28 Sony Corporation Voiced/unvoiced decision based on frequency band ratio
US5612729A (en) * 1992-04-30 1997-03-18 The Arbitron Company Method and system for producing a signature characterizing an audio broadcast signal
US5432852A (en) * 1993-09-29 1995-07-11 Leighton; Frank T. Large provably fast and secure digital signature schemes based on secure hash functions
US5862260A (en) * 1993-11-18 1999-01-19 Digimarc Corporation Methods for surveying dissemination of proprietary empirical data
US6505160B1 (en) * 1995-07-27 2003-01-07 Digimarc Corporation Connected audio and other media objects
US6512796B1 (en) * 1996-03-04 2003-01-28 Douglas Sherwood Method and system for inserting and retrieving data in an audio signal
US5918223A (en) * 1996-07-22 1999-06-29 Muscle Fish Method and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information
US6570991B1 (en) * 1996-12-18 2003-05-27 Interval Research Corporation Multi-feature speech/music discrimination system
US6061680A (en) * 1997-04-15 2000-05-09 Cddb, Inc. Method and system for finding approximate matches in database
US6230207B1 (en) * 1997-04-15 2001-05-08 Cddb, Inc. Network delivery of interactive entertainment synchronized to playback of audio recordings
US6230192B1 (en) * 1997-04-15 2001-05-08 Cddb, Inc. Method and system for accessing remote data based on playback of recordings
US6240459B1 (en) * 1997-04-15 2001-05-29 Cddb, Inc. Network delivery of interactive entertainment synchronized to playback of audio recordings
US6161132A (en) * 1997-04-15 2000-12-12 Cddb, Inc. System for synchronizing playback of recordings and display by networked computer systems
US6330593B1 (en) * 1997-04-15 2001-12-11 Cddb Inc. System for collecting use data related to playback of recordings
US6154773A (en) * 1997-04-15 2000-11-28 Cddb, Inc. Network delivery of interactive entertainment complementing audio recordings
US5987525A (en) * 1997-04-15 1999-11-16 Cddb, Inc. Network delivery of interactive entertainment synchronized to playback of audio recordings
US20020101989A1 (en) * 1997-06-02 2002-08-01 Vishal Markandey Data protection system
US6304523B1 (en) * 1999-01-05 2001-10-16 Openglobe, Inc. Playback device having text display and communication with remote database of titles
US20020023020A1 (en) * 1999-09-21 2002-02-21 Kenyon Stephen C. Audio identification system and method
US6571144B1 (en) * 1999-10-20 2003-05-27 Intel Corporation System for providing a digital watermark in an audio signal
US6829368B2 (en) * 2000-01-26 2004-12-07 Digimarc Corporation Establishing and interacting with on-line media collections using identifiers in media signals
US6675174B1 (en) * 2000-02-02 2004-01-06 International Business Machines Corp. System and method for measuring similarity between a set of known temporal media segments and a one or more temporal media streams
US6539395B1 (en) * 2000-03-22 2003-03-25 Mood Logic, Inc. Method for creating a database for comparing music
US6453252B1 (en) * 2000-05-15 2002-09-17 Creative Technology Ltd. Process for identifying audio content
US20020087565A1 (en) * 2000-07-06 2002-07-04 Hoekman Jeffrey S. System and methods for providing automatic classification of media entities according to consonance properties
US20030086341A1 (en) * 2001-07-20 2003-05-08 Gracenote, Inc. Automatic identification of sound recordings
US20030028796A1 (en) * 2001-07-31 2003-02-06 Gracenote, Inc. Multiple step identification of recordings
US20030135513A1 (en) * 2001-08-27 2003-07-17 Gracenote, Inc. Playlist generation, delivery and navigation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Jain et al., "Algorithms for Clustering Data", 1988, Prentice Hall, pp. ix-xi and 14-16. *

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8583656B1 (en) 2009-04-23 2013-11-12 Google Inc. Fast covariance matrix generation
US8396325B1 (en) 2009-04-27 2013-03-12 Google Inc. Image enhancement through discrete patch optimization
US8611695B1 (en) 2009-04-27 2013-12-17 Google Inc. Large scale patch search
US8867859B1 (en) 2009-04-28 2014-10-21 Google Inc. Illumination estimation for images
US8391634B1 (en) 2009-04-28 2013-03-05 Google Inc. Illumination estimation for images
US8385662B1 (en) * 2009-04-30 2013-02-26 Google Inc. Principal component analysis based seed generation for clustering analysis
US8660370B1 (en) * 2009-04-30 2014-02-25 Google Inc. Principal component analysis based seed generation for clustering analysis
US9026034B2 (en) 2010-05-04 2015-05-05 Project Oda, Inc. Automatic detection of broadcast programming
US9020415B2 (en) 2010-05-04 2015-04-28 Project Oda, Inc. Bonus and experience enhancement system for receivers of broadcast media
US8798393B2 (en) 2010-12-01 2014-08-05 Google Inc. Removing illumination variation from images
US20120254806A1 (en) * 2011-03-30 2012-10-04 Google Inc. System and method for dynamic, feature-based playlist generation
US8319087B2 (en) * 2011-03-30 2012-11-27 Google Inc. System and method for dynamic, feature-based playlist generation
US8732739B2 (en) 2011-07-18 2014-05-20 Viggle Inc. System and method for tracking and rewarding media and entertainment usage including substantially real time rewards
WO2013106343A3 (en) * 2012-01-09 2015-05-14 Viggle, Inc. Method and system for identifying a media program from an audio signal associated with the media program
US8938119B1 (en) 2012-05-01 2015-01-20 Google Inc. Facade illumination removal
CN103870476A (en) * 2012-12-12 2014-06-18 百度在线网络技术(北京)有限公司 Retrieval method and device
US9922118B2 (en) 2015-04-28 2018-03-20 International Business Machines Corporation Creating an audio file sample based upon user preferences
US10372754B2 (en) 2015-04-28 2019-08-06 International Business Machines Corporation Creating an audio file sample based upon user preferences
US20170092089A1 (en) * 2015-09-30 2017-03-30 Tianjin Hualai Technology Co., Ltd. Security monitoring apparatus, camera having the same and security monitoring method
CN110072183A (en) * 2019-03-14 2019-07-30 天津大学 Passive type location fingerprint base construction method based on intelligent perception
WO2022186952A1 (en) * 2021-03-04 2022-09-09 Gracenote Inc. Methods and apparatus to fingerprint an audio signal
US11798577B2 (en) 2021-03-04 2023-10-24 Gracenote, Inc. Methods and apparatus to fingerprint an audio signal

Also Published As

Publication number Publication date
US7853344B2 (en) 2010-12-14
US7277766B1 (en) 2007-10-02
US20070282935A1 (en) 2007-12-06

Similar Documents

Publication Publication Date Title
US7277766B1 (en) Method and system for analyzing digital audio files
US7326848B2 (en) System and methods for providing automatic classification of media entities according to tempo properties
US7574276B2 (en) System and methods for providing automatic classification of media entities according to melodic movement properties
US7532943B2 (en) System and methods for providing automatic classification of media entities according to sonic properties
US7756874B2 (en) System and methods for providing automatic classification of media entities according to consonance properties
US7035873B2 (en) System and methods for providing adaptive media property classification
JP4945877B2 (en) System and method for recognizing sound / musical signal under high noise / distortion environment
US8497417B2 (en) Intervalgram representation of audio for melody recognition
EP2659480B1 (en) Repetition detection in media data
CN100461168C (en) Systems and methods for generating audio thumbnails
TWI484473B (en) Method and system for extracting tempo information of audio signal from an encoded bit-stream, and estimating perceptually salient tempo of audio signal
Burred et al. Hierarchical automatic audio signal classification
JP2015505992A (en) Low complexity iterative detection in media data
KR100512143B1 (en) Method and apparatus for searching of musical data based on melody
You et al. Comparative study of singing voice detection methods
Wieczorkowska et al. Identification of a dominating instrument in polytimbral same-pitch mixes using SVM classifiers with non-linear kernel
KR20100007102A (en) Online digital contents management system
Lerch An introduction to audio content analysis: Music Information Retrieval tasks and applications
Doherty et al. Pattern matching techniques for replacing missing sections of audio streamed across wireless networks
KR20020053979A (en) Apparatus and method for contents-based musical data searching
Endrjukaite et al. Time-dependent genre recognition by means of instantaneous frequency spectrum based on Hilbert-Huang transform
US20230129350A1 (en) Section-based music similarity searching
Lagrange et al. Spectral similarity metrics for sound source formation based on the common variation cue
Lampropoulos et al. A middleware system for Web-based digital music libraries
Ong Computing structural descriptions of music through the identification of representative excerpts from audio files

Legal Events

Date Code Title Description
AS Assignment

Owner name: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT, NE

Free format text: SECURITY INTEREST;ASSIGNORS:APTIV DIGITAL, INC., A DELAWARE CORPORATION;GEMSTAR DEVELOPMENT CORPORATION, A CALIFORNIA CORPORATION;INDEX SYSTEMS INC, A BRITISH VIRGIN ISLANDS COMPANY;AND OTHERS;REEL/FRAME:027039/0168

Effective date: 20110913

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: INDEX SYSTEMS INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: ROVI SOLUTIONS CORPORATION, CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: GEMSTAR DEVELOPMENT CORPORATION, CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: ROVI GUIDES, INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: ROVI TECHNOLOGIES CORPORATION, CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: ALL MEDIA GUIDE, LLC, CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: STARSIGHT TELECAST, INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: APTIV DIGITAL, INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: ROVI CORPORATION, CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: TV GUIDE INTERNATIONAL, INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702

Owner name: UNITED VIDEO PROPERTIES, INC., CALIFORNIA

Free format text: PATENT RELEASE;ASSIGNOR:JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT;REEL/FRAME:033396/0001

Effective date: 20140702