Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20030028796 A1
Publication typeApplication
Application numberUS 10/208,189
Publication dateFeb 6, 2003
Filing dateJul 31, 2002
Priority dateJul 31, 2001
Also published asEP1421521A2, US8468357, US20100158488, US20100161656, WO2003012695A2, WO2003012695A3
Publication number10208189, 208189, US 2003/0028796 A1, US 2003/028796 A1, US 20030028796 A1, US 20030028796A1, US 2003028796 A1, US 2003028796A1, US-A1-20030028796, US-A1-2003028796, US2003/0028796A1, US2003/028796A1, US20030028796 A1, US20030028796A1, US2003028796 A1, US2003028796A1
InventorsDale Roberts, David Hyman, Stephen White
Original AssigneeGracenote, Inc.
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Multiple step identification of recordings
US 20030028796 A1
Abstract
Multiple information is extracted from an unknown recording and information associated therewith. Associated information includes the filename, if the recording is a computer file in, e.g., MP3 format, or table of contents (TOC) data, if the recording is on a removable medium, such as a compact disc. At least one and preferably several algorithmically determined fingerprints are extracted from the recording using one or more fingerprint extraction methods. The information extracted is compared with corresponding information in a database maintained for reference recordings. Identification starts with the most accurate and efficient method available, e.g., using a hash ID, a unique ID or text. Fingerprint matching is used to confirm other matches and validation is performed by comparing the duration of the unknown and a possibly matching reference recording.
Images(7)
Previous page
Next page
Claims(67)
What is claimed is:
1. A method of identifying recordings, comprising:
extracting information about an unknown recording stored in media possessed by a user and at least one algorithmically determined fingerprint from at least one portion of the unknown recording;
determining a possible identification of the unknown recording using at least one piece of the information extracted from the unknown recording and an identification database of corresponding information for reference recordings; and
identifying the unknown recording when the possible identification based on each of the at least one piece of the information in combination with the at least one algorithmically determined fingerprint identifies a single reference recording with respective confidence levels.
2. A method as recited in claim 1,
wherein the identification database is maintained by a provider of identification services, and
wherein said determining uses a unique identifier from the provider of identification services when the unique identifier is associated with the unknown recording and otherwise uses text associated with the unknown recording when text is associated with the unknown recording.
3. A method as recited in claim 2, further comprising validating said identifying by comparing an extracted length of the unknown recording with a stored length of the single reference recording.
4. A method as recited in claim 3, wherein the text associated with the unknown recording includes a filename of the recording.
5. A method as recited in claim 3, wherein the text associated with the unknown recording includes an ID3 tag for the recording.
6. A method as recited in claim 1, wherein the at least one algorithmically determined fingerprint is extracted from at least one of audio and video information in the at least one portion of the unknown recording.
7. A method as recited in claim 1,
wherein the at least one algorithmically determined fingerprint includes at least two fingerprints, and
wherein said identifying requires each of the at least two fingerprints to identify the single reference recording with respective confidence levels.
8. A method as recited in claim 1, further comprising validating said identifying by comparing an extracted length of the unknown recording with a stored length of the single reference recording.
9. A method as recited in claim 8, further comprising:
repeating said extracting, determining and identifying for a plurality unknown recordings from a plurality users;
monitoring unsuccessful identifications of the unknown recordings; and
detecting a possible error in the identification database from a pattern of errors.
10. A method as recited in claim 9, wherein said monitoring includes receiving information from the users indicating that said identifying was incorrect.
11. A method as recited in claim 9,
wherein said monitoring includes storing the at least one algorithmically determined fingerprint and an identifier of the single reference recording when said validating is not successful, and
wherein said method further comprises indicating the possible error in the identification database when substantially different fingerprints are stored for a single identifier.
12. A method as recited in claim 9, wherein said method further comprises indicating the possible error when the at least one algorithmically determined fingerprint matches one of the reference recordings, but the unknown recording is associated with ID3 Tag information different from that of the one of the reference recordings.
13. A method as recited in claim 9, further comprising correcting the possible error based on the information extracted from the unknown recordings.
14. A method as recited in claim 8, further comprising indicating a possible error in the identification database when the at least one algorithmically determined fingerprint is substantially similar to one of the reference recordings, but substantially different information is extracted from the unknown recording.
15. A method as recited in claim 1, further comprising delivering related data for the unknown recording from a supplemental database to supplement data embedded within the unknown recording for display and user manipulation.
16. A method of identifying recordings, comprising:
extracting fingerprints from at least one portion of an unknown recording using a plurality of algorithms;
determining a possible identification of the unknown recording using at least two of the fingerprints extracted from the unknown recording and at least one database of correspondingly generated fingerprints for reference recordings; and
identifying the unknown recording when the possible identification based on each of the fingerprints identifies a single reference recording with respective confidence levels.
17. A method as recited in claim 16, wherein each fingerprint is extracted from at least one of audio and video information in the at least one portion of the unknown recording.
18. A method as recited in claim 16, further comprising validating said identifying by comparing a length of the unknown recording with a stored length of the single reference recording.
19. A method as recited in claim 18,
wherein said extracting is performed by client equipment possessed by a user,
wherein said determining, identifying and validating are performed by at least one server under control of a provider of identification services, and
wherein said method further comprises:
transmitting a unique identifier associated with the single reference recording from the at least one server to the client equipment after said validating is successful; and
associating the unique identifier with the unknown recording in the client equipment.
20. A method as recited in claim 19, further comprising:
comparing the unique identifier with a permission list of stored identifiers in the at least one database;
indicating that the recording may be shared if there is a match for the unique identifier in the permission list.
21. A method as recited in claim 19, further comprising:
comparing the unique identifier with a block list of stored identifiers in the at least one database;
indicating that the recording may not be shared if there is a match for the unique identifier in the block list.
22. A method of obtaining reference information stored in a database used to identify unknown recordings, comprising:
obtaining non-waveform data associated with a recording possessed by a user of the database for identification of recordings possessed by the user;
extracting at least one fingerprint from at least one portion of the recording; and
storing the at least one fingerprint as identifying information for the recording, when a match is found in the database for the non-waveform data.
23. A method as recited in claim 22, wherein the non-waveform data indicates the length of the recording.
24. A method as recited in claim 23, wherein the recording is permanently stored on a removable medium and the non-waveform data is derived from table of contents data for the recording.
25. A method as recited in claim 22, wherein the non-waveform data includes text associated with the recording.
26. A method as recited in claim 25, wherein the non-waveform data includes and ID3 tag.
27. A method as recited in claim 26,
wherein the ID3 tag includes encoded information,
wherein the database is maintained by a provider of identification services and the encoded information is generated under control of the provider of identification services, and
wherein said method further comprises validating the non-waveform data by decoding the encoded information prior to said storing of the identifying information in the database.
28. A method as recited in claim 25, wherein the non-waveform data includes a watermark.
29. A method as recited in claim 25, wherein the non-waveform data includes media information regarding source media type.
30. A method as recited in claim 29, wherein the media information identifies the source media type as CD-R.
31. A method as recited in claim 29, wherein the media information identifies the source media type as CD-DA.
32. A method as recited in claim 29, wherein the media information identifies the source media type as a digital file.
33. A method as recited in claim 29, wherein the media information identifies the source media type as a digital versatile disc.
34. A method as recited in claim 25, wherein the text includes a filename of the recording.
35. A method as recited in claim 25, wherein the text includes a title of the recording.
36. A method as recited in claim 25, wherein the text includes an artist name of a participant in creation of the recording.
37. A method as recited in claim 25, wherein the text includes an album name associated with the recording.
38. A method as recited in claim 22, wherein said obtaining and extracting are performed by client equipment possessed by a plurality users for different copies of the recording and different users extract different fingerprints from the recording.
39. A method as recited in claim 38, further comprising:
maintaining the database on at least one server under control of a provider of identification services, and
transmitting from the at least one server to the client equipment, extraction instructions on which of the different fingerprints each of the client equipment extracts.
40. A method as recited in claim 39, further comprising:
transmitting the non-waveform data from the client equipment to the at least one server; and
selecting the extraction instructions by the at least one server for said transmitting to the client equipment based on the non-waveform data.
41. A method as recited in claim 40, further comprising updating the extraction instructions based at least in part on frequency of receipt of the non-waveform data for the recording.
42. A method as recited in claim 40, wherein said selecting of the extraction instructions is based at least in part on type of the client equipment receiving the extraction instructions.
43. A method as recited in claim 40, wherein said selecting of the extraction instructions is based at least in part on geographical location of the client equipment receiving the extraction instructions.
44. A method as recited in claim 40, wherein said selecting of the extraction instructions is based at least in part on software operating on the client equipment receiving the extraction instructions.
45. A method as recited in claim 40, further comprising updating the extraction instructions based at least in part on number of users who have supplied the identifying information.
46. A method as recited in claim 40, wherein said selecting of the extraction instructions is based at least in part on quality of the copies of the recording.
47. A method as recited in claim 40, further comprising transmitting the at least one fingerprint from the client equipment to the at least one server at a time specified by the extraction instructions.
48. A method as recited in claim 47, further comprising storing the at least one fingerprint at the client equipment until a specified number of fingerprints are ready for said transmitting.
49. A method as recited in claim 47, wherein said transmitting of the at least one fingerprint occurs when a communication channel with the at least server is available.
50. A method as recited in claim 47, wherein said transmitting of the at least one fingerprint for a first recording accessed by a piece of client equipment occurs with said transmitting of the non-waveform data for a second recording accessed by the piece of client equipment.
51. A method as recited in claim 47,
wherein the recording is permanently stored on a removable medium and the client equipment generates at least one encoded file from the recording, and
wherein said transmitting transmits the at least one fingerprint before encoding the recording.
52. A method as recited in claim 47,
wherein the recording is permanently stored on a removable medium and the client equipment generates at least one encoded file from the recording, and
wherein said transmitting transmits the at least one fingerprint after encoding one track of the removable medium.
53. A method as recited in claim 47,
wherein the recording is permanently stored on a removable medium and the client equipment generates at least one encoded file from the recording, and
wherein said transmitting transmits the at least one fingerprint after receiving an indication that encoding of the removable medium has been completed.
54. A method as recited in claim 22, wherein the database includes the identifying information for musical recordings.
55. A method as recited in claim 22, wherein the database includes the identifying information for video recordings.
56. A method as recited in claim 22, further comprising:
detecting a quality of the at least one fingerprint;
identifying another copy of the recording using the at least one fingerprint; and
replacing the at least one fingerprint with a higher quality fingerprint when the other copy of the recording produces the higher quality fingerprint.
57. A method as recited in claim 56, wherein said detecting of the quality is based on an encoding technique used for the recording.
58. A method as recited in claim 56, wherein said detecting of the quality is based on a media type used to store the recording.
59. A method as recited in claim 56, wherein said detecting of the quality is based on error correction capability of user equipment accessing the recording.
60. A method as recited in claim 59, wherein said detecting of the quality assigns higher quality when hardware error correction is used than when software error correction is by the user equipment.
61. A method as recited in claim 56, wherein said detecting of the quality is based on number of errors detected during said extracting of the fingerprint.
62. A method as recited in claim 22,
wherein said obtaining and extracting are performed by client equipment possessed by a plurality users for different copies of the recording, and
wherein said method further comprises:
comparing the at least one fingerprint obtained from one of the users with the at least one fingerprint extracted from at least one other user; and
updating the at least one fingerprint in the database based on said comparing.
63. A method as recited in claim 62, wherein said updating is performed after said comparing determines that fingerprints from different users have a predetermined correlation.
64. A method as recited in claim 62, wherein said updating combines fingerprints from different users for storage in the database.
65. A system for identifying recordings, comprising:
an extraction unit to extract information about an unknown recording stored in media possessed by a user and at least one algorithmically determined fingerprint from at least one portion of the unknown recording; and
an identification unit, coupled to said extraction unit, to make a possible identification of the unknown recording using at least one piece of the information extracted from the unknown recording and an identification database of corresponding information for reference recordings, and to identify the unknown recording when the possible identification based on each of the at least one piece of the information in combination with the at least one algorithmically determined fingerprint identifies a single reference recording with respective confidence levels.
66. A system for identifying recordings, comprising:
an extraction unit to extract fingerprints from at least one portion of an unknown recording using a plurality of algorithms; and
an identification unit, coupled to said extraction unit, to make a possible identification of the unknown recording using at least two of the fingerprints extracted from the unknown recording and at least one database of correspondingly generated fingerprints for reference recordings, and to identify the unknown recording when the possible identification based on each of the fingerprints identifies a single reference recording with respective confidence levels.
67. A system for obtaining reference information stored in a database used to identify unknown recordings, comprising:
a receiving unit to obtain non-waveform data associated with a recording possessed by a user of the database for identification of recordings possessed by the user;
an extraction unit to extract at least one fingerprint from at least one portion of the recording; and
a storage unit, coupled to said receiving unit and said extraction unit, to store the at least one fingerprint as identifying information for the recording, when a match is found in the database for the non-waveform data.
Description
    CROSS-REFERENCE TO RELATED APPLICATION(S)
  • [0001]
    This application is related to and claims priority to U.S. provisional application entitled DIGITAL MUSIC MULTIPLE STEP IDENTIFICATION METHOD AND SYSTEM having serial No. 60/308,594, by Dale T. Roberts, et al., filed Jul. 31, 2001, and incorporated by reference herein.
  • BACKGROUND OF THE INVENTION
  • [0002]
    1. Field of the Invention
  • [0003]
    The present invention is directed to recognition of recordings from their content, and, more particularly to combining fingerprint recognition with other information about a recording to increase reliability of recognition and to accomplish reliable recognition efficiently by using the least expensive forms of recognition first and layering on more complex forms as needed.
  • [0004]
    2. Description of the Related Art
  • [0005]
    There are many uses for recognition of audio (and video) recordings. Many of the uses relate to compensation or control by the rights holders for reproduction and performance of the works recorded. This use of such systems has increased in importance since the development of file sharing software, such as Napster, and the many other similar services available at the end of the twentieth century and the beginning of the twenty first century. Although the need for accurate recognition has been significant for several years, no system has been successful in meeting this need.
  • [0006]
    Another use of recording recognition is to provide added value to users when listening (or watching) recordings. One example is the CDDB Music Recognition Service from Gracenote, Inc. of Berkeley, Calif. which recognizes compact discs (CDs) and supplies information regarding a recognized CD, such as album name, artist, track names and access to related content on the Internet, including album covers, artist and fan websites, etc. While the CDDB service is effective for recognizing compact discs, there are several draw backs in using it to recognize files that are not stored on a removable disc, such as CD or DVD.
  • [0007]
    All audio fingerprinting techniques have “blind spots”, places where a system using that technique sees similarities and differences in audio where it shouldn't. By relying on just one fingerprinting technique, single source solutions are less accurate when encountering a ‘blind spot’.
  • [0008]
    One of the more popular uses for the Gracenote CDDB system is in applications that digitally encode audio files into MP3 and other formats. These encoding applications utilize Gracenote's CDDB service to recognize the compact disc being encoded and to write the correct metadata into the title and ID tags. Gracenote's CDDB service returns a unique ID (TUID) for each track and supports the insertion of such IDs in the ID3V2 tags for MP3 files. The TUID is both hashed and proprietary, and can only be read by the Gracenote system. However, the ID3V2 tags can easily be manipulated to store a TUID for one file in the ID3V2 tag for another file and therefore, the TUID alone is not a reliable identifier of the audio content in a file.
  • [0009]
    Gracenote's CDDB service also provides text matching capability that can be utilized to identify digital audio files from their file names, file paths, ID tags (titles), etc. by matching the text extracted by a client device to a metadata database of track, artist, and album names. Although this text matching utilizes user-generated spelling variants associated with each record to improve recognition, there has been no way to verify that the text matches the audio content of the recording once the recording has been separated from a compact disc and stored in a file in any format.
  • SUMMARY OF THE INVENTION
  • [0010]
    An aspect of the present invention is maximizing identification of recordings while minimizing resource usage.
  • [0011]
    Another aspect of the present invention is using multiple identification methods so that resource intensive methods, such as audio fingerprinting, are employed only when necessary.
  • [0012]
    A further aspect of the invention is minimization of processing of unidentified data.
  • [0013]
    Yet another aspect of the present invention is to use the least expensive recognition technique, with progressively more expensive recognition techniques layered onto the process until a desired confidence level is reached.
  • [0014]
    A still further aspect of the invention is validation of content-based identification of a recording by comparing text associated with an unidentified recording and text associated with identification records.
  • [0015]
    Yet another aspect of the present invention is use of recording identification methods from different sources to increase reliability.
  • [0016]
    A still further aspect of the invention is validation of content-based recording identification using fuzzy track length analysis.
  • [0017]
    Yet another aspect of the invention is automatic extraction of identification data for use in a reference database and for identification of recordings.
  • [0018]
    A still further aspect of the invention is that unidentified recordings are periodically re-run through the system to determine if recently added data or recently improved techniques will result in recognition.
  • [0019]
    The above aspects can be attained by a method of identifying recordings by extracting information about an unknown recording stored in media possessed by a user and at least one algorithmically determined fingerprint from at least one portion of the unknown recording; determining a possible identification of the unknown recording using at least one piece of the information extracted from the unknown recording and an identification database of corresponding information for reference recordings; and identifying the unknown recording when the possible identification based on each of the at least one piece of the information in combination with the at least one algorithmically determined fingerprint identifies a single reference recording with respective confidence levels. The at least one portion of the unknown recording may contain audio, video or both.
  • [0020]
    Preferably, the database is maintained by a provider of identification services which supplies unique identifiers that can be recognized only by servers under the control of the provider of identification services. The unique identifiers are associated with recordings once they have been identified. Subsequently, copies of the recordings are recognized using the unique identifiers to greatly speed up the process. The unique identifiers optionally are cached in high-speed RAM or specially indexed database tables.
  • [0021]
    When non-waveform data is not available for an unknown recording, the unknown recording is preferably identified by extracting fingerprints from at least one portion of the unknown recording using a plurality of algorithms; determining a possible identification of the unknown recording using at least two of the fingerprints extracted from the unknown recording and at least one database of correspondingly generated fingerprints for reference recordings; and identifying the unknown recording when the possible identification based on each of the fingerprints identifies a single reference recording with respective confidence levels.
  • [0022]
    Preferably, an existing database, used to identify recordings possessed by users, which does not contain fingerprint information is expanded by obtaining non-waveform data associated with a recording possessed by a user of the database; extracting at least one fingerprint from at least one portion of the recording; and storing the at least one fingerprint as identifying information for the recording, when a match is found in the database for the non-waveform data. One example is that during the process of encoding digital music files from an audio CD possessed by a user, a recognition system can be used to identify the audio CD so that fingerprints extracted during the encoding process can be directly associated with the audio CD using a unique ID system.
  • [0023]
    Recognition of recordings using either fingerprints or unique identifiers is preferably validated by other information maintained in the identification database, such as the length of the recording or a numeric identifier embedded within the recording. Information about recordings that do not pass validation or match some, but not all of the information used for identification, may be stored for later analysis of the reason for the error. If the fingerprints are obtained as described above, there may have been an error in obtaining the fingerprint. Therefore, errors may be output to an operator, or the system could correct the information stored in the database, based on recognition of patterns in the information that is stored for improper matches. For example, if a large percentage of matching fingerprints are stored, but the other information consistently does not match them, there could be an error in the fingerprint database which needs to be flagged to an operator.
  • [0024]
    The present invention includes a system for identifying recordings that includes an extraction unit to extract information about an unknown recording stored in media possessed by a user and at least one algorithmically determined fingerprint from at least one portion of the unknown recording; and an identification unit, coupled to the extraction unit, to make a possible identification of the unknown recording using at least one piece of the information extracted from the unknown recording and an identification database of corresponding information for reference recordings, and to identify the unknown recording when the possible identification based on each of the at least one piece of the information in combination with the at least one algorithmically determined fingerprint identifies a single reference recording with respective confidence levels.
  • [0025]
    The present invention also includes a system for identifying recordings that includes an extraction unit to extract fingerprints from at least one portion of an unknown recording using a plurality of algorithms, and an identification unit, coupled to said extraction unit, to make a possible identification of the unknown recording using at least two of the fingerprints extracted from the unknown recording and at least one database of correspondingly generated fingerprints for reference recordings, and to identify the unknown recording when the possible identification based on each of the fingerprints identifies a single reference recording with respective confidence levels.
  • [0026]
    In either of the systems described above, the extraction unit is typically a client unit connected by a network, such as the Internet, to at least one server as the identification unit. The client device may be a personal computer with a drive accessing the recording, a consumer electronics device with a network connection, or a server computer transmitting the unknown recording from one location to another. Furthermore, a portion of the database may be available locally and the extraction unit and identification unit may reside in the same device and share components.
  • [0027]
    The present invention also includes a system for obtaining reference information stored in a database used to identify unknown recordings, including a receiving unit to obtain non-waveform data associated with a recording possessed by a user of the database for identification of recordings possessed by the user; an extraction unit to extract at least one fingerprint from at least one portion of the recording; and a storage unit, coupled to said receiving unit and said extraction unit, to store the at least one fingerprint as identifying information for the recording, when a match is found in the database for the non-waveform data.
  • [0028]
    These together with other aspects and advantages which will be subsequently apparent, reside in the details of construction and operation as more fully hereinafter described and claimed, reference being had to the accompanying drawings forming a part hereof, wherein like numerals refer to like parts throughout.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0029]
    [0029]FIG. 1 is a functional block diagram of a system according to the present invention.
  • [0030]
    [0030]FIG. 2 is flowchart of a fingerprint extraction according to the present invention.
  • [0031]
    [0031]FIG. 3 is a flowchart of a method of recognizing unknown recordings.
  • [0032]
    FIGS. 4A-4C are a block diagram of a system according to the present invention.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • [0033]
    According to the present invention, a suite of identification components are provided in a system like that illustrated in FIG. 1 to facilitate analysis and identification of audio (and video) files utilizing multiple methods. Preferably, an existing database 90 containing recording identifiers and text data is combined with text-based digital audio and audio fingerprinting identification methods. Preferably, the text data in database 90 is obtained from user submissions and includes user-submitted spelling variants. One such database is available as the CDDB Music Recognition Service from Gracenote, Inc..
  • [0034]
    As illustrated in FIG. 1, a recording 100 is accessed by client device 110 via any conventional method, such as reading a digital audio file from a hard drive or a compact disc. Information is extracted from recording 100 and associated information (metadata). Fingerprints are extracted from recording 100, as described in more detail below. The information that is extracted from the metadata includes the duration of the recording which is the track length (from the TOC) for a CD track, the filename and ID3 tag if the recording is in an MP3 file, and the table of contents (TOC) data if the recording is on a CD. If the file containing the recording was produced by a client device operating according to the invention, a unique ID will be extracted from the ID3 file, but initially it will be assumed that information is not available.
  • [0035]
    In an exemplary embodiment, the extracted information is sent from client 110 to server 120 to determine a possible identification of the unknown recording using at least one piece of the information extracted from recording 100 and a database 130 of correspondingly generated fingerprints for reference recordings. If text or a unique ID were extracted, an attempt is made to find a match. If a match is found using the text or unique ID, at least one algorithmically determined fingerprint is compared with the fingerprint(s) stored in the matching records to determine whether there is a single reference recording that matches the information extracted from recording 100 with respective confidence levels for each item of information that matches. If no matches can be found based on text and unique ID, an attempt is made to identify the a single reference recording using at least two of the fingerprints extracted from recording 100. If a single reference recording is located using either method, preferably the duration of recording 100 is compared with the duration of the single reference recording as a final validation step.
  • [0036]
    Preferably related metadata is used for validation of the match obtained by fingerprint recognition. Like any recognition system fingerprinting can produce erroneous results. Without a validation component such an error can propagate throughout the system and return erroneous data to large percentages of users. The use of validation criteria such as track length comparison enables the system to catch potential errors and flag them for validation.
  • [0037]
    A system according to the present invention preferably includes custom result reporting and flexible administrative interfaces 130 to enable weighting of various identification methods and the order of their engagement. Analysis of successful match rates for specific identification methods allows an administrator to manipulate the identifying criteria for each component to maximize the identification probability. A system according to the present invention preferably incorporates usage data from over 28 million users utilizing the CDDB database via Gracenote Data Services division, to help guide results 140.
  • [0038]
    The flexibility of a system according to the present invention allows different configurations to be used for identifying recordings in different environments. An application that monitors streaming audio, for example, requires a very different system and solution architecture than one that identifies files in a peer-to-peer system, or one that identifies analog input. However the present invention can be configured for identification of recordings in each of these situations.
  • [0039]
    A system according to the present invention maximizes identification while minimizing resource usage. The use of multiple identification methods ensures that more resource intensive methods, such as audio fingerprinting are employed only when necessary. The use of multiple audio fingerprinting technologies reduces data collision and covers any “blind spots” in a given audio fingerprint technology. The “blind spots” found in single source fingerprinting systems, are avoided by using multiple sources for different fingerprinting techniques. This also provides the ability to fine tune deployment for specific target applications.
  • [0040]
    Preferably, fingerprints are obtained using multiple fingerprint recognition services using the method illustrated in FIG. 2. This increases the ability of the system to accurately recognize recordings of various types.
  • [0041]
    As illustrated in FIG. 2, when unidentified (unknown) recording 100 is accessed by fingerprint extraction client 110, if possible conventional TOC/file recognition is performed by recognition system 210 and results 220 are returned to fingerprint client 110. Results 220 include a unique identifier (TUID) that points into a master metadata database (not shown in FIG. 2), if the TUID is found. Recording 100 is also processed by fingerprint extractor 230 using at least one and preferably several different algorithmically derived fingerprint extraction systems to obtain fingerprint(s) which are stored in fingerprint/ID send cache 240. As described below in more detail, instructions are received regarding when fingerprint uploader 250 should send the fingerprints to fingerprint recognition server 120.
  • [0042]
    In fingerprint recognition server 120, the fingerprints transmitted by fingerprint uploader 250 are initially stored in fingerprint receive cache 260. The fingerprints then undergo fingerprint validation 270 using an algorithmic comparator that attempts to cross-correlate fingerprints for a recording with fingerprints uploaded and extracted by different end users. If it is found that the fingerprints are substantially similar, they would be validated. This is not the only method that's available for validation, but serves as one example of a process that could be used to reject bad data.
  • [0043]
    In this embodiment, fingerprints that are determined to be valid and related undergo stitching 280. For example, if fingerprints are taken from 30 second segments of the recording, the fingerprints are assembled into a continuous fingerprint stream. This could simplify recognition of segments of the recording. The resulting fingerprints are stored in fingerprint database 290 associated with existing database 90 (FIG. 1).
  • [0044]
    The CDDB database has in part been generated through user submissions to create a metadata database with over 12 million tracks and 900,000 albums as of mid-2002. This database contains both basic metadata (artist, album, and track names) as well as extended data (genre, label, etc.).
  • [0045]
    A similar distributed collection method may be utilized in the creation of a waveform database using the system illustrated in FIG. 1. In the case where recording 100 is a raw audio waveform, e.g., when a CD is encoded into another format, such as an MP3 file, client device 110 obtains non-waveform data associated with recording 100 which is possessed by a user of database 90 and executes extraction algorithm(s) to extract fingerprints from at least one portion of the recording. The fingerprints are then sent to server 120 with a unique ID, preferably derived from the TOC of the CD. When the unique ID is available, i.e., , when a match is found in the database for the non-waveform data, server 120 is able to associate the appropriate metadata in database 90 and the fingerprint(s) with same level of accuracy as identification of CDs by the existing database 90 which is provided for identification of recordings possessed by users. Fingerprints dynamically gathered in this manner may be sent to a fingerprint collection server (not shown in FIG. 1) which would accumulate fingerprints from authenticated clients, as described in more detail below, prior to storing the at least one fingerprint as identifying information for the recording.
  • [0046]
    Multiple fingerprint gathering extractors can also be run over a set of static waveforms from a commercial encoder such as Loudeye or Muse. The challenge with this approach is associating the fingerprints with the appropriate metadata. The method described above enables audio fingerprints to be logically associated with parent records and associated back to the original audio source. In the preferred embodiment, the unique ID provides differentiation between live and studio versions of the same song while simultaneously linking those records to the same artist and their respective albums.
  • [0047]
    Preferably server(s) 120 store information in a parallel record set that are linked with unique IDs. When client 110 asks server 120 to recognize media (CD, digital audio file, video file) server 120 may also return a record about how fingerprints should be gathered for this particular CD. This is called the Gathering Instructions Record (GIR). The GIR may include a set of instructions that the remote fingerprint gathering code follows. The record may be pre-computed in off hours or may be dynamically computed at the time of recognition.
  • [0048]
    Server 120 may use information it knows about the popularity of a CD to drive decisions about gathering. Everything about a rare CD could be gathered, because the opportunity to get the fingerprints would not want to be missed (even if it was somewhat burdensome to the user). The opposite situation could be true for a very popular CD. The load may be distributed across many users so that they would not even notice that any work for fingerprint gathering was occurring.
  • [0049]
    The rules and procedures for building the GIR may be manual, automated and may change over time. They may also be applied uniquely to specific users, applications or geographic locations.
  • [0050]
    In one embodiment, the server dynamically gathers fingerprints by modifying the GIR to remove fingerprints that have been gathered previously. The frequency of updating GIRs may vary from instant to delays of days, weeks or months. Some example instructions that may be included in the GIR are:
  • [0051]
    A list of track and segments to be gathered and their priority.
  • [0052]
    A fingerprint generator algorithm to use.
  • [0053]
    Parameters that tell the fingerprint generator how to process the fingerprint, such as:
  • [0054]
    Frequency of audio samples
  • [0055]
    Bands of the frequency domain to process
  • [0056]
    Resolution of the fingerprint
  • [0057]
    Desired Quality of Audio
  • [0058]
    When to do the fingerprint gathering, such as
  • [0059]
    Before encoding the track
  • [0060]
    After encoding the track
  • [0061]
    In parallel with encoding the track
  • [0062]
    Instructions for caching the fingerprint and when to transmit it back to the server, such as
  • [0063]
    Before encoding the track
  • [0064]
    After encoding the track
  • [0065]
    After the CD has been fully encoded
  • [0066]
    When the communication channel back to the server is not busy
  • [0067]
    When the next CD is looked up
  • [0068]
    When a group of fingerprints is ready for transmission
  • [0069]
    Instructions to take CPU power into the process so as to not overload the computer
  • [0070]
    Preferably, the system attempts to improve the quality of the fingerprints during operation. Quality of the source signal, the parameters used for fingerprinting, along with improvements in the fingerprinting algorithms will result in a complex quality matrix that is used by server 120 to determine what fingerprints to gather if higher quality is available. An example of source quality is provided below: Preferably, database 90 or a similar database maintained by fingerprint collection server(s) stores the source quality for fingerprints stored in the database, so that when a fingerprint from higher quality source is available, the fingerprint may be replaced.
    Source Quality Table
    Name Bit Rate Compression Error Correction Quality Index
    CD_Audio_HEC 44100 kbps None Hardware 1
    CD_Audio_SEC 44100 kbps None Software 2
    CD_Audio 44100 kbps None None 3
    CDR_Audio 44100 kbps None None 4
    CDR_Made_From_MP3 44100 kbs mp3 None 5
    MP3_File  160 kbps mp3 None 6
  • [0071]
    Fingerprints dynamically gathered may contain information that helps validate quality. Information such as errors while reading from the media may be sent up to the fingerprint collector. The system may reject fingerprints that had high error rates from the source media.
  • [0072]
    As noted above, instead of immediately storing a fingerprint, multiple fingerprints for a recording may be gathered in by a fingerprint collection server prior to being added to the database. These fingerprints may be compared algorithmically to determine their correlation. If correlation is not adequate then additional fingerprints may be gathered until adequate correlation is achieved and one of the fingerprints or a composite fingerprint is stored in the database. This prevents bad fingerprints from becoming part of the database.
  • [0073]
    Stitching of the segmented fingerprints may be necessary since slight variations in timing could result in overlap of the fingerprints. Algorithmic stitching could result in a higher quality continuous fingerprint. Simple stitching appends segmented fingerprints in order of appearance in the recording. Complex stitching could involve scaling different qualities of fingerprints to the lowest common denominator and then appending them in order of their appearance in the recording. Preferably some form of mathematical fitting is utilized if the fingerprint segmentation contains jitter, so that appending is a fuzzy process rather simple addition of the datastream.
  • [0074]
    One example of audio fingerprinting that can be used is described in the U.S. patent application entitled Automatic Identification of Sound Recordings, filed by Maxwell Wells et al. on Jul. 22, 2002 and incorporated herein by reference. However, any known algorithmically derived fingerprinting technique may be used, not only for digital audio, but also video, TV programs (both analog and digital) and DVDs. Appropriate identifiers and recognition techniques will be used for the media to be recognized in a particular application.
  • [0075]
    The present invention provides great flexibility and can be utilized for a wide variety of environments, including MP3 recognition in a peer-to-peer environment, or identification of an audio stream for monitoring and reporting purposes. No other solution is known to use multiple recognition components; so it is the only solution that can be customized to meet the needs of any audio (or video) recognition application.
  • [0076]
    A functional description for a deployment of the present invention in a peer-to-peer application will be described below with reference to FIG. 3. In this embodiment, audio files are identified before providing public access to them, to determine if the files are allowed in the system, a process known as “filter-in”.
  • [0077]
    Client device 110 (FIG. 1) extracts information 310 (FIG. 3) from an audio file at the time of upload to server 120 (FIG. 1). The extracted information preferably includes non-waveform data, such as a unique ID, ID3 tag, filename text data, track duration, etc. and fingerprint(s) extracted from the recording and sent to server 120 for recognition.
  • [0078]
    The initial match 320 is performed against the unique ID, if present. Use of Gracenote's TUID enables a match to be returned with 99.9% accuracy. This is also the least resource intensive recognition method and can achieve very fast recognition rates. If the unique ID is present the system moves to the validation stage. If no unique ID is present the system attempts identification using the next recognition methods 330.
  • [0079]
    In this embodiment, text-based identification is tried next, using a metadata database, such as the Gracenote CDDB service which contains over 900,000 albums and over 12 million songs. Text matching utilizes available text, such as the filename, file path or text within the ID3 tag for MP3 files, to provide a set of data from which to attempt recognition. If an acceptable match is returned, the system moves to the validation stage. If a successful match is not returned, the system attempts identification utilizing the next recognition method.
  • [0080]
    The next step is fingerprint identification, in this case using audio fingerprints. The fingerprints from an unknown recording are compared to the fingerprints in database 90 for reference recordings, one fingerprint at a time (or in parallel using different processors for different fingerprints). Each fingerprinting technology returns a match and a level of confidence. If a single reference recording has acceptable confidence levels the system moves to the validation stage. If an unsuccessful match is returned the system can, depending on the target application, ask the user for validation of the most likely result or it can return a “no match found” result.
  • [0081]
    Validation is a key component to any successful recognition system. Preferably, key file attributes such as the duration of the recoding, are used to validate that a file is what the recognition system says it is by comparing an extracted length of the unknown recording with a stored length of the single reference recording.
  • [0082]
    Preferably heuristic and voting algorithms 340 are used to determine if a match is what the system says it is. This self-monitoring reduces the possibility that the system returns inaccurate data that pollutes the system. The heuristics may be manually controlled or algorithmically controlled to produce the best match. These heuristics may also be used to determine which recognition techniques to apply and in what sequence.
  • [0083]
    The administrator of each application can determine the level of accuracy needed by each stage (or component) of the system, and therefore has explicit control in optimizing the system. For example, if a 90% aggregate match is required the system administrator can use administrative interfaces 130 to adjust the levels of acceptable return to 90% and a successful result will not be generated unless that threshold is met. The administrator can also set result levels for each component. For example, a 99% text match can be required but only an 85% audio fingerprint match.
  • [0084]
    Once a successful identification is returned the file will be retagged 350 with the unique ID allowing for population of the file with the correct ID throughout the system. As a result, future identification of the file will require the least resource intensive recognition method.
  • [0085]
    The unique ID (TUID) assigned to the file is then matched 360 against a list 370 of TUIDs populated through the submission of Title/Artist pairs 370 by labels, publishers, and content owners of those files allowed in the system. In one embodiment, if the TUID is present in the database, the file is allowed to be shared, but if the TUID is not present in the database, the file is blocked. In another embodiment, if the TUID is present in the database, the file is blocked. Either of these embodiments could be applied to files recognized as they are accessed by a user, or transmitted from one computer to another.
  • [0086]
    As illustrated in FIG. 4A, an embodiment of the present invention uses a plurality of related databases. Master metadata database 410 contains information on title, artist/author name, owner name and date. Related databases include audio fingerprint database 430 and video fingerprint database 440 which form fingerprint database 290 (FIG. 2). Also included are track length/TOC database 450, text database 460, and hash ID database 470 and guaranteed unique ID database 480.
  • [0087]
    As illustrated in FIG. 4B, when unidentified (unknown) recording 100 is accessed by client device 110, information is extracted, including fingerprints 540, 550, metadata 560 and unique ID 570, if present. In addition, the duration 580 of the recording is determined and a numerical hash 590 is calculated. The extracted fingerprints are compared with fingerprints 600, 610. Similarly, matching 620, 630, 640 is performed on the numerical hash, text and unique ID. If a reference recording is located, validation is performed by comparing the duration of unidentified recording 100 with the duration of the reference recording. Results 660-710 with a level of confidence for each method of comparison is supplied to result aggregator 730.
  • [0088]
    If no reference recording is found 750 matching unidentified recording 100, the extracted information 540-590 and results are stored in unrecognized holding bin 760 for periodic resubmission to recognition server 120 (FIGS. 2 & 4B). In this embodiment, if a reference recording is located 770 with a low aggregate confidence level, post recognition processing 780 is performed by applying heuristics 790, or a manual review 810, e.g., by presenting one or more possible matches to the user and receiving the user's selection in response. The results of such user selections may be included in the heuristics stored in heuristics database 820. If post recognition processing 780 results in identification of a single reference recording or result aggregator 730 outputs recognized results 770 with a high aggregate confidence level, the hash ID is generated 810 and sent to hash database 480 and client device 110, so that the hash and unique ID (TUID) can be stored in the ID3 tag, if a file is being created.
  • [0089]
    In one embodiment, the system learns by watching errors in repeated attempts at recognition of similar files to improve its results. It also may receive manual stimulus from users who indicate that there are errors in the results. This allows recognition to be continuously validated over time. For example a file could be recognized by a system according to the invention, then over time the system determines that recognition of that file was flawed, and indicates to an operator that there was something wrong. In another embodiment, the system determines what is wrong by monitoring non-fingerprint based data and changing the recognition results accordingly.
  • [0090]
    The present invention can be utilized to identify any audio content for tracking purposes. Digital audio streams, analog inputs or local audio files, can all be tracked. Such a tracking system could be a server side tracking system deployed at the point of audio delivery and integrated with a reporting, digital rights management (DRM) system, or rights payment system. If the audio content being tracked was from a non-participating third party a client version of the system may be deployed to monitor the content being distributed. In either case, multiple identification methods would be utilized to ensure the highest rate of accuracy.
  • [0091]
    Utilizing waveform recognition as a digital rights management component is possible, and can be deployed to compare user created digital audio files with lists of approved content. This enables a filter-in approach within a peer-to-peer file sharing architecture such as the one described above.
  • [0092]
    Audio fingerprinting technologies can be used as an anti-piracy tool, and can be customized to the type of audio being investigated. In the case of pirated CDs, the Gracenote's CDDB CD service may be utilized to provide table of content (TOC) recognition to augment audio fingerprinting technologies.
  • [0093]
    Identification is the enabling component to deliver value-added services. Without explicit knowledge of the content being distributed it is impossible to distribute value-added content and services that relates to that audio content.
  • [0094]
    The many features and advantages of the invention are apparent from the detailed specification and, thus, it is intended by the appended claims to cover all such features and advantages of the invention that fall within the true spirit and scope of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope of the invention. For example the system and method have been described as using a unique identifier. However, a hashed identifier could be used instead.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4843562 *Jun 24, 1987Jun 27, 1989Broadcast Data Systems Limited PartnershipBroadcast information classification system and method
US5751672 *Jul 26, 1995May 12, 1998Sony CorporationCompact disc changer utilizing disc database
US5884298 *Dec 20, 1996Mar 16, 1999Cygnet Storage Solutions, Inc.Method for accessing and updating a library of optical discs
US5918223 *Jul 21, 1997Jun 29, 1999Muscle FishMethod and article of manufacture for content-based analysis, storage, retrieval, and segmentation of audio information
US5925843 *Feb 12, 1997Jul 20, 1999Virtual Music Entertainment, Inc.Song identification and synchronization
US5987525 *Apr 15, 1997Nov 16, 1999Cddb, Inc.Network delivery of interactive entertainment synchronized to playback of audio recordings
US6061680 *Jul 16, 1999May 9, 2000Cddb, Inc.Method and system for finding approximate matches in database
US6154773 *Apr 15, 1998Nov 28, 2000Cddb, Inc.Network delivery of interactive entertainment complementing audio recordings
US6161132 *Aug 24, 1999Dec 12, 2000Cddb, Inc.System for synchronizing playback of recordings and display by networked computer systems
US6230192 *Jul 16, 1999May 8, 2001Cddb, Inc.Method and system for accessing remote data based on playback of recordings
US6230207 *Jul 16, 1999May 8, 2001Cddb, Inc.Network delivery of interactive entertainment synchronized to playback of audio recordings
US6240459 *Jul 16, 1999May 29, 2001Cddb, Inc.Network delivery of interactive entertainment synchronized to playback of audio recordings
US6304523 *Jan 5, 1999Oct 16, 2001Openglobe, Inc.Playback device having text display and communication with remote database of titles
US6330593 *Aug 24, 1999Dec 11, 2001Cddb Inc.System for collecting use data related to playback of recordings
US6345256 *Dec 1, 1998Feb 5, 2002International Business Machines CorporationAutomated method and apparatus to package digital content for electronic distribution using the identity of the source content
US6400996 *Feb 1, 1999Jun 4, 2002Steven M. HoffbergAdaptive pattern recognition based control system and method
US6453252 *May 15, 2000Sep 17, 2002Creative Technology Ltd.Process for identifying audio content
US6505160 *May 2, 2000Jan 7, 2003Digimarc CorporationConnected audio and other media objects
US6829368 *Jan 24, 2001Dec 7, 2004Digimarc CorporationEstablishing and interacting with on-line media collections using identifiers in media signals
US6931451 *Mar 28, 2000Aug 16, 2005Gotuit Media Corp.Systems and methods for modifying broadcast programming
US6941275 *Oct 5, 2000Sep 6, 2005Remi SwierczekMusic identification system
US6990453 *Apr 20, 2001Jan 24, 2006Landmark Digital Services LlcSystem and methods for recognizing sound and music signals in high noise and distortion
US7174293 *Jul 13, 2001Feb 6, 2007Iceberg Industries LlcAudio identification system and method
US7194752 *Oct 19, 1999Mar 20, 2007Iceberg Industries, LlcMethod and apparatus for automatically recognizing input audio and/or video streams
US7302574 *Jun 21, 2001Nov 27, 2007Digimarc CorporationContent identifiers triggering corresponding responses through collaborative processing
US7328153 *Jul 22, 2002Feb 5, 2008Gracenote, Inc.Automatic identification of sound recordings
US7349552 *Jan 6, 2003Mar 25, 2008Digimarc CorporationConnected audio and other media objects
US7415129 *Jul 10, 2007Aug 19, 2008Digimarc CorporationProviding reports associated with video and audio content
US7461136 *Nov 2, 2005Dec 2, 2008Digimarc CorporationInternet linking from audio and image content
US7548851 *Oct 11, 2000Jun 16, 2009Jack LauDigital multimedia jukebox
US7587602 *Jan 11, 2006Sep 8, 2009Digimarc CorporationMethods and devices responsive to ambient audio
US7590259 *Oct 29, 2007Sep 15, 2009Digimarc CorporationDeriving attributes from images, audio or video to obtain metadata
US8468357 *Mar 9, 2010Jun 18, 2013Gracenote, Inc.Multiple step identification of recordings
US20010004338 *Oct 30, 1997Jun 21, 2001Sony Electronics Inc.Compact disc changer utilizing disc database
US20020033844 *Sep 11, 2001Mar 21, 2002Levy Kenneth L.Content sensitive connected content
US20020052885 *Sep 11, 2001May 2, 2002Levy Kenneth L.Using embedded data with file sharing
US20020059208 *Jul 26, 2001May 16, 2002Mototsugu AbeInformation providing apparatus and method, and recording medium
US20020083060 *Apr 20, 2001Jun 27, 2002Wang Avery Li-ChunSystem and methods for recognizing sound and music signals in high noise and distortion
US20020116195 *Feb 21, 2002Aug 22, 2002International Business Machines CorporationSystem for selling a product utilizing audio content identification
US20020133499 *Aug 20, 2001Sep 19, 2002Sean WardSystem and method for acoustic fingerprinting
US20030037010 *Apr 3, 2002Feb 20, 2003Audible Magic, Inc.Copyright detection and protection system and method
US20030061490 *Sep 23, 2002Mar 27, 2003Abajian Aram ChristianMethod for identifying copyright infringement violations by fingerprint detection
US20030105739 *Oct 11, 2002Jun 5, 2003Hassane EssafiMethod and a system for identifying and verifying the content of multimedia documents
US20040034650 *Aug 15, 2002Feb 19, 2004Microsoft CorporationMedia identifier registry
US20080052783 *Oct 26, 2007Feb 28, 2008Levy Kenneth LUsing object identifiers with content distribution
US20090158318 *Dec 9, 2008Jun 18, 2009Levy Kenneth LMedia Methods and Systems
US20100281545 *Nov 4, 2010Levy Kenneth LUsing Embedded Data with File Sharing
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US6973451Oct 10, 2003Dec 6, 2005Sony CorporationMedium content identification
US6993532May 30, 2001Jan 31, 2006Microsoft CorporationAuto playlist generator
US7024424Mar 11, 2005Apr 4, 2006Microsoft CorporationAuto playlist generator
US7082394Jun 25, 2002Jul 25, 2006Microsoft CorporationNoise-robust feature extraction using multi-layer principal component analysis
US7248715Sep 20, 2001Jul 24, 2007Digimarc CorporationDigitally watermarking physical media
US7269596 *Oct 17, 2003Sep 11, 2007Sony United Kingdom LimitedAudio and/or video generation apparatus
US7277766Oct 24, 2000Oct 2, 2007Moodlogic, Inc.Method and system for analyzing digital audio files
US7296031Mar 11, 2005Nov 13, 2007Microsoft CorporationAuto playlist generator
US7313571Oct 31, 2005Dec 25, 2007Microsoft CorporationAuto playlist generator
US7313591Jul 18, 2003Dec 25, 2007Microsoft CorporationMethods, computer readable mediums and systems for requesting, retrieving and delivering metadata pages
US7334023 *Mar 26, 2003Feb 19, 2008Kabushiki Kaisha ToshibaData transfer scheme for reducing network load using general purpose browser on client side
US7359900 *Jul 29, 2003Apr 15, 2008All Media Guide, LlcDigital audio track set recognition system
US7428572Sep 8, 2005Sep 23, 2008Microsoft CorporationTransferring metadata to a client
US7440975Dec 21, 2005Oct 21, 2008Musicgiants, Inc.Unified media collection system
US7451078Dec 30, 2004Nov 11, 2008All Media Guide, LlcMethods and apparatus for identifying media objects
US7477739Jan 21, 2003Jan 13, 2009Gracenote, Inc.Efficient storage of fingerprints
US7526506Sep 23, 2005Apr 28, 2009Microsoft CorporationInterlinking sports and television program listing metadata
US7548934Mar 30, 2006Jun 16, 2009Microsoft CorporationAuto playlist generator
US7549052Feb 11, 2002Jun 16, 2009Gracenote, Inc.Generating and matching hashes of multimedia content
US7549175 *Apr 17, 2007Jun 16, 2009Sony CorporationRecording medium, recording method, recording apparatus, reproduction apparatus, data transmission method, and server device
US7567899Dec 30, 2004Jul 28, 2009All Media Guide, LlcMethods and apparatus for audio recognition
US7574451 *Nov 2, 2004Aug 11, 2009Microsoft CorporationSystem and method for speeding up database lookups for multiple synchronized data streams
US7644077Oct 21, 2004Jan 5, 2010Microsoft CorporationMethods, computer readable mediums and systems for linking related data from at least two data sources based upon a scoring algorithm
US7647128Apr 22, 2005Jan 12, 2010Microsoft CorporationMethods, computer-readable media, and data structures for building an authoritative database of digital audio identifier elements and identifying media items
US7668059Jul 11, 2005Feb 23, 2010Sony CorporationCommercial/non-commercial medium test
US7672873Sep 10, 2004Mar 2, 2010Yahoo! Inc.Music purchasing and playing system and method
US7685210Dec 30, 2005Mar 23, 2010Microsoft CorporationMedia discovery and curation of playlists
US7706570Feb 9, 2009Apr 27, 2010Digimarc CorporationEncoding and decoding auxiliary signals
US7707221Apr 2, 2003Apr 27, 2010Yahoo! Inc.Associating and linking compact disc metadata
US7711564Jun 27, 2002May 4, 2010Digimarc CorporationConnected audio and other media objects
US7711838Nov 9, 2000May 4, 2010Yahoo! Inc.Internet radio and broadcast method
US7720852Jun 22, 2006May 18, 2010Yahoo! Inc.Information retrieval engine
US7747864Jun 29, 2006Jun 29, 2010Mircosoft CorporationDVD identification and managed copy authorization
US7761513May 18, 2004Jul 20, 2010Sony CorporationInformation recording device, information recording method, and information recording program
US7788684Oct 8, 2003Aug 31, 2010Verance CorporationMedia monitoring, management and information system
US7849131May 12, 2006Dec 7, 2010Gracenote, Inc.Method of enhancing rendering of a content item, client system and server system
US7853344Aug 16, 2007Dec 14, 2010Rovi Technologies CorporationMethod and system for analyzing ditigal audio files
US7877408Feb 5, 2008Jan 25, 2011Rovi Technologies CorporationDigital audio track set recognition system
US7890374Oct 24, 2000Feb 15, 2011Rovi Technologies CorporationSystem and method for presenting music to consumers
US7904503Aug 21, 2001Mar 8, 2011Gracenote, Inc.Method of enhancing rendering of content item, client system and server system
US7921296May 7, 2007Apr 5, 2011Gracenote, Inc.Generating and matching hashes of multimedia content
US7958485 *Nov 21, 2007Jun 7, 2011General Electric CompanyMethods and systems for managing content dependency deployment
US7979464Apr 11, 2007Jul 12, 2011Motion Picture Laboratories, Inc.Associating rights to multimedia content
US8005258Sep 25, 2009Aug 23, 2011Verance CorporationMethods and apparatus for enhancing the robustness of watermark extraction from digital host content
US8005724Mar 26, 2003Aug 23, 2011Yahoo! Inc.Relationship discovery engine
US8069255Jun 18, 2003Nov 29, 2011AT&T Intellectual Property I, .L.P.Apparatus and method for aggregating disparate storage on consumer electronics devices
US8121843Apr 23, 2007Feb 21, 2012Digimarc CorporationFingerprint methods and systems for media signals
US8131708 *Oct 30, 2008Mar 6, 2012Vobile, Inc.Methods and systems for monitoring and tracking videos on the internet
US8150096Mar 23, 2006Apr 3, 2012Digimarc CorporationVideo fingerprinting to identify video content
US8170273Apr 27, 2010May 1, 2012Digimarc CorporationEncoding and decoding auxiliary signals
US8259938Jun 19, 2009Sep 4, 2012Verance CorporationEfficient and secure forensic marking in compressed
US8271333Oct 30, 2001Sep 18, 2012Yahoo! Inc.Content-related wallpaper
US8280103Nov 19, 2010Oct 2, 2012Verance CorporationSystem reactions to the detection of embedded watermarks in a digital host content
US8316238Oct 25, 2006Nov 20, 2012Verizon Patent And Licensing Inc.Method and system for providing image processing to track digital information
US8340348Dec 25, 2012Verance CorporationMethods and apparatus for thwarting watermark detection circumvention
US8346567Aug 6, 2012Jan 1, 2013Verance CorporationEfficient and secure forensic marking in compressed domain
US8352259Jun 20, 2009Jan 8, 2013Rovi Technologies CorporationMethods and apparatus for audio recognition
US8352331Apr 30, 2001Jan 8, 2013Yahoo! Inc.Relationship discovery engine
US8451086Jan 30, 2012May 28, 2013Verance CorporationRemote control signaling using audio watermarks
US8458156 *May 18, 2012Jun 4, 2013Google Inc.Learning common spelling errors through content matching
US8468357Mar 9, 2010Jun 18, 2013Gracenote, Inc.Multiple step identification of recordings
US8483423Apr 20, 2006Jul 9, 2013Sony Pictures Entertainment Inc.Fingerprinting of data
US8490131 *Nov 5, 2009Jul 16, 2013Sony CorporationAutomatic capture of data for acquisition of metadata
US8495075 *Mar 8, 2006Jul 23, 2013Apple Inc.Fuzzy string matching of media meta-data
US8533481Nov 3, 2011Sep 10, 2013Verance CorporationExtraction of embedded watermarks from a host content based on extrapolation techniques
US8538066Sep 4, 2012Sep 17, 2013Verance CorporationAsymmetric watermark embedding/extraction
US8549307Aug 29, 2011Oct 1, 2013Verance CorporationForensic marking using a common customization function
US8595315Oct 19, 2011Nov 26, 2013At&T Intellectual Property I, L.P.Apparatus and method for aggregating disparate storage on consumer electronics devices
US8601504Jun 20, 2003Dec 3, 2013Verance CorporationSecure tracking system and method for video program content
US8615104Nov 3, 2011Dec 24, 2013Verance CorporationWatermark extraction based on tentative watermarks
US8615506 *Jan 27, 2012Dec 24, 2013Vobile, Inc.Methods and systems for monitoring and tracking videos on the internet
US8620967Jun 11, 2009Dec 31, 2013Rovi Technologies CorporationManaging metadata for occurrences of a recording
US8677400Sep 30, 2009Mar 18, 2014United Video Properties, Inc.Systems and methods for identifying audio content using an interactive media guidance application
US8681978Dec 17, 2012Mar 25, 2014Verance CorporationEfficient and secure forensic marking in compressed domain
US8682026Nov 3, 2011Mar 25, 2014Verance CorporationEfficient extraction of embedded watermarks in the presence of host content distortions
US8689337Feb 27, 2007Apr 1, 2014Vobile, Inc.Systems and methods of fingerprinting and identifying video objects
US8700641 *Aug 1, 2011Apr 15, 2014Google Inc.Detecting repeating content in broadcast media
US8726304Sep 13, 2012May 13, 2014Verance CorporationTime varying evaluation of multimedia content
US8738354Jun 19, 2009May 27, 2014Microsoft CorporationTrans-lingual representation of text documents
US8745403Nov 23, 2011Jun 3, 2014Verance CorporationEnhanced content management based on watermark extraction records
US8745404Nov 20, 2012Jun 3, 2014Verance CorporationPre-processed information embedding system
US8781967Jul 7, 2006Jul 15, 2014Verance CorporationWatermarking in an encrypted domain
US8791789May 24, 2013Jul 29, 2014Verance CorporationRemote control signaling using audio watermarks
US8806517May 10, 2010Aug 12, 2014Verance CorporationMedia monitoring, management and information system
US8811655Sep 4, 2012Aug 19, 2014Verance CorporationCircumvention of watermark analysis in a host content
US8838977Apr 5, 2011Sep 16, 2014Verance CorporationWatermark extraction and content screening in a networked environment
US8838978Apr 5, 2011Sep 16, 2014Verance CorporationContent access management using extracted watermark information
US8869222Sep 13, 2012Oct 21, 2014Verance CorporationSecond screen content
US8886531Jan 13, 2010Nov 11, 2014Rovi Technologies CorporationApparatus and method for generating an audio fingerprint and using a two-stage query
US8918382May 8, 2013Dec 23, 2014Google Inc.Learning common spelling errors through content matching
US8918428Mar 13, 2012Dec 23, 2014United Video Properties, Inc.Systems and methods for audio asset storage and management
US8923548Nov 3, 2011Dec 30, 2014Verance CorporationExtraction of embedded watermarks from a host content using a plurality of tentative watermarks
US8977067Apr 1, 2013Mar 10, 2015Google Inc.Audio identification using wavelet-based signatures
US9009482Sep 26, 2013Apr 14, 2015Verance CorporationForensic marking using a common customization function
US9055239Jul 19, 2007Jun 9, 2015Verance CorporationSignal continuity assessment using embedded watermarks
US9069771 *Dec 8, 2009Jun 30, 2015Xerox CorporationMusic recognition method and system based on socialized music server
US9106964Feb 8, 2013Aug 11, 2015Verance CorporationEnhanced content distribution using advertisements
US9117270Jun 2, 2014Aug 25, 2015Verance CorporationPre-processed information embedding system
US9153006Aug 15, 2014Oct 6, 2015Verance CorporationCircumvention of watermark analysis in a host content
US9189955Jul 28, 2014Nov 17, 2015Verance CorporationRemote control signaling using audio watermarks
US9208334Oct 25, 2013Dec 8, 2015Verance CorporationContent management using multiple abstraction layers
US9251549Jul 23, 2013Feb 2, 2016Verance CorporationWatermark extractor enhancements based on payload ranking
US20020023123 *Jul 26, 1999Feb 21, 2002Justin P. MadisonGeographic data locator
US20020028000 *Jun 21, 2001Mar 7, 2002Conwell William Y.Content identifiers triggering corresponding responses through collaborative processing
US20020095429 *Jan 14, 2002Jul 18, 2002Lg Electronics Inc.Method of generating digital item for an electronic commerce activities
US20020111993 *Feb 9, 2001Aug 15, 2002Reed Erik JamesSystem and method for detecting and verifying digitized content over a computer network
US20020126872 *Dec 19, 2001Sep 12, 2002Brunk Hugh L.Method, apparatus and programs for generating and utilizing content signatures
US20020146148 *Sep 20, 2001Oct 10, 2002Levy Kenneth L.Digitally watermarking physical media
US20020157099 *Jul 12, 2001Oct 24, 2002Schrader Joseph A.Enhanced television service
US20020157101 *Jul 12, 2001Oct 24, 2002Schrader Joseph A.System for creating and delivering enhanced television services
US20020178410 *Feb 11, 2002Nov 28, 2002Haitsma Jaap AndreGenerating and matching hashes of multimedia content
US20030021441 *Jun 27, 2002Jan 30, 2003Levy Kenneth L.Connected audio and other media objects
US20030046399 *Jul 10, 2001Mar 6, 2003Jeffrey BoulterOnline playback system with community bias
US20030061490 *Sep 23, 2002Mar 27, 2003Abajian Aram ChristianMethod for identifying copyright infringement violations by fingerprint detection
US20030167173 *Jan 6, 2003Sep 4, 2003Levy Kenneth L.Connected audio and other media objects
US20030174861 *Jan 6, 2003Sep 18, 2003Levy Kenneth L.Connected audio and other media objects
US20030187960 *Mar 26, 2003Oct 2, 2003Kabushiki Kaisha ToshibaData transfer scheme for reducing network load using general purpose browser on client side
US20030229537 *Mar 26, 2003Dec 11, 2003Dunning Ted E.Relationship discovery engine
US20030236661 *Jun 25, 2002Dec 25, 2003Chris BurgesSystem and method for noise-robust feature extraction
US20040009763 *Jun 20, 2003Jan 15, 2004Stone Chris L.Secure tracking system and method for video program content
US20040030725 *Jul 11, 2003Feb 12, 2004Pioneer CorporationInformation reproducing and recording apparatus, method for reproducing and recording information and information recorded medium
US20040034441 *Aug 16, 2002Feb 19, 2004Malcolm EatonSystem and method for creating an index of audio tracks
US20040073916 *Oct 8, 2003Apr 15, 2004Verance CorporationMedia monitoring, management and information system
US20040085342 *Oct 17, 2003May 6, 2004Williams Michael JohnAudio and/or video generation apparatus
US20040249859 *Mar 15, 2004Dec 9, 2004Relatable, LlcSystem and method for fingerprint based media recognition
US20050010671 *Jun 18, 2003Jan 13, 2005Sbc Knowledge Ventures, L.P.Apparatus and method for aggregating disparate storage on consumer electronics devices
US20050015551 *Jul 18, 2003Jan 20, 2005Microsoft CorporationMethods, computer readable mediums and systems for requesting, retrieving and delivering metadata pages
US20050027689 *Jul 29, 2003Feb 3, 2005Aec One Stop Group, Inc.Digital audio track set recognition system
US20050091268 *Dec 3, 2004Apr 28, 2005Meyer Joel R.Systems and methods of managing audio and other media
US20050187968 *Apr 28, 2005Aug 25, 2005Dunning Ted E.File splitting, scalable coding, and asynchronous transmission in streamed data transfer
US20050197906 *Sep 10, 2004Sep 8, 2005Kindig Bradley D.Music purchasing and playing system and method
US20050198061 *Feb 17, 2005Sep 8, 2005David RobinsonProcess and product for selectively processing data accesses
US20050229204 *Apr 22, 2003Oct 13, 2005Koninklijke Philips Electronics N.V.Signal processing method and arragement
US20050249075 *Jul 11, 2005Nov 10, 2005Laronne Shai ACommercial/non-commercial medium test
US20060020879 *Sep 8, 2005Jan 26, 2006Microsoft CorporationTransferring metadata to a client
US20060041753 *Aug 11, 2003Feb 23, 2006Koninklijke Philips Electronics N.V.Fingerprint extraction
US20060075237 *Oct 31, 2003Apr 6, 2006Koninklijke Philips Electronics N.V.Fingerprinting multimedia contents
US20060090020 *Oct 8, 2004Apr 27, 2006Time Trax Technologies CorporationConnector for satellite radio-computer interface
US20060106867 *Nov 2, 2004May 18, 2006Microsoft CorporationSystem and method for speeding up database lookups for multiple synchronized data streams
US20060136502 *Dec 21, 2005Jun 22, 2006Musicgiants, Inc.Unified media collection system
US20060149533 *Dec 30, 2004Jul 6, 2006Aec One Stop Group, Inc.Methods and Apparatus for Identifying Media Objects
US20060149552 *Dec 30, 2004Jul 6, 2006Aec One Stop Group, Inc.Methods and Apparatus for Audio Recognition
US20060156374 *Feb 14, 2003Jul 13, 2006Hu Carl CAutomatic synchronization of audio and video based media services of media content
US20060177096 *Apr 20, 2006Aug 10, 2006Sony Pictures Entertainment, Inc.Fingerprinting of Data
US20060229878 *May 27, 2004Oct 12, 2006Eric ScheirerWaveform recognition method and apparatus
US20060242193 *Jun 22, 2006Oct 26, 2006Dunning Ted EInformation retrieval engine
US20060242198 *Apr 22, 2005Oct 26, 2006Microsoft CorporationMethods, computer-readable media, and data structures for building an authoritative database of digital audio identifier elements and identifying media items
US20060253207 *Apr 22, 2005Nov 9, 2006Microsoft CorporationMethods, computer-readable media, and data structures for building an authoritative database of digital audio identifier elements and identifying media items
US20070050409 *Dec 16, 2005Mar 1, 2007Harris CorporationSystem, methods, and program product to trace content genealogy
US20070073649 *May 18, 2004Mar 29, 2007Hiroyuki KikkojiInformation recording device, information recording method, and information recording program
US20070078773 *Aug 31, 2006Apr 5, 2007Arik CzerniakPosting digital media
US20070106405 *Aug 21, 2006May 10, 2007Gracenote, Inc.Method and system to provide reference data for identification of digital content
US20070110089 *Nov 27, 2003May 17, 2007AdvestigoSystem for intercepting multimedia documents
US20070110272 *May 16, 2006May 17, 2007Sharma Ravi KEncoding and Decoding Signals for Digital Watermarking
US20070156268 *Nov 28, 2006Jul 5, 2007Galvin Brian MProviding audiographs through a web service
US20070168388 *Dec 30, 2005Jul 19, 2007Microsoft CorporationMedia discovery and curation of playlists
US20070195460 *Apr 17, 2007Aug 23, 2007Sony CorporationRecording medium, recording method, recording apparatus, reproduction apparatus, data transmission method, and server device
US20070214130 *Mar 8, 2006Sep 13, 2007Apple Computer, Inc.Fuzzy string matching of media meta-data
US20070239675 *Mar 29, 2006Oct 11, 2007Microsoft CorporationWeb search media service
US20070250716 *Apr 23, 2007Oct 25, 2007Brunk Hugh LFingerprinting of Media Signals
US20080002854 *Jul 19, 2007Jan 3, 2008Verance CorporationSignal continuity assessment using embedded watermarks
US20080005802 *Jun 29, 2006Jan 3, 2008Microsoft CorporationDVD identification and managed copy authorization
US20080027931 *Feb 27, 2007Jan 31, 2008Vobile, Inc.Systems and methods for publishing, searching, retrieving and binding metadata for a digital object
US20080040807 *Feb 27, 2007Feb 14, 2008Vobile, Inc.Systems and methods of fingerprinting and identifying digital versatile disc
US20080104404 *Oct 25, 2006May 1, 2008Mci, Llc.Method and system for providing image processing to track digital information
US20080126323 *Feb 5, 2008May 29, 2008Vladimir Askold BogdanovDigital audio track set recognition system
US20080133575 *Oct 17, 2007Jun 5, 2008Kabushiki Kaisha ToshibaAudio reproducing apparatus, metadata acquiring system for audio reproducing apparatus and metadata acquiring method for audio reproducing apparatus
US20080140702 *Oct 31, 2007Jun 12, 2008Iofy CorporationSystem and Method for Correlating a First Title with a Second Title
US20080183845 *Oct 16, 2007Jul 31, 2008Kabushiki Kaisha ToshibaData transfer scheme for reducing network load using general purpose browser on client side
US20080201201 *Sep 25, 2007Aug 21, 2008Sms.AcMethods and systems for finding, tagging, rating and suggesting content provided by networked application pods
US20080209502 *Apr 11, 2007Aug 28, 2008Seidel Craig HAssociating rights to multimedia content
US20080255685 *Mar 15, 2005Oct 16, 2008Fumio IsozakiAudio Information Output Apparatus, Audio Information Output Method, and Computer Product
US20080263360 *May 7, 2007Oct 23, 2008Gracenote, Inc.Generating and matching hashes of multimedia content
US20080274687 *May 2, 2007Nov 6, 2008Roberts Dale TDynamic mixed media package
US20080288365 *May 17, 2007Nov 20, 2008Fisher Iii William WMethods, media, and systems for payment determination
US20080288411 *May 17, 2007Nov 20, 2008Devon CopleyMethods, media, and systems for tracking and encrypting content usage
US20080288504 *May 17, 2007Nov 20, 2008Fisher Iii William WMethods, media, and systems for recording and reporting content usage
US20080288629 *May 17, 2007Nov 20, 2008Fisher Iii William WMethods, media, and systems for tracking content usage over a network
US20090106297 *Mar 13, 2008Apr 23, 2009David Howell WrightMethods and apparatus to create a media measurement reference database from a plurality of distributed sources
US20090132587 *Nov 21, 2007May 21, 2009David John SteinerMethods and systems for managing content dependency deployment
US20090133067 *Nov 19, 2007May 21, 2009Itay ShermanMulti-media enhancement channel
US20090259690 *Jun 20, 2009Oct 15, 2009All Media Guide, LlcMethods and apparatus for audio recognitiion
US20090324006 *Dec 31, 2009Jian LuMethods and systems for monitoring and tracking videos on the internet
US20100158488 *Mar 9, 2010Jun 24, 2010Gracenote, Inc.Multiple step identification of recordings
US20100161656 *Mar 9, 2010Jun 24, 2010Gracenote, Inc.Multiple step identification of recordings
US20100318586 *Jun 11, 2009Dec 16, 2010All Media Guide, LlcManaging metadata for occurrences of a recording
US20100322468 *Apr 27, 2010Dec 23, 2010Sharma Ravi KEncoding and Decoding Auxiliary Signals
US20100324883 *Jun 19, 2009Dec 23, 2010Microsoft CorporationTrans-lingual representation of text documents
US20110035035 *Feb 10, 2011Rovi Technologies CorporationMethod and system for analyzing digital audio files
US20110041154 *Feb 17, 2011All Media Guide, LlcContent Recognition and Synchronization on a Television or Consumer Electronics Device
US20110078020 *Sep 30, 2009Mar 31, 2011Lajoie DanSystems and methods for identifying popular audio assets
US20110078729 *Sep 30, 2009Mar 31, 2011Lajoie DanSystems and methods for identifying audio content using an interactive media guidance application
US20110102684 *May 5, 2011Nobukazu SugiyamaAutomatic capture of data for acquisition of metadata
US20110137855 *Dec 8, 2009Jun 9, 2011Xerox CorporationMusic recognition method and system based on socialized music server
US20110173185 *Jul 14, 2011Rovi Technologies CorporationMulti-stage lookup for rolling audio recognition
US20110231436 *Sep 22, 2011Seidel Craig HAssociating rights to multimedia content
US20120059845 *Aug 1, 2011Mar 8, 2012Google Inc.Detecting Repeating Content In Broadcast Media
US20120179666 *Jul 12, 2012Vobile, Inc.Methods and systems for monitoring and tracking videos on the internet
US20140282657 *Dec 27, 2013Sep 18, 2014Turner Broadcasting System, Inc.Method and system for automatic content recognition (acr) integration for smarttvs and mobile communication devices
CN1742492BFeb 14, 2003Jul 20, 2011汤姆森特许公司Automatic synchronization of audio and video based media services of media content
EP1595200A2 *Feb 20, 2004Nov 16, 2005Sony Electronics Inc.Medium content identification
EP1872199A2 *Mar 16, 2006Jan 2, 2008Microsoft CorporationMethods, computer-readable media, and data structures for building an authoritative database of digital audio identifier elements and identifying media items
EP2074527A2 *Oct 25, 2007Jul 1, 2009Verizon Business Global LLCMethod and system for providing image processing to track digital information
EP2811416A1 *Jun 6, 2013Dec 10, 2014Vestel Elektronik Sanayi ve Ticaret A.S.An identification method
WO2004017180A2 *Aug 18, 2003Feb 26, 2004Digital Innovations LlcSystem and method for creating an index of audio tracks
WO2006112843A1 *Apr 19, 2005Oct 26, 2006Sean WardDistributed acoustic fingerprint based recognition
WO2006115617A2Mar 16, 2006Nov 2, 2006Microsoft CorpMethods, computer-readable media, and data structures for building an authoritative database of digital audio identifier elements and identifying media items
WO2010071455A1 *Dec 15, 2009Jun 24, 2010Muller Montgomerie Media LimitedFile transfer method and apparatus
WO2011019473A1 *Jul 15, 2010Feb 17, 2011Rovi Technologies CorporationContent recognition and synchronization on a television or consumer electronics device
Classifications
U.S. Classification713/193, 707/E17.028, 707/E17.009
International ClassificationG10L15/10, G10L11/00, G10L15/00, G06F17/30
Cooperative ClassificationG06F17/30758, G06F17/30787, G06F17/30817, G06F17/30825, G06F17/30796, G06F17/30743
European ClassificationG06F17/30V2, G06F17/30V3E, G06F17/30V1T, G06F17/30V1A, G06F17/30U3E, G06F17/30U1
Legal Events
DateCodeEventDescription
May 13, 2004ASAssignment
Owner name: GRACENOTE, INC., CALIFORNIA
Free format text: CHANGE OF NAME;ASSIGNOR:CDDB, INC.;REEL/FRAME:015341/0243
Effective date: 20020625
Jun 13, 2008ASAssignment
Owner name: GRACENOTE, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ROBERTS, DALE T.;HYMAN, DAVID C.;WHITE, STEPHEN HELLING;REEL/FRAME:021097/0860;SIGNING DATES FROM 20080505 TO 20080605
Mar 19, 2014ASAssignment
Owner name: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT, IL
Free format text: SECURITY INTEREST;ASSIGNOR:GRACENOTE, INC.;REEL/FRAME:032480/0272
Effective date: 20140314