MUSIC STEERING WITH AUTOMATICALLY DETECTED MUSICAL ATTRIBUTES
Described is a technology by which a playback list comprising similar songs is automatically built based on automatically detected/generated song attributes, such as by extracting numeric features of each song. The attributes may be downloaded from a remote connection, and/or may be locally generated on the playback device. To build a playlist, a seed song's attributes may be compared against attributes of other songs to determine which other songs are similar to the seed song and thus included in the playlist. Another way to build a playlist is based on similarity of songs to a set of user provided-attributes, such as corresponding to moods or usage modes such as “resting” “reading” “jogging” or “driving” moods/modes. The playlist may be dynamically adjusted based on user interaction with the device, such as when a user skips a song, queues a song, or dequeues a song.
Latest Patents:
This application is a Continuation of U.S. application Ser. No. 14/171,770 (allowed), filed Feb. 3, 2014, which claims the benefit of U.S. application Ser. No. 12/132,621, filed Jun. 4, 2008, now U.S. Pat. No. 8,642,872, issued Feb. 4, 2014, which claims benefit to U.S. provisional application No. 61/033,065, filed Mar. 3, 2008. Each of the aforementioned applications are hereby incorporated by reference in their originally filed forms.
BACKGROUNDMany people have portable music devices such as an MP3 player, a Microsoft® Zune™ device, or other portable music players, and many of those devices contain hundreds or thousands of songs in their owners' personal music collections. With such large music collections, selecting songs to listen to becomes a challenge. Users want to listen to different songs at different times, but often do not have the time or inclination to repeatedly select songs to play from their personal collection. Also, it is difficult for users to select songs on the move with portable devices, such as when jogging or driving.
Current music players provide a “shuffle” function, which randomly selects songs for playback. This is a basic and simplistic solution that is very limited in its ability to satisfy users' requirements in changing songs. In general, the shuffle function provides users with no real control over what is played.
SUMMARYBriefly, various aspects of the subject matter described herein are directed towards a technology by which songs that are downloaded for playing on a music playing device have attributes automatically detected/generated for them, with those attributes used to select songs for recommended playback. For example, given a seed song, the attributes of the seed song may be compared against attributes of other songs to determine which of the other songs are similar to the seed song. Another way to build a playlist is based on similarity of songs to a set of user provided-attributes, such as corresponding to moods or usage modes such as “resting” “reading” “jogging” or “driving” moods/modes. Those songs which are deemed similar are built into a playlist, whereby a user has a subset of similar songs that are playable without needing to select each of those similar songs to play them. The playlist may be dynamically adjusted based on user interaction, such as when a user skips a song, queues a song, or dequeues a song.
The attributes may be automatically detected by extracting numeric features of the song. The attributes may be downloaded from a remote connection, such as provided in conjunction with the song, and/or may be locally generated on the playback device.
In one aspect, a music playback device is coupled to attribute detection logic that generates attributes of songs, and steering logic that uses the attributes to determine (or guide to) a set of similar songs to play on the playback device. The attribute detection logic may be incorporated into the playback device, and/or may be external to the playback device, and coupled thereto via communication means. Then, for example, upon receiving a song, the automatically generated attributes associated with that song may be used to build a playlist comprising the song and at least one other song having similar attributes.
The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Various aspects of the technology described herein are generally directed towards music “steering,” which in general provides users with a “smart” shuffle and/or selection mechanism in that songs are automatically selected for a playlist based upon their characteristics, such as with respect to their similarity to another song's characteristics. In general, this provides a user with an easy and simple way to control what is being played. Further, users can steer the selection process by providing feedback, whether directly or by taking actions such as to manually skip or dequeue a song, or queue a song.
In general, music steering provides interactive music playlist generation through music content analysis, music recommendation, and music filtering. In one example implementation, this is accomplished by automatically detecting some number (e.g. fifty) musical attributes (tags) from each song, by using the tags to build a music recommendation list based on matching the similarity of tags from various songs, and by incorporating implicit (or possibly explicit) feedback to update the recommendation list. As will be understood, because the tags are automatically generated, a more scalable mechanism that is more feasible for a personal music collection is provided.
While many of the examples described herein are generally directed towards a music player and a user's direct interaction with that music player, it is understood that these are only examples. For example, a user may interact with a personal computer or other device, which may playback songs and thus benefit directly from the technology described herein, or may transfer information to a music player such that the user indirectly interacts with the music player via the personal computer or the like. As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing and music playback in general.
As shown in
Turning to an aspect referred to as musical attributes detection, musical attributes detection is generally directed towards a way to measure the similarity between songs by automatically analyzing some number of categories (e.g., ten) that describe musical attributes (properties), which are then processed into a number of tags (e.g., fifty) associated with that song. In one example implementation, categories include genre, instruments, vocal, texture, production, tonality, rhythm, tempo, valence, and energy. Tags may be based on the categories, e.g., there are fifteen tags for “genre”, such as hard rock, folk, jazz and rap, twelve tags for “instruments”, such as acoustic guitar, acoustic piano, drum, and bass, three tags for “tempo”, including fast, moderato, and slow, and so forth. Some tags may have binary yes/no values, while others may be non-binary (e.g., continuous) values, such as “tempo in BPM” (beats per minute).
As described below, musical attributes are detected (which may alternatively be considered as tag generation) for each song, which becomes the basis for music similarity measurement, and thus music steering. This is represented in
In one example implementation generally exemplified by the flow diagram of
A next stage, referred to herein as learning, models each tag as a Gaussian Mixture Model (or GMM, a common technique in pattern recognition) using a set of training data and the extracted features. This is represented in
A third stage, referred to herein as automatic annotation, leverages GMM to select the most probable tag (step 308) from each category based on posterior; that is, in one example:
where x represents the features of a song, and ck is the k-th tag in one category. Note that while some tag categories are mutually exclusive (e.g., slow/fast), others are not (e.g., instrument). For mutually non-exclusive categories, tags with a probability above a threshold may be chosen; for some categories, e.g., instruments and genres, a slightly modified criterion may be used, in which multiple tags can be selected. In this example, the number of tags is determined as (in one example):
where th is a threshold.
The above provides a generic mechanism that has been found generally suitable for a large range of tags. For specific categories, steps 302-306 may be replaced by category-specific detectors. For example, the “tempo” attribute may be determined by using the well-known technique of repetitive-pattern analysis.
As represented at step 310, the most probable tags are then associated with the song in some appropriate way, which may be dependent on the data format of the song. For example, tags may be embedded into a song's data, appended to a song, included as header information, linked by an identifier, and so forth. Popular audio-file formats such as MP3 and WMA allow embedding of such meta-information. Such associations may be used for locally-generated or remotely-generated tags. A URL may be associated with the song, to provide access to the tags and/or other metadata; this allows improved tag sets to be used over time, such as whenever better tag generation algorithms are developed.
In the above example, each tag is assumed independent. However, some tags may be related, such as one indicating “hard rock” and another indicating “strong rhythm”; a “slow tempo” tag is typically related to a “weak rhythm” tag. Thus, further refinement may be performed to model such correlations, generally to improve the accuracy of tag annotations.
Turning to another aspect referred to herein as music steering, as mentioned above, music steering is generally directed towards interactive music playlist generation through music content analysis, music recommendation, and/or music filtering. To this end, as generally described with reference to
In the example flow diagram of
As can be seen in the example of
In one example implementation, the similarity between songs is measured based on the above obtained tags, in which each song is represented by a profile comprising a 50-dimensional feature vector indicating the presence or absence of each tag. The similarity between two songs may be measured, e.g., by cosine distance as is known in vector space models. A cosine distance may then be used as the weight. Other pattern recognition technologies (instead of a vector space model) may be used to determining similarity. For example, probabilistic models such as a GMM-based recommendation function may have trained parameters that are possibly refined based upon actual user data.
In the example of
At this time, a recommended song playlist is generated. Without user interaction, the player will play the recommended songs one by one. Users can also check the detected tags of each song, and check why these songs were considered similar by the system.
In the alternative example flow diagram of
In the example of
Turning to another aspect referred to as implicit relevance feedback, in general the system may continuously (or occasionally) refine the recommendation list based on user interactions. This may be dynamic, e.g., the playlist is updated as the user interacts, and/or after each session. Note that explicit feedback is straightforward, e.g., a user may manually edit a playlist to add and remove songs.
For interaction that indicates positive or negative feedback, implicit relevance feedback in the steering logic (e.g., 134, 234) operates to modify the playback list. In this example, if a user adds a song to a playback queue, it is considered positive feedback (e.g., for this session, as recorded into a session memory), as generally represented via steps 504 and 506. Conversely, if a user skipped a song that started to play, or dequeued a song, that is considered negative feedback (steps 508 and 510). Step 512 repeats the process for any other user interactions with this session; when the session is complete, the session memory is processed at step 514 based on the positive and negative feedback data therein to refine the recommendation list.
Other, more granular feedback is feasible. For example, a user can provide ratings for songs, although this requires additional user interaction. However, granularity may be provided via other data combined with implicit actions; for example the playlist may be modified when a user skips a song, but the extent of the modification may be based on how much of a song the user skips, e.g., a lesser modification is made if a user skips ahead after hearing most of a song, as opposed to a greater modification if the user skips it right away. Further, the age and/or popularity of the song may be a factor, e.g., if a user skips or dequeues a currently popular song, the user simply may be tired of hearing it right now; however if a user skips or dequeues an older or obscure (never very popular) song, the user may be indicating a more lasting dislike. Such popularity data may be downloaded like any other information.
Another aspect (which may be related to granularity of feedback) is that a playlist need not contain each song only once, but may instead contain certain songs more than once. The frequency of occurrence may be based on various data, such as how similar a song is to the seed song or user-provided (filtering) attributes, personal popularity, current popularity (personal or general public), user interaction, (e.g., queued songs get more frequency, skipped or dequeued songs get less), and so forth. A shuffling mechanism can ensure that a song is not replayed too often (e.g., in actual time and/or in relative order).
Users can preview the recommended songs and interact with them, such as (as represented in
where q is the profile of initial seed song, q′ is the updated profile, s+ and s are the songs of positive feedback and negative feedback, and λ is the updating rate.
Another aspect relates to mood-based filtering, such as in the form of a music sifter comprising of a set of sliders or the like by which a user may choose types of songs for a playlist. For example, if users have no specific song to choose as a seed song to generate a playlist (a music “radio station”), the user may set up a music sifter, such as via a set of sliders with each slider corresponding to one music attribute. For example, a “tempo” slider may be moved up or down to indicate a desire for songs with a fast, moderate or slow tempo (a set of radio buttons may be provided to submit a like indication). In addition to sliders, other user interface mechanisms such as checkboxes or the like may be used, e.g., a user can use checkboxes to individually select types of instruments.
Via filtering, users can set up (and persist) one or more such sifters, such as one for each certain mood. For example, when a user is reading, the user may choose to listen to some light music or some calmer-type songs, e.g., by filtering to obtain a slow tempo, as well as setting other. When jogging, a more energetic, rhythmic set of songs may be desired, with appropriate sifter settings chosen to that end. In addition to user-mood customized settings, common scenarios such as “read” “relax” “exercise” “driving” and so forth may be preloaded or downloaded to a device from a remote source, which are then used to find similarities with the user's personal collection. Users can share their “mood” setups in different scenarios.
Some music devices, such as Zune™, have a wireless connection. With a wireless connection, users can easily share their click-through data (e.g., the implicit-feedback data or any explicit feedback data) and/or their tag sets to improve (e.g., tune) the recommendation algorithms, and/or weights of the tags, which are used in the similarity measure between songs. Users can also share playlists or sets of tags (e.g., a user's mood-chosen tags) with other users. Any improved algorithm may be downloaded to the device, e.g., like a patch. Training data may be obtained based on actual user usage patterns. Note that users can also share their data through a wired connection, such as by uploading information through a personal computer.
Moreover, the music service may advertise music to users, such as by inserting new songs (e.g., temporarily, or only a portion thereof, or as a giveaway promotion such as by a new artist) not yet owned by a user into the recommended music playlist. In this way, the users can discover and purchase new songs they like. This is a non-intrusive form of advertising for products (songs) that are highly-relevant to the user, in order to allow a user to discover and purchase new music.
Turning to an example interface design,
With a touch screen device, the button-style design representing each song facilitates using a thumb for interaction, e.g., a leftward thumb movement may be used to move the song from the suggestion section 662 to the waiting queue 664, while a rightward movement will discard it. Based on these interactions, the songs in the recommended playlist will be correspondingly adjusted as described above. Note that for smaller and/or non-touch sensitive devices, a still meaningful use of this technology may be provided with only a single button—“skip”—or the like to allow the users to skip songs and correspondingly update the songs in the recommended playlist.
Other aspects are directed to the profiles, and include that profiles need not be yes/no groupings of attribute tags, but can instead have more granular values. For example, a tempo tag indicating fast may have a value indicating very fast, fast, or somewhat fast (but not quite moderate), or simply a numeric value denoting BPM (beats per minute). Further, the weights in the vector corresponding to the tags within a profile need not be the same, e.g., genre may have more weight than tempo when determining similarity. The weights may be initially trained using training data, and/or retrained based on user feedback.
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.
Claims
1. A method of selecting media files for a playlist, the method comprising:
- managing the playlist by providing, for potential addition of media items to the playlist, access to a set of media records, each media record comprising an association with a respective media item and a respective tag set comprised of attribute tags corresponding to media content of the associated media item, each attribute tag of each tag set indicating either an attribute of the corresponding media item and/or a value of an attribute of the corresponding media item;
- computing scores for at least some of the media items in the set of media items, respectively, by accessing tag weights for at least some of the attribute tags, respectively, and applying the accessed tag weights to relevant attribute tags of at least some of the media items;
- receiving indications of user interactions corresponding to control of playing at least some of the media items from the playlist, and, for each indication, computing tag weights for respective attribute tags of the correspondingly controlled media file;
- computing ranking scores for at least some of the respective media files, where a ranking score for a media file is computed as a function of whichever of the weights correspond to the attribute tags in the tag set of the corresponding media file; and
- selecting a media file to include in the playlist according to the ranking score thereof.
2. A method according to claim 1, wherein the indicated user interactions comprise interactively skipping or replaying at least some of the media items, and wherein the user interactions steer which media items are automatically included in the playlist by some types of interactions increasing tag weights of attribute tags of correspondingly controlled media files and by other types of interactions decreasing tag weights of attribute tags of correspondingly controlled media files.
3. A method according to claim 1, wherein the playlist initially comprises a seed media item, and some of the indications correspond to user interactions directed to media items automatically added to the playlist based on corresponding ranking scores computed according to the tag weights.
4. A method according to claim 1, wherein some of the tag sets have attribute tags not found in other of the tag sets, and wherein some of the tag sets have attribute tags in common with each other.
5. A method according to claim 4, wherein which attribute tags have respectively computed tag weights depends on which media items in the playlist have indications of interactions directed thereto.
6. A method according to claim 5, wherein the tag weights form a set of tags that is dynamically constructed according to cumulative user interactions with the playlist as indicated by the received indications.
7. A method performed by one or more computing devices, the method for managing a dynamic playlist, the method comprising:
- providing access to a library of playable media items, wherein ranking weights of respective media content attributes are maintained in association with the playlist, and wherein media items are selected from the library for inclusion in the playlist based on similarity scores computed by applying the ranking weights to media content attributes of the media items;
- playing the playlist while the playlist includes a first media item that is being played;
- while the first media item is being played from the playlist, receiving a first indication indicating a user interaction with the first media item in the playlist;
- based on the first indication being directed to the first media item in the playlist, updating whichever of the ranking weights are determined to correspond to media content attributes that are associated with the first media item;
- selecting a second media item to include in the playlist by applying the updated ranking weights to corresponding media content attributes of the second media item to compute a score for the second media item, and selecting the second media item based on the score for the second media item;
- while playing the playlist and while the second media item is in the playlist, receiving a second indication indicating a user interaction directed to the second media item in the playlist; and
- based on the second indication being directed to the second media item in the playlist, updating whichever of the ranking weights are determined to correspond to media content attributes that are associated with the second media item.
8. A method according to claim 7, wherein the interactions control selection or deselection of the media items for playing and/or control playing of the media items.
9. A method according to claim 8, wherein the indications indicate positive and negative interactions, wherein each indication indicating a positive interaction is handled by increasing corresponding ranking weights and each indication indicating a negative interaction is handled by decreasing corresponding ranking weights.
10. A method according to claim 9, wherein positive interactions include interactively adding a media item to the playlist.
11. A method according to claim 7, wherein playing the playlist comprises streaming the media items from one or server computers over a network.
12. A method according to claim 7, further comprising: after the ranking weights are updated according to the second indication, applying the ranking weights to select a third media item by applying the ranking weights to media content attributes of the third media item to determine a score for the third media item, wherein selection of the third media item depends on the updatings of the ranking weights according to the first and second indications.
13. A method according to claim 7, wherein a media content attribute associated with the first media item is not associated with the second media item, and a media content attribute of the second media item is not associated with the first media item.
14. A method according to claim 7, wherein one of the indications corresponds to skipping a portion of a media item, wherein the portion is not all of the media item, and wherein a ranking weight is computed in proportion with an extent of the portion that was skipped.
15. A method performed by one or more computing devices comprising processing hardware and storage hardware, the method comprising:
- accessing a set of media records stored by the storage hardware, each media record comprising an identifier of a corresponding media item and a plurality of content attributes representing attributes of content of the corresponding media item;
- receiving interaction records, each interaction record associated with a corresponding media item and each interaction record indicating a type of user interaction that occurred to control playing of the media item corresponding thereto;
- maintaining, by the processing hardware, a set of attribute weights each respectively associated with a different content attribute represented in the set of media records;
- updating, by the processing hardware, the set of weights according to the interaction records by, for each interaction record determined to indicate a first type of user interaction: increasing weights corresponding to content attributes in the media record of the media item that corresponds to the user interaction, and for each interaction record determined to indicate a second type of user interaction: decreasing weights corresponding content attributes in the media record of the media item that corresponds to the user interaction.
16. A method according to claim 15, further comprising computing similarity scores for some of the media records based on the set of attribute weights, where a score for a media record is computed by combining whichever of the attribute weights correspond to content attributes in the media record.
17. A method according to claim 15, further comprising computing measures of distance of the media records from the attribute weights.
18. A method according to claim 17, further comprising selecting for inclusion in the playlist a media record having a distance that is closest to the attribute weights.
19. A method according to claim 17, further comprising selecting for inclusion in the playlist a plurality of media records based on their respective distances and ordering the selected media records according to their respective distances.
20. A method according to claim 15, further comprising a filter to select which of the attribute weights are to be used to compute similarity scores for the media records.
Type: Application
Filed: May 25, 2016
Publication Date: Sep 15, 2016
Applicant:
Inventors: Lie Lu (Beijing), Frank Torsten Bernd Seide (Beijing), Gabriel White (San Francisco, CA)
Application Number: 15/164,818