ADAPTIVE EVENT TIMELINE IN CONSUMER IMAGE COLLECTIONS
A method for organizing an event timeline for a digital image collection, includes using a processor for detecting events in the digital image collection and each event's associated timespan; determining the detected events that are significant in the digital image collection; and organizing the event timeline so that the event timeline shows the significant events and a clustered representation of the other events, made available to the user at different time granularities. The organized event timeline is also used for selecting images for generating output.
Latest Intellectual Ventures Fund 83 LLC Patents:
- Method for managing privacy of digital images
- Method for deleting data files in an electronic device
- Image display system related-information-access tabs
- System and method for providing customized product configuration and service plan options for a camera device
- Imaging device for capturing self-portrait images
This application is a Continuation of U.S. application Ser. No. 12/710,666, filed Feb. 23, 2010, incorporated herein by reference in its entirety.
FIELDThe disclosure relates generally to the field of digital image processing, and in particular to a method for organizing groups of digital images for efficient management and retrieval of events in consumer image collections.
BACKGROUNDThe proliferation of digital cameras, camera phones, and scanners has lead to an explosion of digital images and videos, creating large personal multimedia databases. Since taking digital pictures is easy and practically free, consumers no longer restrict picture-taking to important events and special occasions. Images are being captured frequently, and of day-to-day occurrences in the consumers' life. Since a typical user has already accumulated many years of digital images and videos, browsing the collection to find images and videos taken during particular events is a very time-consuming process for the consumer.
There has been work in grouping images into events. U.S. Pat. No. 6,606,411 and U.S. Pat. No. 6,351,556 disclose algorithms for clustering image content by temporal events and sub-events. The above two patents teach how to cluster images and videos in a digital image collection into temporal events and sub-events. The terms “event” and “sub-event” are used in an objective sense to indicate the products of a computer mediated procedure that attempts to match a user's subjective perceptions of specific occurrences (corresponding to events) and divisions of those occurrences (corresponding to sub-events). Another method of automatically organizing images into events is disclosed in U.S. Pat. No. 6,915,011. The events detected are chronologically ordered in a timeline from earliest to latest.
Using the above methods, the amount of browsing required by the user to locate a particular event can be reduced by viewing representatives of the events along a timeline, instead of each image thumbnail. However, a typical user can still generate over 100 of such events for each year, and more prolific picture-takers can easily exceed a few hundred detected events per year. There is still a need to create an overall event timeline structure that is adaptable to the user's picture taking behavior as well as the time granularity of the events in their collections, and the ability to select a sub-set of events that summarizes the overall collection or a given time period in the collection.
SUMMARYIn accordance with the present disclosure, there is provided a method for organizing an event timeline for a digital image collection comprising, using a processor for:
(a) detecting events in the digital image collection and each event's associated timespan;
(b) determining the detected events that are significant in the digital image collection; and
(c) organizing the event timeline so that the detected events on the event timeline between the detected significant events are grouped into clusters on the event timeline.
Since the number of images and videos in a typical user's collection is growing rapidly and each user might have a different picture-taking behavior, there is a critical need to provide the user efficient access and retrieval of images and important events from their collections. In this disclosure, a user's picture-taking behavior in terms of detected events is translated into a timeline, where there is a data point for each time step. Then an adaptive event-based timeline is created where both the significance of the detected events as well as the time granularity of the events are taken into account.
The organization and retrieval of images and videos is a problem for the typical consumer. It is useful for a user to be able to easily access and browse an overview of events in their image collection. Technology disclosed in prior art permits the classification of images in a collection into events, but not the ability to adaptively create an event timeline that can be tailored to individual users according to their picture-taking behaviors. As a result, there is a lack of effective mechanisms for finding and accessing events that are of importance to the users. This disclosure creates an effective event-based timeline at different time scales, where both the significance of the detected events as well as the time granularity of the events is taken into account. This organized event timeline can be used for selecting images and determining emphasis when creating outputs such as albums and slideshows from a collection of images.
The present disclosure can be implemented in computer systems as will be well known to those skilled in the art. In the following description, some embodiments of the present disclosure will be described as software programs. Those skilled in the art will readily recognize that the equivalent of such a method can also be constructed as hardware or software within the scope of the disclosure.
Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present disclosure. Other aspects of such algorithms and systems, and hardware or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein can be selected from such systems, algorithms, components, and elements known in the art. Given the description as set forth in the following specification, all software implementation thereof is conventional and within the ordinary skill in such arts. Videos in a collection are included in the term “images” in the rest of the description.
The present disclosure can be implemented in computer hardware and computerized equipment. For example, the method can be performed in a digital camera, a multimedia smart phone, a digital printer, on an internet server, on a kiosk, and on a personal computer. Referring to
It should also be noted that the present disclosure can be implemented in a combination of software or hardware and is not limited to devices which are physically connected or located within the same physical location. One or more of the devices illustrated in
Referring to
Referring to
Referring to
The method of U.S. Pat. No. 6,606,411 also provides a few key parameters for adjusting the number of events and sub-events generated from any given collection of images. The main parameter, “granularity”, determines the overall granularity of event clustering. Valid values are in the range [−1, 1] with 0 being the “optimal” tuned setting. Larger values increase the number of events found by the algorithm while smaller values decrease the number of events. The actual internal mechanism has to do with adjusting the various thresholds for the two-means clustering. Some of these thresholds include high and low thresholds for global histogram comparisons and high and low thresholds for multi-block histogram comparisons to determine whether to merge two events or two sub-events. The events detected continue to be chronologically ordered in a timeline from earliest to latest.
Referring to
Referring to
One of the challenges of creating an output album (hard copy) or slideshow (soft copy) from a collection of images is the selection of images to include and the emphasis to be placed on each of the selected images. Referring to
n=(significance factor*actual image count) for significant events
and n=actual image count for the rest of the (non-significant) events. Then the weight assigned to any event k is
and the number of images assigned to this event is the (weight*number of images specified by the output specifications). Note that fractional number of images assigned will need to be rounded to whole numbers, and therefore, some events can receive 0 images assigned to them (if they are assigned less than 0.5 images).
The selection of a given number of images from a specific event can be done simply based on sampling the event images at equal intervals. There are also more intelligent methods for performing this selection—taking into account image quality, diversity and specific semantic features such as people and objects present—which can be used in this step. For example, the selection can be based on the technical quality of the image (A. Loui, M. Wood, A. Scalise, and J. Birkelund, “Multidimensional image value assessment and rating for automated albuming and retrieval,” Proc. IEEE Intern. Conf. on Image Processing (ICIP), San Diego, Calif., Oct. 12-15, 2008) or aesthetic quality of an image (C. Cerosaletti, and A. Loui, “Measuring the perceived aesthetic quality of photographic images,” Proc. 1st International Workshop on Quality of Multimedia Experience (QoMEX), San Diego, Calif., Jul. 29-31, 2009). However, this is not a focus of the present disclosure which aims to provide a selection and weighting mechanism at the event-level.
The selected images are provided to an emphasis operator 825. The emphasis operator 825 generates emphasis scores to highlight particular images from the selected set. In one embodiment, images from non-significant events have the default emphasis score, whereas the selected images that belong to significant events are assigned emphasis scores by multiplying the default score by a significance factor greater than 1.0. The emphasis scores can be based on the score for the technical quality of the image (: A. Loui, M. Wood, A. Scalise, and J. Birkelund, “Multidimensional image value assessment and rating for automated albuming and retrieval,” Proc. IEEE Intern. Conf. on Image Processing (ICIP), San Diego, Calif., Oct. 12-15, 2008) or aesthetic quality of an image (C. Cerosaletti, and A. Loui, “Measuring the perceived aesthetic quality of photographic images,” Proc. 1st International Workshop on Quality of Multimedia Experience (QoMEX), San Diego, Calif., Jul. 29-31, 2009), or a combination of the two scores. The output creation module 850 uses the emphasis score and image selection to produce a creative presentation of the images. For hard copy (such as photo-books, calendars, cards), the emphasis score can control the size of the image in the output or the prominence of position in the page. For soft copy (such as a slideshow on digital photo frames, computer monitor or other display devices), the emphasis score can be used to assign screen time for the image i.e. images with higher emphasis scores will be displayed for longer duration. Higher emphasis scores can also provide triggers for highlighting such as decorative boxes around the image in both forms of presentation.
This disclosure has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the disclosure.
PARTS LIST
- 10 Computer
- 20 Processor
- 30 Memory
- 50 Input device
- 60 Keyboard
- 70 Display device
- 80 Network connection
- 105 Digital image collection
- 110 Event detector
- 115 Significant event detector
- 120 Event timeline organizer
- 150 Organized event timeline
- 215 Event detectors at different granularities
- 320 Event timeline organizer
- 350 Organized event timeline
- 415 Significant event detector at different granularities
- 420 Additional inputs
- 425 Time granularity selector
- 450 Organized event timeline
- 810 Organized event timeline
- 815 Output specifications
- 820 Media selection module
- 825 Emphasis operator
- 850 Output creation module
Claims
1. A method comprising:
- detecting, by a computing device, a plurality of events in a plurality of digital images and an associated timespan for each event of the plurality of events;
- generating a time series that indicates a number of images captured during a time period covered by the digital image collection;
- identifying a set of events by analyzing the time series using time-series modeling; and
- organizing, by the computing device, the plurality of events into an event timeline, such that detected events on an event timeline between the determined set of events are grouped into clusters on the event timeline.
2. The method of claim 1, wherein the detecting the plurality of events comprises detecting the plurality of events based on a selected time granularity.
3. The method of claim 2, wherein selecting the granularity of event detection is based on choosing the fewest detected events that preserve the significant events during a given time period.
4. The method of claim 1, further comprising enabling user browsing of images associated with at least one detected event of the plurality of events.
5. The method of claim 1, further comprising:
- presenting a display of the event timeline; and
- receiving a user selection of the event timeline.
6. The method of claim 4, further comprising enabling user browsing of images associated with at least one detected event of the plurality of events for the selected first event timeline.
7. The method of claim 1, further comprising:
- producing an emphasis score using an emphasis operator; and
- controlling a presentation of an output of images based on the emphasis score.
8. The method of claim 1, further comprising analyzing the time series to find a model that identifies the significant events.
9. The method of claim 8, wherein the identifying comprises determining when the model is less able to predict actual data of the time series.
10. A system comprising:
- a memory; and
- a processor coupled to the memory, wherein the processor is configured to: detect a plurality of events in a plurality of digital images and an associated timespan for each event of the plurality of events; generate a time series that indicates a number of images captured during a time period covered by the digital image collection; identify a set of events by analyzing the time series using time-series modeling; and organize the plurality of events into an event timeline, such that detected events on an event timeline between the determined set of events are grouped into clusters on the event timeline.
11. The system of claim 10, wherein the detection of the plurality of events comprises a detection of the plurality of events based on a selected time granularity.
12. The system of claim 11, wherein the selection of the granularity of event detection is based on choosing the fewest detected events that preserve the significant events during a given time period.
13. The system of claim 10, wherein the processor is further configured to enable user browsing of images associated with at least one detected event of the plurality of events.
14. The system of claim 10, wherein the processor is further configured to:
- present a display of the event timeline; and
- receive a user selection of the event timeline.
15. The system of claim 10, wherein the processor is further configured to:
- produce an emphasis score using an emphasis operator; and
- control a presentation of an output of images based on the emphasis score.
16. A non-transitory computer-readable medium having instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations comprising:
- detecting a plurality of events in a plurality of digital images and an associated timespan for each event of the plurality of events;
- generating a time series that indicates a number of images captured during a time period covered by the digital image collection;
- identifying a set of events by analyzing the time series using time-series modeling; and
- organizing the plurality of events into an event timeline, such that detected events on an event timeline between the determined set of events are grouped into clusters on the event timeline.
17. The computer-readable medium of claim 16, wherein the detecting the plurality of events comprises detecting the plurality of events based on a selected time granularity.
18. The computer-readable medium of claim 17, wherein selecting the granularity of event detection is based on choosing the fewest detected events that preserve the significant events during a given time period.
19. The computer-readable medium of claim 16, further comprising enabling user browsing of images associated with at least one detected event of the plurality of events.
20. The computer-readable medium of claim 16, further comprising:
- presenting a display of the event timeline; and
- receiving a user selection of the event timeline.
Type: Application
Filed: Jan 22, 2014
Publication Date: May 15, 2014
Applicant: Intellectual Ventures Fund 83 LLC (Las Vegas, NV)
Inventors: Madirakshi Das (Penfield, NY), Alexander C. Loui (Penfield, NY)
Application Number: 14/160,680