Abstract: A system is provided for determining when supplemental content unresponsive to a user input is to be presented during a user interaction. The system determines an action, performance of which, may trigger output of supplemental content. The system may determine when, during a user interaction corresponding to performance of the action, supplemental content may be presented. The system may use constraint data indicating a device type via which the content may be presented, and a time duration during which the supplemental content may be presented. If the constraint data is satisfied, the system may determine to present the supplemental content.

Abstract: Techniques for performing tests by a speech processing system are described. A speech processing system may apply a hash function to a group of user identifiers, a content provider identifier, and time data to segment the group of user identifiers into non-holdout, holdout control, and holdout treatment groups. A non-holdout group represents users not subject to a test for a duration of time. A holdout control group represents users subject to a test for the duration of time, but which do not receive unresponsive content during the duration of time. A holdout treatment group represents users subject to a test and which receive unresponsive content over the duration of time. User activity of the holdout control and holdout treatment groups may be compared to measure an effectiveness of the unresponsive content. The measured effectiveness may be used to determine a ranking of the content provider at runtime.