Abstract: The disclosed architecture enables user feedback in the form of gestures, and optionally, voice signals, of one or more users, to interact with a search engine framework. For example, document relevance, document ranking, and output of the search engine can be modified based on the capture and interpretation of physical gestures of a user. The recognition of a specific gesture is detected based on the physical location and movement of the joints of a user. The architecture captures emotive responses while navigating the voice-driven and gesture-driven interface, and indicates that appropriate feedback has been captured. The feedback can be used to alter the search query, personalize the response using the feedback collected through the search/browsing session, modifying result ranking, navigation of the user interface, modification of the entire result page, etc., among many others.