Abstract: A method, and non-transitory computer readable medium that perform a method for generating background music tailored for a movie scene, in a smart audio-visual display device. The method includes receiving video of the movie scene. Processing circuitry detects speech signals in the movie scene and extracts visual information and, when speech signals are detected, spoken dialogue from the movie scene. Descriptive text is generated from the visual information. Emotion categories are detected based on the visual information. The spoken dialogue is transcribed into transcribed text. A large language model (LLM) translates the descriptive text, emotion categories and transcribed text into text-based low-level musical instrument conditions. A text-to-music model is guided, by the low-level musical instrument conditions, to generate audio tokens that resonate with the movie scene. Music signals are output in accordance with the audio tokens in synchronism with the movie scene.

Abstract: A method, and non-transitory computer readable medium that perform a method for generating background music tailored for a movie scene, in a smart audio-visual display device. The method includes receiving video of the movie scene. Processing circuitry detects speech signals in the movie scene and extracts visual information and, when speech signals are detected, spoken dialogue from the movie scene. Descriptive text is generated from the visual information. Emotion categories are detected based on the visual information. The spoken dialogue is transcribed into transcribed text. A large language model (LLM) translates the descriptive text, emotion categories and transcribed text into text-based low-level musical instrument conditions. A text-to-music model is guided, by the low-level musical instrument conditions, to generate audio tokens that resonate with the movie scene. Music signals are output in accordance with the audio tokens in synchronism with the movie scene.