Abstract: The invention relates to a method and system for annotating image regions with specific concepts based on multimodal user input. The system (10) comprises an identification unit (11) for the identification of a region of interest on a multidimensional image; an automatic speech recognition unit (12) for recognizing speech input in a natural language; a natural language understanding unit (13) which interprets the speech input in the context of a specific application domain; a fusion unit (14) which combines the multimodal user input from the identification unit (11) and the natural language understanding unit (13); and an annotation unit (15) which annotates the result of the natural language understanding unit (13) on the image regions and optionally provides user feedback about the annotation process. Thus, the system advantageously facilitates a user's task to annotate specific image regions with standardized key concepts based on multimodal speech-based user input.