Abstract: Systems and methods for answering queries by applying deductive reasoning using a data structure based on knowledge derived from sensory data are provided. A method may include acquiring multimodal sensory data and extracting a plurality of salient features of the one or more objects. Data structures that associate one or more essential characteristics with each of the plurality of salient features may be created and the spatial relationships among the one or more objects in visual scenes may be mapped. An ontology that encompasses the typical relationships among the classes of objects in a plurality of datasets may be created and one or more axioms may be established.