Abstract: Optimizing augmented reality scenes by using depth information to accurately display interactions between real objects and synthetic objects is described. A stream of depth data associated with a real scene of an augmented reality display and a stream of color data associated with the real scene may be received. The stream of depth data may be processed to construct a first mesh and the first mesh may be projected into a color space associated with the stream of color data to construct a second mesh. In some examples, a position of the synthetic objects respective to real objects in the real scene may be determined and/or queries may be conducted to determine how the synthetic objects interact with the real objects in the real scene. Based at least on constructing the second mesh, determining positions, and/or conducting queries, one or more synthetic objects may be drawn into the real scene.