Abstract: A facility for generating 3D models for mixed reality applications using generative artificial intelligence displays a mixed reality (MR) development environment. The facility obtains input describing a subject of the 3D model to be generated. The input may be a textual description of the subject or an image depicting the subject. The facility determines a target MR device with which the 3D model is to be displayed. The facility establishes a prompt based on the model description and the target device, and submits the prompt to a generative artificial intelligence (AI) model. The facility receives a 3D model from the generative AI model and displays the 3D model in the mixed reality development environment. Based on the mixed reality development environment, the facility creates a mixed reality application executable by the target MR device to display the 3D model.

Abstract: A facility for providing an XR device-based tool for cross-platform content creation and display is disclosed. The facility causes a definition of a mixed reality (MR) procedure to be presented by an editing MR device in a virtual application window. Input is received via the editing MR device. The facility interprets the received input as one or more editing actions performed with respect to the MR procedure definition and alters the MR procedure definition accordingly. The facility creates a device-independent representation of the MR procedure based on the altered MR procedure definition.

Abstract: The facility provides an environment for creating a mixed reality (MR) application. Input specifying a property of a networked device is to be accessible in the mixed reality application is received. Properties of one or more networked devices are identified. A property selection interface is provided in the environment for creating the mixed reality application. The facility receives, via the property selection interface, selection of a networked device of the one or more networked devices and a property of the networked device. The facility configures the MR application to provide, to a device executing the MR application, access to the selected property of the selected networked device. The MR device can execute the MR application to access the selected property of the selected networked device.

Abstract: A portion of a source mixed reality (MR) experience is retrieved. Then, the portion of the source MR experience is used to generate a serialized representation including a hierarchy of tagged elements. The hierarchy of tagged elements includes a plurality of MR step elements collectively defining a procedure to be performed by a viewer of an MR experience. Each MR step element has child elements that include an MR step number indicating a position of the MR step in the MR procedure and an MR step ID element indicating an identity of the MR step. The serialized representation is deserialized to generate a portion of a target MR experience to be edited in an MR development tool. The portion of the target MR experience is usable to cause each MR step in the plurality of MR steps to be graphically represented in the MR development tool.

Abstract: A facility for providing an XR device-based tool for cross-platform content creation and display is disclosed. The facility causes a definition of a mixed reality (MR) procedure to be presented by an editing MR device in a virtual application window. Input is received via the editing MR device. The facility interprets the received input as one or more editing actions performed with respect to the MR procedure definition and alters the MR procedure definition accordingly. The facility creates a device-independent representation of the MR procedure based on the altered MR procedure definition.

Abstract: A facility for assisting a user in a mixed reality (MR) experience using generative artificial intelligence is disclosed. A user query and MR data associated with the MR experience is received and used to create a prompt to provide to a generative artificial intelligence (GAI) model. In some embodiments, the MR data includes source data, procedure data that is based on the source data and corresponds to a plurality of MR steps of the MR experience, and a current step indicator that corresponds to a current MR step that is being provided to the user. The facility provides the prompt to the AI model. Then, the facility creates content based on output of the AI model and causes the content to be provided to the user in the MR experience, such as by outputting it audibly to the user via simulated speech.

Abstract: A directive template that includes sample directive code for a directive to be generated and natural language annotation is retrieved. Then, a natural language statement specifying a functionality for the directive in a mixed reality experience is received from a user. A prompt is generated based on the directive template and the statement and is submitted to an artificial intelligence model. A response from the artificial intelligence model is received that includes a proposed directive providing the specified functionality. An attempt to translate the proposed directive into machine code is made. If the translation is successful, the proposed directive is added to a source code directory accessible for execution in a visual programming interface for mixed reality experiences. An indication of the directive is displayed in the visual programming interface for mixed reality experiences, enabling the directive to be incorporated into MR experiences.

Abstract: Systems and methods for generating structured data for mixed reality applications using generative artificial intelligence are disclosed. Unstructured data including instructional content is received. A schema to which the unstructured data is to conform is established. Then, a system task prompt based on the unstructured instructional content and the schema input is established. The system task prompt is provided to a generative artificial intelligence model. Structured data is received from the generative artificial intelligence model. A procedure usable in a mixed reality application is constructed using the structured data.

Abstract: A facility for providing an XR device-based tool for cross-platform content creation and display is disclosed. The facility causes a definition of a mixed reality (MR) procedure to be presented by an editing MR device in a virtual application window. Input is received via the editing MR device. The facility interprets the received input as one or more editing actions performed with respect to the MR procedure definition and alters the MR procedure definition accordingly. The facility creates a device-independent representation of the MR procedure based on the altered MR procedure definition.

Abstract: A portion of a source mixed reality (MR) experience is retrieved. Then, the portion of the source MR experience is used to generate a serialized representation including a hierarchy of tagged elements. The hierarchy of tagged elements includes a plurality of MR step elements collectively defining a procedure to be performed by a viewer of an MR experience. Each MR step element has child elements that include an MR step number indicating a position of the MR step in the MR procedure and an MR step ID element indicating an identity of the MR step. The serialized representation is deserialized to generate a portion of a target MR experience to be edited in an MR development tool. The portion of the target MR experience is usable to cause each MR step in the plurality of MR steps to be graphically represented in the MR development tool.

Abstract: Techniques for developing mixed reality applications in connection with a virtual development environment are disclosed. A reference model corresponding to a physical object is received. Then, the reference model is displayed in a virtual development environment. First input arranging an anchor with the reference model in the virtual development environment is received. Then, second input specifying an animation to apply to the reference model is received. The arrangement in the virtual development environment is used to construct a mixed reality application usable to display, in response to detecting an instance of the anchor in a physical environment, the animated reference model in accordance with the arrangement in the virtual development environment.