Abstract: Once a technician detects and reports an equipment failure, diagnostic systems and methods herein automatically generate a diagnostic plan, detailing necessary parts, areas of interest, and actions to be taken. Using augmented reality (AR) indicators, a system guides the technician while a perception module analyzes their actions to recommend next steps. A planning component uses knowledge graph (KG) and LLM to create the diagnostic plan, without requiring extensive data labeling or model training. A tracking component enhances 3D detections by employing perception sensors and a 2D nested object detection model. A guiding component simplified processed for novice technicians by integrating 2D models and AR interactions to ensure efficient and accurate diagnostics.

Abstract: Example implementations involve systems and methods to advance data acquisition systems for automated visual inspection using a mobile camera infrastructure. The example implementations address the uncertainty of localization and navigation under semi-controlled environments. The approach combines object detection models and navigation planning to control the quality of visual inputs in the inspection process. The solution guides the operator (human or robot) to collect only valid viewpoints to achieve higher accuracy. Finally, the learning models and navigation planning are generalized to multiple type and size of inspection objects.