Abstract: Open-source software is prevalent in the development of new technologies. Monitoring software updates for vulnerabilities is expensive and time consuming. Online discussions surrounding new software updates can often provide vital information regarding emerging risks. It is presented a novel approach for automating surveillance of software through the use of natural language processing methods on open-source issues. Further, the potential of virtual adversarial training, a popular semi-supervised learning technique, is used to leverage the vast amounts of unlabeled data available to achieve improved performance. On industry data, it is found that a hierarchical attention network with virtual adversarial training that utilizes the innate document structure to encapsulate the text can be used with good results.