Abstract: Techniques described and suggested herein include various systems and methods for using customer-initiated security tests to generate training data for use in improving detection and mitigation capabilities related network intrusion and data security attacks. Such techniques may include implementing machine learning techniques to refine security models used therewith. For example, customers of a computing resource provider may notify the computing resource provider that a security test is scheduled to occur. In response, in some embodiments, information related to the security test may be used to improve an implemented security model.