Abstract: A system for providing code suggestions according to licensing criteria is described. The system comprises computing devices that implement a code suggestion service. The code suggestion service receives a request that specifies licensing criteria via an interface of the code suggestion service. The code suggestion service determines respective licenses for respective source code files according to a source code attribution database from parsing the plurality of source code files that are applicable to the plurality of source code files. The code suggestion service generates a set of candidate code suggestions based, at least in part, on the plurality of source code files. The code suggestion service determines code suggestions from the set of candidate code suggestions that satisfy the licensing criteria based on the respective licenses. The code suggestion service provides the code suggestions determined from the set of candidate source code files that satisfy the licensing criteria.

Abstract: Evaluation data sets may be programmatically generated for code generation models. An evaluation data set is obtained that includes items that correspond to different evaluation tests for a code generation system. The individual items of the evaluation data set maybe converted, including the conversion of a function signature for the items, the test statements for the items and using a code generation system to generate the body of the function.