Abstract: A system for providing code suggestions according to licensing criteria is described. The system comprises computing devices that implement a code suggestion service. The code suggestion service receives a request that specifies licensing criteria via an interface of the code suggestion service. The code suggestion service determines respective licenses for respective source code files according to a source code attribution database from parsing the plurality of source code files that are applicable to the plurality of source code files. The code suggestion service generates a set of candidate code suggestions based, at least in part, on the plurality of source code files. The code suggestion service determines code suggestions from the set of candidate code suggestions that satisfy the licensing criteria based on the respective licenses. The code suggestion service provides the code suggestions determined from the set of candidate source code files that satisfy the licensing criteria.

Abstract: Evaluation data sets may be programmatically generated for code generation models. An evaluation data set is obtained that includes items that correspond to different evaluation tests for a code generation system. The individual items of the evaluation data set maybe converted, including the conversion of a function signature for the items, the test statements for the items and using a code generation system to generate the body of the function.

Abstract: Pre-fix matching may constrain the generation of next token predictions. Input text to perform a next token prediction may be received. Multiple tokens may be determined from the input text, including a partial token. From possible tokens, one or more matching possible tokens with the partial token may be identified. Next token predictions may then be filtered using the identified possible tokens in order to ensure that the partial token is matched.