Abstract: Using indentation to trim output of a language synthesis model. A computer system determines a first indentation level associated with a cursor within an editor buffer. The computer system identifies a block of line(s) from an output from a language synthesis model. Each line in the block meets at least one criterion from a set of criteria, including a first criterion that the line is an initial line in the output from the language synthesis model, a second criterion that the line has a second indentation level that is greater than the first indentation level, and a third criterion that the line contains a continuation token. The computer system identifies a particular line, immediately following the block, that does not meet any criterion from the set of criteria. The computer system trims the output from the language synthesis model from the particular line onward.

Abstract: A computer system identifies a code synthesis model and a set of software repositories, each comprising corresponding source code and corresponding tests for testing the corresponding source code. The computer system tests a plurality of generated source code predictions using the set of software repositories. The testing includes, for each software repository in the set of software repositories, identifying a portion of the corresponding source code of the software repository that is covered by the corresponding tests of the software repository, using the code synthesis model to generate a source code prediction for the portion of the corresponding source code, and using the corresponding tests of the software repository to test the source code prediction that was generated using the code synthesis model. Based on testing the plurality of generated source code predictions using the set of software repositories, the computer system generates an evaluation of the code synthesis model.

Abstract: A computer system is configured to identify a position of a cursor in an editor where a code file is displayed and identify one or more wish items based on the position of the cursor or metadata associated with the code file. The computer system is further configured to identify one or more first portions of text from the one or more wish items that are relevant to text immediately preceding the cursor, and prioritize the one or more first portions of text to identify a particular first portion of text that is most relevant to the text immediately preceding the cursor. The computer system then generates a second portion of text based on the particular first portion of text and suggest that the second portion of text be entered at the cursor.