Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a sequence of data elements that includes a respective data element at each position in a sequence of positions. In one aspect, a method includes: for each position after a first position in the sequence of positions: obtaining a current sequence of data element embeddings that includes a respective data element embedding of each data element at a position that precedes the current position, obtaining a sequence of latent embeddings, and processing: (i) the current sequence of data element embeddings, and (ii) the sequence of latent embeddings, using a neural network to generate the data element at the current position. The neural network includes a sequence of neural network blocks including: (i) a cross-attention block, (ii) one or more self-attention blocks, and (iii) an output block.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for executing depth-parallel training of a neural network. One of the methods includes receiving an input sequence; and at each processing time step in a sequence of processing time steps: processing an input item using a first layer block in a stack of layer blocks to generate a first block output; for each subsequent layer block, processing a block output generated by the preceding layer block at the preceding processing time step to generate a current block output; computing i) a current error in an output item generated by the final layer block and ii) a current gradient of the current error; generating a parameter update for the final layer block; for each particular layer block that is not the final layer block, computing a current gradient for the particular layer block and generating a parameter update.

Abstract: Identification tags are used to distinguish a unique tagged item from among a plurality of items. Tagged items are searched by scanning the items with an interrogation signal. The tag emits an observable signal when it receives an identification that matches the identification contained in the tag. When an interrogation signal is not present, the tags sleep at a very low power level or are passively unpowered. A quasi-passive wake-up function employs a continuous low power standby mode, comparing a received signal to a predetermined signal and waking up the device when the predetermined signal is received. The tags can talk back to the interrogator, permitting them to respond with an identification code or to be remotely programmable. The tags can be powered by alternative power sources, including by solar cells or photocells capturing ambient light or a signal received from the interrogator.