Abstract: Technologies are disclosed for mapping documents to candidate duplicate documents in a document corpus. A bitset optimized inverted index is created for a document corpus. A document is received for which candidate duplicate documents in the document corpus are to be identified. The document is tokenized using adaptive tokenization. A determination made as to whether tokens in the document are represented in the bitset optimized inverted index. A list of candidate duplicate documents is created for tokens represented in the optimized inverted index utilizing in-memory bitsets that map tokens to documents that contain the tokens in the document corpus.

Abstract: Disclosed are various embodiments for the management of dependencies on data. Multiple jobs are executed in one or more computing devices. It is determined that a first one of the jobs consumes data produced by a second one of the jobs, and a dependency is recorded. It is then determined whether the dependency is satisfied. When the dependency is determined to be unsatisfied, execution of the first one of the jobs is suspended. When the second one of the jobs produces the data, the dependency is updated as being satisfied. Execution of the first one of the jobs is resumed in response to the dependency being updated as being satisfied.