Abstract: Apparatus and method for managing data objects in a distributed data storage system, such as a cloud computing environment. In some embodiments, a data object is encrypted using a user encryption key to generate ciphertext. A first hash function is applied to the ciphertext and an audit encryption key to generate a first hash value. An audit value is formed by combining the ciphertext and the first hash value, and the audit value is locally encrypted and stored to non-volatile memory (NVM) of each of a plurality of storage nodes. An audit process is performed to confirm each of the encrypted replicas store identical copies of the ciphertext. This is carried out by decrypting the ciphertext and applying a second hash function to the ciphertext and the audit encryption key by each storage node to form a plurality of second hash values which are then compared by an audit processor.