Abstract: A method for applying bloom filter on a large data set consisting of key-value pairs, using at least one processor includes: partitioning large data-set of key-value pairs into data chunks; determining Bloom filter Vector and number of segments in the vector for each data chunk; Encoding all keys of a given Chunk into a Bloom filter vector; Determining the segment-id of a given key using H (0) hash function; Encoding Key into a Bloom filter segment with the determined segment-id, using a K-bit array produced by H1, . . . Hk functions; and Packing of segments into extent data structures where each extent includes segments of different chunks, but with the same segment-id wherein a single extent filters multiple chunks, depending on a packing factor (the number of segments packed into a single extent).

Abstract: The present invention provides a method for applying bloom filter on a large data set consisting of key-value pairs, using at least one processor. The method comprising the step of: partitioning large data-set of key-value pairs into data chunks; determining Bloom filter Vector and number of segments in the vector for each data chunk; Encoding all keys of a given Chunk into a Bloom filter vector; Determining the segment-id of a given key using H (0) hash function; Encoding Key into a Bloom filter segment with said determined segment-id, using a K-bit array produced by H1, . . . Hk functions; and Packing of segments into extent data structures where each extent includes segments of different chunks, but with the same segment-id. wherein a single extent filters multiple chunks, depending on a packing factor (the number of segments packed into a single extent).