Abstract: A system and a method for spill management during the shuffle phase of a map-reduce job performed in a distributed computer system on distributed files. A spilling protocol is provided for handling the spilling of intermediate data based on at least one popularity attribute of key-value pairs of the input data on which the map-reduce job is performed. The spilling protocol includes an assignment order to storage resources belonging to the computer system based on the at least one popularity attribute. The protocol can be deployed in computer systems with heterogeneous storage resources. Additionally, pointers or tags can be assigned to improve shuffle phase performance. The distributed file systems that are most suitable are ones usable by Hadoop, e.g., Hadoop Distributed File System (HDFS).

Abstract: Data surveillance techniques are presented for the detection of security issues, especially of the kind where privileged data may be stolen by steganographic, data manipulation or any form of exfiltration attempts. Such attempts may be made by rogue users or admins from the inside of a network, or from outside hackers who are able to intrude into the network and impersonate themselves as legitimate users. The system and methods use a triangulation process whereby analytical results pertaining to data protocol, user-behavior and packet content are combined to establish a baseline for the data. Subsequent incoming data is then scored and compared against the baseline to detect any security anomalies. The techniques are also applicable for detecting performance issues indicative of a system malfunction or deterioration.

Abstract: A method and high availability cluster configuration for capture of data packets propagating in an external network into the cluster through a wired-tap connection between the cluster and at least one router of the external network. Captured packets are supplied with popularity tags and striped over high-access-rate non-transitory storage media belonging to a capture pool of serving nodes that are specially provisioned and selected by the master node. The captured packets are either devolved or demoted to low-access-rate non-transitory storage media of at least two serving nodes not including any backup master based on popularity metrics contained their tags. The demotion, devolution as well as determination of popularity metrics and setting of a popularity threshold are managed by a distributed management protocol.

Abstract: A system and a method for spill management during the shuffle phase of a map-reduce job performed in a distributed computer system on distributed files. A spilling protocol is provided for handling the spilling of intermediate data based on at least one popularity attribute of key-value pairs of the input data on which the map-reduce job is performed. The spilling protocol includes an assignment order to storage resources belonging to the computer system based on the at least one popularity attribute. The protocol can be deployed in computer systems with heterogeneous storage resources. Additionally, pointers or tags can be assigned to improve shuffle phase performance. The distributed file systems that are most suitable are ones usable by Hadoop, e.g., Hadoop Distributed File System (HDFS).

Abstract: A system and a method for spill management during the shuffle phase of a map-reduce job performed in a distributed computer system on distributed files. A spilling protocol is provided for handling the spilling of intermediate data based on at least one popularity attribute of key-value pairs of the input data on which the map-reduce job is performed. The spilling protocol includes an assignment order to storage resources belonging to the computer system based on the at least one popularity attribute. The protocol can be deployed in computer systems with heterogeneous storage resources. Additionally, pointers or tags can be assigned to improve shuffle phase performance. The distributed file systems that are most suitable are ones usable by Hadoop, e.g., Hadoop Distributed File System (HDFS).

Abstract: A method and high availability cluster configuration for capture of data packets propagating in an external network into the cluster through a wired-tap connection between the cluster and at least one router of the external network. Captured packets are supplied with popularity tags and striped over high-access-rate non-transitory storage media belonging to a capture pool of serving nodes that are specially provisioned and selected by the master node. The captured packets are either devolved or demoted to low-access-rate non-transitory storage media of at least two serving nodes not including any backup master based on popularity metrics contained their tags. The demotion, devolution as well as determination of popularity metrics and setting of a popularity threshold are managed by a distributed management protocol.

Abstract: A system and a method for spill management during the shuffle phase of a map-reduce job performed in a distributed computer system on distributed files. A spilling protocol is provided for handling the spilling of intermediate data based on at least one popularity attribute of key-value pairs of the input data on which the map-reduce job is performed. The spilling protocol includes and assignment order to storage resources belonging to the computer system based on the at least one popularity attribute. The protocol can be deployed in computer systems with heterogeneous storage resources. Additionally, pointers or tags can be assigned to improve shuffle phase performance. The distributed file systems that are most suitable are ones usable by Hadoop, e.g., Hadoop Distributed File System (HDFS).