Abstract: In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.

Abstract: One example packet processing device includes a buffer, and the packet processing device obtains a to-be-measured packet. In response to determining that occupied storage space in the buffer is less than a preset threshold, the packet processing device reads the to-be-measured information from the buffer, and modifies, based on the to-be-measured information and a first algorithm, a pieces of data in first measurement data corresponding to the to-be-measured packet, where a is a positive integer. In response to determining that occupied storage space in the buffer is greater than or equal to a preset threshold, the packet processing device modifies, based on to-be-measured information and a second algorithm, w pieces of data in second measurement data corresponding to the to-be-measured packet, where w is a positive integer, and w is less than a.

Abstract: A data processing method is disclosed, and the method includes: encoding a data chunk of a predetermined size, to generate an error-correcting data chunk corresponding to the data chunk, where the data chunk includes a data object, and the data object includes a key, a value, and metadata; and generating a data chunk index and a data object index, where the data chunk index is used to retrieve the data chunk and the error-correcting data chunk corresponding to the data chunk, the data object index is used to retrieve the data object in the data chunk, and each data object index is used to retrieve a unique data object.

Abstract: One example packet processing device includes a buffer, and the packet processing device obtains a to-be-measured packet. In response to determining that occupied storage space in the buffer is less than a preset threshold, the packet processing device reads the to-be-measured information from the buffer, and modifies, based on the to-be-measured information and a first algorithm, a pieces of data in first measurement data corresponding to the to-be-measured packet, where a is a positive integer. In response to determining that occupied storage space in the buffer is greater than or equal to a preset threshold, the packet processing device modifies, based on to-be-measured information and a second algorithm, w pieces of data in second measurement data corresponding to the to-be-measured packet, where w is a positive integer, and w is less than a.

Abstract: A data processing method is disclosed, and the method includes: encoding a data chunk of a predetermined size, to generate an error-correcting data chunk corresponding to the data chunk, where the data chunk includes a data object, and the data object includes a key, a value, and metadata; and generating a data chunk index and a data object index, where the data chunk index is used to retrieve the data chunk and the error-correcting data chunk corresponding to the data chunk, the data object index is used to retrieve the data object in the data chunk, and each data object index is used to retrieve a unique data object.

Abstract: In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.

Abstract: A method, an apparatus, and a system for identifying an abnormal IP data stream, which are used to improve identification accuracy. The method provided by the embodiments of the present invention includes: receiving Y elements sent by a data collection node; mapping the Y elements to N buckets; acquiring a bucket in the N buckets as a target bucket; acquiring r upper traffic limits of a first object in r buckets within the current time interval, the first object is any object mapped to the target bucket; and identifying, according to a preset abnormal object type and the r upper traffic limits within the current time interval, whether the first object is an abnormal object, where the preset abnormal object type is a heavy hitter or a heavy changer.

Abstract: A method, an apparatus, and a system for identifying an abnormal IP data stream, which are used to improve identification accuracy. The method provided by the embodiments of the present invention includes: receiving Y elements sent by a data collection node; mapping the Y elements to N buckets; acquiring a bucket in the N buckets as a target bucket; acquiring r upper traffic limits of a first object in r buckets within the current time interval, the first object is any object mapped to the target bucket; and identifying, according to a preset abnormal object type and the r upper traffic limits within the current time interval, whether the first object is an abnormal object, where the preset abnormal object type is a heavy hitter or a heavy changer.

Abstract: An apparatus and method for improving synchronization between threads in a multi-core processor system are provided. An apparatus includes a memory, a first processor core, and a second processor core. The memory includes a shared ring buffer for storing data units, and stores a plurality of shared variables associated with accessing the shared ring buffer. The first processor core runs a first thread and has a first cache associated therewith. The first cache stores a first set of local variables associated with the first processor core. The first thread controls insertion of data items into the shared ring buffer using at least one of the shared variables and the first set of local variables. The second processor core runs a second thread and has a second cache associated therewith. The second cache stores a second set of local variables associated with the second processor core.

Abstract: An apparatus and method for improving synchronization between threads in a multi-core processor system are provided. An apparatus includes a memory, a first processor core, and a second processor core. The memory includes a shared ring buffer for storing data units, and stores a plurality of shared variables associated with accessing the shared ring buffer. The first processor core runs a first thread and has a first cache associated therewith. The first cache stores a first set of local variables associated with the first processor core. The first thread controls insertion of data items into the shared ring buffer using at least one of the shared variables and the first set of local variables. The second processor core runs a second thread and has a second cache associated therewith. The second cache stores a second set of local variables associated with the second processor core.

Abstract: Methods and apparatuses are provided for detecting traffic patterns in a data network. A sequential hashing scheme can be utilized that has D hash arrays. Each hash array i, wherein 1?i?D, includes Mi independent hash tables each having K buckets, with each of the buckets having an associated traffic total. Each of the keys corresponds with a single bucket of each of the Mi independent hash tables of each hash array i. The keys of the data network are partitioned into D words. As traffic is received for a key, a traffic total of each bucket that corresponds with a key is updated. The hash arrays can then be utilized to identify high traffic buckets of the independent hash tables having a traffic total greater than a threshold value. The high traffic buckets can be used to detect significant traffic patterns of the data network.

Abstract: Methods and apparatuses are provided for detecting traffic patterns in a data network. A sequential hashing scheme can be utilized that has D hash arrays. Each hash array i, wherein 1?i?D, includes Mi independent hash tables each having K buckets, with each of the buckets having an associated traffic total. Each of the keys corresponds with a single bucket of each of the Mi independent hash tables of each hash array i. The keys of the data network are partitioned into D words. As traffic is received for a key, a traffic total of each bucket that corresponds with a key is updated. The hash arrays can then be utilized to identify high traffic buckets of the independent hash tables having a traffic total greater than a threshold value. The high traffic buckets can be used to detect significant traffic patterns of the data network.