Abstract: The invention provides methods of encoding content for distribution over a network and methods for decoding encoded content which has been distributed over the network. In a first example in which the content is divided into a plurality of segments and each segment comprising a plurality of blocks of data, the method comprises selecting a segment from the plurality of segments and selecting at least two blocks of the selected segment from a store of blocks. A new encoded block is created from a linear combination of the selected blocks.

Abstract: A data center has a plurality of secure processing units; a plurality of data stores holding encrypted data records; and a network connecting the secure processing units and the data stores. The secure processing units comprise computing functionality configured to execute a data processing operation in parallel on the secure processing units by being configured to read encrypted records from the stores, process one or more of the encrypted records within the secure processing units, send one or more of the encrypted records to the stores. The data center is configured to carry out a secret shuffle of the data records to protect the privacy of data processed in the data center from an observer observing any one or more of: the reading of the records, the sending of the records, the writing of the records; the secret shuffle comprising a random permutation of the records hidden from the observer.

Abstract: Streaming query resource control is described, for example, to allocate streaming queries to servers in a data center providing a streaming query platform. In various embodiments streaming queries are allocated to servers in a manner seeking to balance load between the servers and also to reduce network traffic costs between data stream sources and the servers. In various examples, query types are taken into account, where a query type is the identity of one or more data stream sources used by the query, and optionally also traffic rates of the data stream sources. In some examples, processes for allocating incoming queries in an online fashion are described and in some examples, processes for allocating queries in an offline fashion are described. In examples, a network traffic cost metric is used which takes into account an incremental network traffic cost of adding a given query at a server.

Abstract: Methods for enforcing confidentiality and integrity of code and data while running the code over the data in a distributed computing system are described. In an embodiment each machine which processes data within the system provides a secure sub-system which is protected from other parts of the machine and which receives encrypted data and encrypted code, processes the data using the received code and outputs encrypted data. When establishing the secure sub-systems, keys are exchanged between the client and secure sub-systems and the secure sub-systems provide an attestation confirming the identity of the code running in the secure sub-systems and confirming that the code is running on genuine secure sub-systems. In another embodiment a data-flow computation system is described in which chunks of input data, each comprising an identifier, are authenticated/encrypted. The identifiers are used within the system to confirm that each chunk is processed exactly once.

Abstract: Graph partitioning for massive scale graphs is described, such as for graphs having vertices representing people and edges representing connections between people in a social networking system; or for graphs where the vertices represent other items and the edges represent relationships between the items. In various embodiments a graph data allocator receives a graph vertex and its edges and allocates the vertex to one of a plurality of clusters each associated with one or more computing devices. In various embodiments the allocation is made by optimizing an objective function which takes into account both a cost of edges between clusters and a cost related to sizes of the clusters. In some examples the cost related to sizes of the clusters comprises a convex function applied to each of the cluster sizes. In examples, computations on the graph data are carried out with reduced runtimes and communications cost.

Abstract: Resource optimization for online services is described. In one example, objects (such as mailboxes or other data associated with an online service) are assigned to network elements (such as servers) by inferring a relationship graph from log data relating to usage of the online service. The graph has a node for each object, and connections between each pair of objects having data items in common. Each connection has a weight relating to the number of common data items. The graph is partitioned into a set of clusters, such that each cluster has nodes joined by connections with a high weight relative to the weight of connections between nodes in different clusters. The objects are then distributed to the network elements such that objects corresponding to nodes in the same cluster are located on the same network element.

Abstract: Methods of generating filters automatically from data processing jobs are described. In an embodiment, these filters are automatically generated from a compiled version of the data processing job using static analysis which is applied to a high-level representation of the job. The executable filter is arranged to suppress rows and/or columns within the data to which the job is applied and which do not affect the output of the job. The filters are generated by a filter generator and then stored and applied dynamically at a filtering proxy that may be co-located with the storage node that holds the data. In another embodiment, the filtered data may be cached close to a compute node which runs the job and data may be provided to the compute node from the local cache rather than from the filtering proxy.

Abstract: A topology management process is implemented in peer-to-peer content distribution clouds using tracker nodes. Tracker nodes have information about available peers and assist peers in finding other peers to connect to. Various algorithms for use at the tracker nodes are described for selecting which peers to return as potentials for forming connections to. In addition, architectures and algorithms to allow efficient scaling of tracker nodes in peer-to-peer clouds are described.

Abstract: A resource allocation framework for wireless/wired networks is described. In an embodiment, methods of end host based traffic management are described which operate separately from the underlying access control protocol within the network (e.g. wireless MAC protocol or TCP). The rate limits for each flow are set based on per-flow weights, which may be user specified, and based on an estimate of the utilization of the shared resource and the rate limits are adjusted periodically so that the resource is not underutilized or saturated. Some embodiments compute a virtual capacity of the resource which is adjusted to optimize the value of the utilization and then the virtual capacity is shared between flows according to the per-flow weights. Methods for estimating the utilization of a wireless network and the capacity of a broadband access link are also described.

Abstract: Resource optimization for online services is described. In one example, objects (such as mailboxes or other data associated with an online service) are assigned to network elements (such as servers) by inferring a relationship graph from log data relating to usage of the online service. The graph has a node for each object, and connections between each pair of objects having data items in common. Each connection has a weight relating to the number of common data items. The graph is partitioned into a set of clusters, such that each cluster has nodes joined by connections with a high weight relative to the weight of connections between nodes in different clusters. The objects are then distributed to the network elements such that objects corresponding to nodes in the same cluster are located on the same network element.

Abstract: A dual mode communication device utilizes a control channel to exploit diversity, history, and context in advance of establishing a broadband data exchange session on a broadband but shorter range wireless data channel, maximizing productive use of such a session. Appropriate diversity for the negotiated session further enhance data transfer, including path diversity, radio technology diversity (e.g., WiMax, Wi-Fi, ultra wideband, Bluetooth), antenna diversity (e.g., MIMO), modulation diversity (e.g., rate selection for 802.11, or symbol length selection to combat multi-path fading), and frequency diversity (e.g., 2.4 GHz versus 5 GHz). Historical information about channel characteristics optimize the selection of channel parameters with respect to the diversity choices. In addition, context information such as location and speed can be used to categorize the historical information that is collected to further optimize channel parameters.

Abstract: A content distribution mechanism that relies on cooperative desktop PCs to distribute content is disclosed. The mechanism distributes content in a robust manner by allowing at least one intermediate network node (i.e., between a source and client) to generate and send packets that contain a linear combination of the portions of content available at the node. Such linear combinations may be created by the source and client using at least a portion of the original content file in either encoded or unencoded form. After the client has received enough linearly independent combinations of packets, the original content may be reconstructed. Further, security for network coding file distribution may be employed to maintain the efficiency and security of the content distribution mechanism. A security server may generate security information using a hashing algorithm including the property of producing security information for each block which survives the process of creating encoded blocks.

Abstract: The invention describes a control node for a content distribution network and a method of automatically verifying content distributed over a network at a node in the network. In the method, a content description is received which comprises a content identifier, a publisher identifier, publisher authorization information and content checking information. The integrity and validity of this information are checked and if both the checks are passed, the content description is stored.

Abstract: Managing content by influencing its distribution in the form of blocks or other units is crucial to the performance of a peer-to-peer content distribution system. We provide a content management module at each peer node to achieve this. The content management module implements a content request/response cycle in order to negotiate about blocks of content with other peer nodes. This cycle comprises an offer request, offer reply and a block request. These steps are preferably followed by data exchange and verification. The negotiation protocol used by our content management module provides a generic method that is advantageously used in conjunction with different encoding schemes such as full network coding and group network coding. In one embodiment we use group network coding and add information to the offer request and offer reply to facilitate efficient content distribution. Block request messages are selected to promote the use of on-the-fly decoding where possible.

Abstract: A resource allocation framework for wireless/wired networks is described. In an embodiment, methods of end host based traffic management are described which operate separately from the underlying access control protocol within the network (e.g. wireless MAC protocol or TCP). The rate limits for each flow are set based on per-flow weights, which may be user specified, and based on an estimate of the utilization of the shared resource and the rate limits are adjusted periodically so that the resource is not underutilized or saturated. Some embodiments compute a virtual capacity of the resource which is adjusted to optimize the value of the utilization and then the virtual capacity is shared between flows according to the per-flow weights. Methods for estimating the utilization of a wireless network and the capacity of a broadband access link are also described.

Abstract: Sampling rules for information dissemination are described which may be applied in a system containing a number of nodes arranged into groups. A target address is selected using one of two methods: selection of an address from the entire address space of the system and selection of an address from a part of the address space which corresponds to set of groups of nodes. The set of groups of nodes is updated when information is successfully disseminated to a node at a target address selected using the first of the two methods. Rules to determine which of the two methods are used for any particular selection operation are also described.

Abstract: An end-host based network management system and methods are described. The methods are performed independently at each end-host within the network based on data on local flows which is shared between end-hosts. In an embodiment, an end-host shares data on constrained local flows with other end-hosts and receives such data from other end-hosts. Based on this data, the end-host determines which flows from other nodes are competing for a shared resource with a constrained local flow and allocates the capacity of the shared resource between all the competing flows. This allocation is then enforced for the local flow by the end-host. Other end-hosts with competing flows perform similar methods and through an iterative process the contention for the shared resource is resolved and the utilization of the shared resource is optimized.

Abstract: In example methods and algorithms, a node in a wireless mesh network calculates an estimated cost for a packet flow through the node. The estimation may be based on the back-log at the node and the cost of downstream neighbor nodes for the flow. Further, selection of a downstream flow and a downstream neighbor node may be based on the estimation. A packet re-ordering algorithm is also described which intercepts packets received at a node and delays delivery of the packet to the IP layer if an earlier packet in the sequence of packets has not been received.

Abstract: A method of transmitting data across a wireless mesh network is described which uses network coding at each of the intermediate nodes between the source node and the destination node. Each intermediate node also controls the rate at which it broadcasts packets based on link congestion and the backlog of packets at each of the possible next-hop nodes for the data flow.

Abstract: A wire protocol is described which implements connection management and other methods to give enhanced peer-to-peer content distribution. Connections between nodes can be placed in a “notify” state when they are idle but may soon yield useful content. This notify state is also used together with a content request/response cycle to allow a peer to evaluate content available at a neighbour. If no suitable content is available a notify state is entered. When new content is later received at the neighbour it is able to inform the requesting node to allow it to restart the content request/response cycle.