Abstract: Approaches are disclosed for distributing messages across multiple data centers where the data centers do not store messages using a same message queue protocol. In some embodiment, a network element translates messages from a message queue protocol (e.g., Kestrel, RABBITMQ, APACHE Kafka, and ACTIVEMQ) to an application layer messaging protocol (e.g., XMPP, MQTT, WebSocket protocol, or other application layer messaging protocols). In other embodiments, a network element translates messages from an application layer messaging protocol to a message queue protocol. Using the new approaches disclosed herein, data centers communicate using, at least in part, application layer messaging protocols to disconnect the message queue protocols used by the data centers and enable sharing messages between messages queues in the data centers.

Abstract: Embodiments include receiving an indication of a data storage module to be associated with a tenant of a distributed storage system, allocating a partition of a disk for data of the tenant, creating a first association between the data storage module and the disk partition, creating a second association between the data storage module and the tenant, and creating rules for the data storage module based on one or more policies configured for the tenant. Embodiments further include receiving an indication of a type of subscription model selected for the tenant, and selecting the disk partition to be allocated based, at least in part, on the subscription model selected for the tenant. More specific embodiments include generating a storage map indicating the first association between the data storage module and the disk partition and indicating the second association between the data storage module and the tenant.

Abstract: In one embodiment, data indicative of the size of an intermediate data set generated by a first resource device is received at a computing device. The intermediate data set is associated with a virtual machine to process the intermediate data set. A virtual machine configuration is determined based on the size of the intermediate data set. A second resource device is selected to execute the virtual machine based on the virtual machine configuration and on an available bandwidth between the first and second resource devices. The virtual machine is then assigned to the second resource device to process the intermediate data set.

Abstract: In one embodiment, a device receives information regarding a data set to be processed by a map-reduce process. The device generates a set of virtual clusters for the map-reduce process based on network bandwidths between nodes of the virtual clusters, each node of the virtual cluster corresponding to a resource device, and associates the data set with a map-reduce process task. The device then schedules the execution of the task by a node of the virtual clusters based on the network bandwidth between the node and a source node on which the data set resides.

Abstract: In one embodiment, a processor can receive data representing a view reflected by a mirror of a plurality of mirrors. The plurality of mirrors may be configured in a space to reflect a plurality of views of structures in the space. The mirror of the plurality of mirrors may include a uniquely identifiable feature distinguishable from other objects in the space. The processor can identify the mirror of the plurality of mirrors according to the uniquely identifiable feature. The processor can also determine an attribute of the structures according to the identified mirror and the data representing the view reflected by the mirror.

Abstract: In one embodiment, a scale out policy service for processing a stream of messages includes a distributed stream processing computation system comprising distributed stream processing nodes, a distributed storage system, and a rules engine. A stream processing engine of the distributed stream processing computation system can receive the stream of messages comprising requests and/or events, and assign a first message to be processed by one or more distributed stream processing nodes based on one or more properties of the message. The one or more distributed stream processing nodes can be communicably connected to the distributed storage system and/or the rules engine to provide (1) an answer in response to the first message and/or (2) cause an action to be executed based on the first message.

Abstract: The present disclosure relates to assignment or generation of reducer virtual machines after the “map” phase is substantially complete in MapReduce. Instead of a priori placement, distribution of keys after the “map” phase over the mapper virtual machines can be used to efficiently reducer tasks in virtualized cloud infrastructure like OpenStack. By solving a constraint optimization problem, reducer VMs can be optimally assigned to process keys subject to certain constraints. In particular, the present disclosure describes a special variable matrix. Furthermore, the present disclosure describes several possible cost matrices for representing the costs determined based on the key distribution over the mapper VMs (and other suitable factors).

Abstract: The present disclosure describes a distributed, advertisement-based, solution for scheduling virtual resources in cloud infrastructures such as the OpenStack. The scheduling algorithm distributes the scheduling requirements and host state feasibility checks to the individual hosts in the datacenter, which can periodically send a summarized advertisement to the scheduler controller listing the number of instances of different type(s) of virtual resources that a particular host can support. The scheduler controller, thus no longer has to compute and maintain individual host states, and the scheduling problem is reduced to selecting the feasible advertisements that satisfy a given request. The solution can be extended to a scenario of multiple scheduler controllers using the same distributed, advertisement-based, approach.

Abstract: The present disclosure describes a distributed, advertisement-based, solution for scheduling virtual resources in cloud infrastructures such as the OpenStack. The scheduling algorithm distributes the scheduling requirements and host state feasibility checks to the individual hosts in the datacenter, which can periodically send a summarized advertisement to the scheduler controller listing the number of instances of different type(s) of virtual resources that a particular host can support. The scheduler controller, thus no longer has to compute and maintain individual host states, and the scheduling problem is reduced to selecting the feasible advertisements that satisfy a given request. The solution can be extended to a scenario of multiple scheduler controllers using the same distributed, advertisement-based, approach.

Abstract: The present disclosure relates to assignment or generation of reducer virtual machines after the “map” phase is substantially complete in MapReduce. Instead of a priori placement, distribution of keys after the “map” phase over the mapper virtual machines can be used to efficiently reducer tasks in virtualized cloud infrastructure like OpenStack. By solving a constraint optimization problem, reducer VMs can be optimally assigned to process keys subject to certain constraints. In particular, the present disclosure describes a special variable matrix. Furthermore, the present disclosure describes several possible cost matrices for representing the costs determined based on the key distribution over the mapper VMs (and other suitable factors).

Abstract: An example method for leveraging hardware accelerators for scalable distributed stream processing in a network environment is provided and includes allocating a plurality of hardware accelerators to a corresponding plurality of bolts of a distributed stream in a network, facilitating a handshake between the hardware accelerators and the corresponding bolts to allow the hardware accelerators to execute respective processing logic according to the corresponding bolts, and performing elastic allocation of hardware accelerators and load balancing of stream processing in the network. The distributed stream comprises a topology of at least one spout and the plurality of bolts. In specific embodiments, the allocating includes receiving capability information from the bolts and the hardware accelerators, and mapping the hardware accelerators to the bolts based on the capability information.

Abstract: In particular embodiments, a method includes, from an indexer in a sensor network, accessing a set of sensor data that includes sensor data aggregated together from sensors in the sensor network, one or more time stamps for the sensor data, and metadata for the sensor data identifying one or more pre-determined attributes of the sensor data. The method includes, at the indexer, generating an index of the set of sensor data according to a multi-dimensional array configured for querying of the set of sensor data along with a plurality of other sets of sensor data. One or more first ones of the dimensions of the multi-dimensional array include time, and one or more second ones of the dimensions of the multi-dimensional array include one or more of the pre-determined sensor-data attributes. The method includes, from the indexer, communicating the index of the set of sensor data for use in responding to one or more queries of the set of sensor data along with a plurality of other sets of sensor data.

Abstract: Techniques are provided to generate and store a network graph database comprising information that indicates a service node topology, and virtual or physical network services available at each node in a network. A service request is received for services to be performed on packets traversing the network between at least first and second endpoints. A subset of the network graph database is determined that can provide the services requested in the service request. A service chain and service chain identifier is generated for the service based on the network graph database subset. A flow path is established through the service chain by flow programming network paths between the first and second endpoints using the service chain identifier.

Abstract: In particular embodiments, a method includes accessing sensor data from sensor nodes in a sensor network and aggregating the sensor data for communication to an indexer in the sensor network. The aggregation of the sensor data includes deduplicating the sensor data; validating the sensor data; formatting the sensor; generating metadata for the sensor data; and time-stamping the sensor data. The metadata identifies one or more pre-determined attributes of the sensor data. The method also includes communicating the aggregated sensor data to the indexer in the sensor network. The indexer is configured to index the aggregated sensor data according to a multi-dimensional array for querying of the aggregated sensor data along with other aggregated sensor data. One or more first ones of the dimensions of the multi-dimensional array include time and one or more second ones of the dimensions of the multi-dimensional include one or more of the pre-determined sensor-data attributes.

Abstract: Techniques are provided to generate and store a network graph database comprising information that indicates a service node topology, and virtual or physical network services available at each node in a network. A service request is received for services to be performed on packets traversing the network between at least first and second endpoints. A subset of the network graph database is determined that can provide the services requested in the service request. A service chain and service chain identifier is generated for the service based on the network graph database subset. A flow path is established through the service chain by flow programming network paths between the first and second endpoints using the service chain identifier.

Abstract: In particular embodiments, a method includes receiving a query for particular sensor data among multiple sensor data from multiple sensors. The plurality of sensor data has been indexed according to a multi-dimensional array. One or more first ones of the dimensions include time, and one or more second ones of the dimensions include one or more pre-determined sensor-data attributes. The method includes translating the query to correspond to the indexing of the plurality of sensor data. The translated query includes one or more values for one or more of the dimensions of the multi-dimensional array. The method includes communicating the translated query to search among the plurality of sensor data according to its indexing to identify the particular sensor data.

Abstract: A method for summarizing capabilities in a hierarchically arranged data center includes receiving capabilities information, wherein the capabilities information is representative of capabilities of respective nodes at a first hierarchical level in the hierarchically arranged data center, clustering nodes based on groups of capabilities information, generating a histogram that represents individual node clusters, and sending the histogram to a next higher level in the hierarchically arranged data center. Relative rankings of capabilities may be used to order a sequence of clustering operations.

Abstract: In one embodiment, a device receives information regarding a data set to be processed by a map-reduce process. The device generates a set of virtual clusters for the map-reduce process based on network bandwidths between nodes of the virtual clusters, each node of the virtual cluster corresponding to a resource device, and associates the data set with a map-reduce process task. The device then schedules the execution of the task by a node of the virtual clusters based on the network bandwidth between the node and a source node on which the data set resides.

Abstract: In one embodiment, a method comprises determining a stochastic distribution of received service requests for services in a data network having a prescribed physical topology; and allocating virtualized resources within the prescribed physical topology for a corresponding service request, based on the stochastic distribution.

Abstract: In one embodiment, data indicative of the size of an intermediate data set generated by a first resource device is received at a computing device. The intermediate data set is associated with a virtual machine to process the intermediate data set. A virtual machine configuration is determined based on the size of the intermediate data set. A second resource device is selected to execute the virtual machine based on the virtual machine configuration and on an available bandwidth between the first and second resource devices. The virtual machine is then assigned to the second resource device to process the intermediate data set.