Abstract: Systems and methods for automatically scaling a big data system. Methods include determining, at a first time, a first number of nodes for a cluster to process a request; assigning an amount of nodes equal to the first number of nodes to the cluster; determining a rate of progress of the request; determining, at a second time based on the rate of progress a second number of nodes; and modifying the amount of nodes to equal the second number of nodes. Systems include a cluster manager, to add and/or remove any nodes; the big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager including a big data interface for communicating with the big data system; a cluster manager interface for communicating with the cluster manager; and a cluster state machine.

Abstract: The present invention is generally directed to systems and methods of providing automatic scaling pure-spot clusters. Such dusters may be dynamically rebalanced for further costs savings. In accordance with some methods of the present invention may include a method of utilizing a cluster in a big data cloud computing environment where instances may include reserved on-demand instances for a set price and on-demand spot instances that may be bid on by a user, the method including: creating one or more stable nodes, comprising spot instances with a bid price above a price for an equivalent on-demand instance; creating one or more volatile nodes, comprising spot instances with a bid price below a price for an equivalent on-demand instance; using one or more of the stable nodes as a master node; and using the volatile nodes as slave nodes.

Abstract: A system and method for determining a result that includes a solution for a discrete non-linear optimization model for an optimization problem is disclosed. A non-linear relationship between a key performance indicator (KPI) to be optimized and at least one decision variable is determined using econometric modeling. A discrete optimization framework for the optimization problem is determined using a plurality of discrete values of decision variables of the optimization problem. A binary optimization framework is determined from the determined discrete optimization framework based on automatically revising an optimization equation of the discrete optimization framework. A continuous framework is determined from the determined binary optimization framework based on imposing additional constraints and based on modifying the optimization problem. The result is generated, and an action is controlled based on the result.

Abstract: The present invention is generally directed to systems and methods of provisioning, and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

Abstract: The present invention is generally directed to systems and methods of provisioning, and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

Abstract: A system and method for determining a result that includes a solution for a discrete non-linear optimization model for an optimization problem is disclosed. A non-linear relationship between a key performance indicator (KPI) to be optimized and at least one decision variable is determined using econometric modeling. A discrete optimization framework for the optimization problem is determined using a plurality of discrete values of decision variables of the optimization problem. A binary optimization framework is determined from the determined discrete optimization framework based on automatically revising an optimization equation of the discrete optimization framework. A continuous framework is determined from the determined binary optimization framework based on imposing additional constraints and based on modifying the optimization problem. The result is generated, and an action is controlled based on the result.

Abstract: The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

Abstract: The present invention is generally directed to systems and methods of provisioning and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

Abstract: The present invention is generally directed to systems and methods of provisioning and using heterogeneous clusters in a cloud-based big data system, the heterogeneous clusters made up of primary instance types and different types of instances, the method including: determining if there are composition requirements of any heterogeneous cluster, the composition requirements defining instance types permitted for use; determining if any of the permitted different types of instances are required or advantageous for use; determining an amount of different types of instances to utilize, this determination based at least in part on an instance weight; provisioning the heterogeneous cluster comprising both primary instances and permitted different types of instances.

Abstract: The present invention is generally directed to systems and methods of providing automatic scaling pure-spot clusters. Such dusters may be dynamically rebalanced for further costs savings. In accordance with some methods of the present invention may include a method of utilizing a cluster in a big data cloud computing environment where instances may include reserved on-demand instances for a set price and on-demand spot instances that may be bid on by a user, the method including: creating one or more stable nodes, comprising spot instances with a bid price above a price for an equivalent on-demand instance; creating one or more volatile nodes, comprising spot instances with a bid price below a price for an equivalent on-demand instance; using one or more of the stable nodes as a master node; and using the volatile nodes as slave nodes.

Abstract: The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

Abstract: Systems and methods for automatically scaling a big data system are disclosed. Methods may include: determining, at a first time, a first optimal number of nodes for a cluster to adequately process a request; assigning an amount of nodes equal to the first optimal number; determining a rate of progress of the request; determining, at a second time based on the rate of progress a second optimal number of nodes; and modifying the number of nodes assigned to the cluster to equal the second optimal number. Systems may include: a cluster manager, to add and/or remove nodes; a big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager, including: a big data interface, for communicating with the big data system; a cluster manager interface, for communicating with a cluster manager instructions for adding and/or removing nodes from a cluster used to process a request; and a cluster state machine.