Abstract: Methods, systems, apparatuses, and computer program products are provided for prefetching data. A workload analyzer may identify job characteristics for a plurality of previously executed jobs in a workload executing on a cluster of one or more compute resources. For each job, identified job characteristics may include identification of an input dataset and an input bandwidth characteristic for the input dataset. A future workload predictor may identify future jobs expected to execute on the cluster based at least on the identified job characteristics. A cache assignment determiner may determine a cache assignment that identifies a prefetch dataset for at least one of the future jobs. A network bandwidth allocator may determine a network bandwidth assignment for the prefetch dataset. A plan instructor may instruct a compute resource of the cluster to load data to a cache local to the cluster according to the cache assignment and the network bandwidth assignment.

Abstract: Methods, systems, apparatuses, and computer program products are provided for prefetching data. A workload analyzer may identify job characteristics for a plurality of previously executed jobs in a workload executing on a cluster of one or more compute resources. For each job, identified job characteristics may include identification of an input dataset and an input bandwidth characteristic for the input dataset. A future workload predictor may identify future jobs expected to execute on the cluster based at least on the identified job characteristics. A cache assignment determiner may determine a cache assignment that identifies a prefetch dataset for at least one of the future jobs. A network bandwidth allocator may determine a network bandwidth assignment for the prefetch dataset. A plan instructor may instruct a compute resource of the cluster to load data to a cache local to the cluster according to the cache assignment and the network bandwidth assignment.

Abstract: Methods, systems, apparatuses, and computer program products are provided for prefetching data. A workload analyzer may identify job characteristics for a plurality of previously executed jobs in a workload executing on a cluster of one or more compute resources. For each job, identified job characteristics may include identification of an input dataset and an input bandwidth characteristic for the input dataset. A future workload predictor may identify future jobs expected to execute on the cluster based at least on the identified job characteristics. A cache assignment determiner may determine a cache assignment that identifies a prefetch dataset for at least one of the future jobs. A network bandwidth allocator may determine a network bandwidth assignment for the prefetch dataset. A plan instructor may instruct a compute resource of the cluster to load data to a cache local to the cluster according to the cache assignment and the network bandwidth assignment.

Abstract: Methods, systems, apparatuses, and computer program products are provided for prefetching data. A workload analyzer may identify job characteristics for a plurality of previously executed jobs in a workload executing on a cluster of one or more compute resources. For each job, identified job characteristics may include identification of an input dataset and an input bandwidth characteristic for the input dataset. A future workload predictor may identify future jobs expected to execute on the cluster based at least on the identified job characteristics. A cache assignment determiner may determine a cache assignment that identifies a prefetch dataset for at least one of the future jobs. A network bandwidth allocator may determine a network bandwidth assignment for the prefetch dataset. A plan instructor may instruct a compute resource of the cluster to load data to a cache local to the cluster according to the cache assignment and the network bandwidth assignment.

Abstract: Described herein is a system and method for ranking and/or taking an action regarding execution of jobs of a shared computing cluster based upon predicted user impact. Information regarding previous executions of a plurality of jobs is obtained, for example, from job execution log(s). Data dependencies of the plurality of jobs are determined. Job impact of each of the plurality of jobs as a function of the determined data dependencies is calculated. User impact of each of the plurality of jobs as a function of the determined data dependencies, the calculated job impact, and time is calculated. The plurality of jobs are ranked in accordance with the calculated user impact. An action is taken in accordance with the ranking of the plurality of jobs. The action can include automatic scheduling of the jobs and/or providing information regarding the rankings to a user.

Abstract: Described herein is a system and method for selecting subexpressions to be materialized. For a predefined storage budget, subexpressions of a set of candidate subexpressions to be materialized to minimize query evaluation cost are selected based upon a calculated utility of the set of candidate subexpressions, interactions of the candidate subexpressions, and, a cost of evaluating the candidate subexpressions. Based upon the subexpressions selected to be materialized, subexpression(s) of the set of candidate subexpressions to use when evaluating particular queries of the set of queries to minimize query evaluation cost are determined.

Abstract: Described herein is a system and method for ranking and/or taking an action regarding execution of jobs of a shared computing cluster based upon predicted user impact. Information regarding previous executions of a plurality of jobs is obtained, for example, from job execution log(s). Data dependencies of the plurality of jobs are determined. Job impact of each of the plurality of jobs as a function of the determined data dependencies is calculated. User impact of each of the plurality of jobs as a function of the determined data dependencies, the calculated job impact, and time is calculated. The plurality of jobs are ranked in accordance with the calculated user impact. An action is taken in accordance with the ranking of the plurality of jobs. The action can include automatic scheduling of the jobs and/or providing information regarding the rankings to a user.

Abstract: Described herein is a system and method for selecting subexpressions to be materialized. For a predefined storage budget, subexpressions of a set of candidate subexpressions to be materialized to minimize query evaluation cost are selected based upon a calculated utility of the set of candidate subexpressions, interactions of the candidate subexpressions, and, a cost of evaluating the candidate subexpressions. Based upon the subexpressions selected to be materialized, subexpression(s) of the set of candidate subexpressions to use when evaluating particular queries of the set of queries to minimize query evaluation cost are determined.

Abstract: A system is provided to improve storage server performance. The system comprises a receiving component of a storage server to receive a partner server's file system metadata while the partner server operates in a normal mode. The partner server's file system metadata is stored in the storage servers metadata cache and is used to mount the partner server's file system when the first server transitions into a takeover mode of operation.

Abstract: A storage server maintains an active file system and at least one read-only, persistent point-in-time image (RPPI) of the active file system. The booting time of the storage server is reduced by delaying the mounting of the RPPI(s) until the storage server receives a request to access a corresponding RPPI after the storage server has already booted.