Abstract: A Web crawler data collection method is provided for collecting information associated with a plurality of queries, which is used to calculate estimates of return probabilities, clicking probabilities and incorrect response probabilities. The estimated return probabilities relate to a probability that a search engine will return a particular Web page in a particular position of a particular query result page. The estimated clicking probabilities relate to a frequency with which a client selects a returned Web page in a particular position of a particular query result. The estimated incorrect response probabilities relate to the probability that a query to a stale version of a particular Web page yields an incorrect or vacuous response. Further, information may be collected regarding the characteristics and update time distributions of a plurality of Web pages.

Abstract: An apparatus and method for making fractional assignments of processing elements to processing nodes for stream-based applications in a distributed computer system includes determining an amount of processing power to give to each processing element. Based on a list of acceptable processing nodes, a determination of fractions of which processing nodes will work on each processing element is made. To update allocations of the amount of processing power and the fractions, the process is repeated.

Abstract: A method of choosing jobs to run in a stream based distributed computer system includes determining jobs to be run in a distributed stream-oriented system by deciding a priority threshold above which jobs will be accepted, below which jobs will be rejected. Overall importance is maximized relative to the priority threshold based on importance values assigned to all jobs. System constraints are applied to ensure jobs meet set criteria.

Abstract: A technique is provided for efficient search engine crawling. First, optimal crawling frequencies, as well as the theoretically optimal times to crawl each Web page, are determined. This is performed under an extremely general distribution model of Web page updates, one which includes both stochastic and generalized deterministic update patterns. Techniques from the theory of resource allocation problems which are extraordinarily computationally efficient, crucial for practicality because the size of the problem in the Web environment is immense. The second part employs these frequencies and ideal crawl times as input, creating an optimal achievable schedule for crawlers. The solution, based on network flow theory, is exact and highly efficient as well.

Abstract: A method is provided for Web crawler data collection. The method includes the step of collecting information associated with a plurality of queries, the information related to results of the queries and/or responses to the queries. Estimates of return probabilities, clicking probabilities and incorrect response probabilities are then calculated at least in part based on the collected information. The estimated return probabilitues relate to a probability that a search engine will return a particular Web page in a particular position of a particular query result page. The estimated clicking probabilities relate to a frequency with which a client selects a returned Web page in a particular position of a particular query result. The estimated incorrect response probabilities relate to the probability that a query to a stale version of a particular Web page yields an incorrect or vacuous response.

Abstract: A method for dynamically placing objects in slots on a web page in response to a current client request for the web page comprises the steps of classifying users into user groups based one or more user-characteristics, accumulating self-learning data based on user click behavior for each user group, matching the current client request with a corresponding user group and scheduling real-time selection of the slots for the objects on the web page based on the self-learning data of the corresponding user group.

Abstract: Apparatus and methods for maximizing service-level-agreement (SLA) profits are provided. The apparatus and methods consist of formulating SLA profit maximization as a network flow model with a separable set of concave cost functions at the servers of a Web server farm. The SLA classes are taken into account with regard to constraints and cost fiction where the delay constraints are specified as the tails of the corresponding response-time distributions. This formulation simultaneously yields both optimal load balancing and server scheduling parameters under two classes of server scheduling policies, Generalized Processor Sharing (GPS) and Preemptive Priority Scheduling (PPS). For the GPS case, a pair of optimization problems are iteratively solved in order to find the optimal parameters that assign traffic to servers and server capacity to classes of requests. For the PPS case, the optimization problems are iteratively solved for each of the priority classes, and an optimal priority hierarchy is obtained.

Abstract: A rating of a plurality of ratings is predicted. The rating is associated with a user of a plurality of users and the rating corresponds to an item of a plurality of items. One of the plurality of ratings, corresponding to at least one of the plurality of items, is provided for each of the plurality of users. A predictability relation between ones of the plurality of users and other ones of the plurality of users is calculated based on ratings provided by users. One of a plurality of nodes is assigned to each of the plurality of users. Ones of the plurality of nodes are connected with other ones of the plurality of nodes by a plurality of edges based on the predictability relation. A graph which includes the plurality of nodes and the plurality of edges is searched for a path from a node assigned to the user of the plurality of users to another node assigned to another user of the plurality of users.

Abstract: A system and method are provided to analyze information stored in a computer data base by detecting clusters of related or correlated data values. Data values stored in the data base represent a set of objects. A data value is stored in the data base as an instance of a set of features that characterize the objects. The features are the dimensions of the feature space of the data base. Each cluster includes not only a subset of related data values stored in the data base but also a subset of features. The data values in a cluster are data values that are a short distance apart, in the sense of a metric, when projected onto a subspace that corresponds to the subset of features of the cluster. A set of k clusters may be detected such that the average number of features of the subsets of features of the clusters is l.

Abstract: Information is analyzed in the form of a plurality of data values that represent a plurality of objects. A set of features that characterize each object of the plurality of objects is identified. The plurality of data values are stored in a database. Each data value corresponds to at least one of the plurality of objects based on the set of features. Ones of the plurality of data values stored in the database are partitioned into a plurality of clusters. Each cluster of the plurality of clusters is assigned to one respective node of a plurality of nodes arranged in a tree hierarchy. Ones of the plurality of nodes of the tree hierarchy are traversed. If desired, information may be analyzed for finding peer groups in e-commerce applications.

Abstract: A method of analyzing information in the form of a plurality of data records. Each data record includes one or more data values. The data values are partitioned into a plurality of data signatures. Data values of data signatures are compared to data values of data records. Based on the result of the comparison an index is associated with each data record. A bound corresponding to the index is calculated based on a user defined target value and an objective function. If desired, information may be analyzed for finding peer groups in e-commerce applications.

Abstract: A computerized method for load balancing in a geographically distributed or clustered system is disclosed. An arbiter assigns clients to nodes. The arbiter partitions clients into groups based on their request load. Each group is dynamically scheduled among nodes, thus avoiding high load groups from being allocated to the same node and overloading the system. If one of the nodes becomes overload, an alarm is generated, so that fewer or no new clients are allocated to the overloaded node.

Abstract: A system and method for caching objects of non-uniform size. A caching logic includes a selection logic and an admission control logic. The admission control logic determines whether an object not currently in the cache is accessed may be cached at all. The admission control logic uses an auxiliary LRU stack which contains the identities and time stamps of the objects which have been recently accessed. Thus, the memory required is relatively small. The auxiliary cache serves as a dynamic popularity list and an object may be admitted to the cache if and only if it appears on the popularity list. The selection logic selects one or more of the objects in the cache which have to be purged when a new object enters the cache. The order of removal of the objects is prioritized based both on the size as well as the frequency of access of the object and may be adjusted by a time to obsolescence factor (TTO).

Abstract: An apparatus and a method for constructing a multidimensional index tree which minimizes the time to access data objects and is resilient to the skewness of the data. This is achieved through successive partitioning of all given data objects by considering one level at a time starting with one partition and using a top-down approach until each final partition can fit within a leaf node. Subdividing the data objects is via a global optimization approach to minimize the area overlap and perimeter of the minimum bounding rectangles covered by each node. The current invention divides the index construction problem into two subproblems: the first one addresses the tightness of the packing (in terms of area, overlap and perimeter) using a small fan out at each index node and the other one handles the fan out issue to improve index page utilization. These two stages are referred to as binarization and compression.

Abstract: A method of performing a parallel join operation on a pair of relations R1 and R2 in a system containing P processors organized into Q clusters of P/Q processors each. The system contains disk storage for each cluster, shared by the processors of that cluster, together with a shared intermediate memory (SIM) accessible by all processors. The relations R1 and R2 to be joined are first sorted on the join column. The underlying domain of the join column is then partitioned into P ranges of equal size. Each range is further divided into M subranges of progressively decreasing size to create MP tasks T.sub.m,p, the subranges of a given range being so sized relative to one another that the estimated completion time for task T.sub.m,p is a predetermined fraction that of task T.sub.m-1,p. Tasks T.sub.

Abstract: Portions of multimedia program (presentation) are repetitively broadcast to receiving stations with subsequent portions being broadcast less frequently than preceding portions. Blocks of at least one of the portions are broadcast in varying permutations from one repetition to a next repetition. Further, each portion is of a length which is proportional to a sum of the lengths of all preceding portions. A receiver is provided with selects blocks to be skipped (in a pyramid type broadcast) based on information indicative of the permutation selected by the server. The receiver determines the number of blocks to skip before buffering the next block for the video being viewed.

Abstract: A method and system are provided for partitioning a database and for balancing the processing load among processing nodes in a data processing system. A database partition is stored at each node and a logical owner node is assigned to each partition. The logical owner becomes responsible for all database requests related to the corresponding partition. Techniques for optimally assigning logical owners to all partitions in a system are provided and are responsive to parameters such as the load at each node, the combined load of the system, and the total number of assignments allowed. Techniques are also provided to accomplish the transfer of partition ownership between processing nodes when multiple transactions related to the partition are simultaneously occurring.