Abstract: An improved system and method for crawl ordering of a web crawler by impact upon search results of a search engine is provided. Content-independent features of uncrawled web pages may be obtained, and the impact of uncrawled web pages may be estimated for queries of a workload using the content-independent features. The impact of uncrawled web pages may be estimated for queries by computing an expected impact score for uncrawled web pages that match needy queries. Query sketches may be created for a subset of the queries by computing an expected impact score for crawled web pages and uncrawled web pages matching the queries. Web pages may then be selected to fetch using a combined query-based estimate and query-independent estimate of the impact of fetching the web pages on search query results.

Abstract: An improved system and method for writing data dependent upon multiple reads in a distributed database is provided. A client may read several data records and may then send a request to a database server to perform a transaction to write a data record dependent upon multiple data records read. A database server may receive the request specifying a transaction to write a data record dependent upon multiple data records read and may perform the transaction by latching a master data record to be written and validating the data records the write depends upon. The multiple data records upon which the write depends may be validated by verifying the multiple data records are current versions of the data records stored in the distributed database. Data intensive applications may use this transaction type in large scale distributed database systems to provide stronger consistency without significantly degrading performance and scalability.

Abstract: An improved system and method is provided for efficiently learning a network of categories using prediction. A learning engine may receive a stream of characters and incrementally segment the stream of characters beginning with individual characters into larger and larger categories. To do so, a prediction engine may be provided for predicting a target category from the stream of characters using one or more context categories. Upon predicting the target category, the edges of the network of categories may be updated. A category composer may also be provided for composing a new category from existing categories in the network of categories, and a new category composed may then be added to the network of categories. Advantageously, iterative episodes of prediction and learning of categories for large scale applications may result in hundreds of thousands of categories connected by millions of prediction edges.

Abstract: An improved system and method is provided for optimizing the reserve price and allocation of web page placements in an online keyword auction using generalized trade reduction. To do so, a generalized trade reduction engine may be provided to convert an implementation of an online keyword auction into a double-sided auction that may optimize the reserve price and allocation of web page placements to maximize revenue. An online keyword auction may be converted into a double-sided market by turning every bidder in the auction into a buyer and every web page placement into a virtual seller. The allocation of buyers and sellers may be divided into disjoint procurement sets and the generalized trade reduction engine may add or remove procurement sets in iterations until conditions of competition among remaining traders may be fulfilled to optimize the reserve price and allocation of web page placements to maximize revenue.

Abstract: An improved system and method is provided for training a multi-class support vector machine to select a common subset of features for classifying objects. A multi-class support vector machine generator may be provided for learning classification functions to classify sets of objects into classes and may include a sparse support vector machine modeling engine for training a multi-class support vector machine using scaling factors by simultaneously selecting a common subset of features iteratively for all classes from sets of features representing each of the classes. An objective function using scaling factors to ensure sparsity of features may be iteratively minimized, and features may be retained and added until a small set of features stabilizes. Alternatively, a common subset of features may be found by iteratively removing at least one feature simultaneously for all classes from an active set of features initialized to represent the entire set of training features.

Abstract: An improved system and method for removing a storage server in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Storage servers may then be removed and column chunks may be redistributed among the remaining storage servers in the column chunk data store.

Abstract: An improved system and method for selecting and visualizing object metadata evolving over time is provided. An application may generate a visualization depicting the temporal evolution of metadata describing objects in an object store over a plurality of time intervals. The application may switch between a visualization of object metadata flowing like a river or cascading like a waterfall over time. A ranked list of metadata items may be determined for some pre-selected intervals during a pre-processing step. Then at runtime when a request may be received for providing a ranked list of metadata items for a query interval, a combination of time intervals from the pre-selected time intervals may be determined that cover the query time interval, and the ranked lists of metadata items for each time interval in the combination of time intervals that cover the query time interval may be aggregated and output for visualization.

Abstract: An improved system and method for recovery from failure of a storage server in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Storage servers may then fail and column chunks may be recreated from parity column chunks and redistributed among the remaining storage servers in the column chunk data store.

Abstract: An improved system and method for importing update data in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns with various partitioning methods. Update data may then be incrementally imported as separate column chunks that may later be merged with the column chunks of the partitioned data table.

Abstract: An improved system and method for a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns as a key with various partitioning methods. There may also be a storage policy for specifying how to partition a data table for distributing column chunks across multiple servers and for specifying a level of redundancy for recovery from failure of storage servers.

Abstract: An improved system and method for a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Any data table may be flexibly partitioned into column chunks using one or more columns as a key with various partitioning methods. There may also be a storage policy for specifying how to partition a data table for distributing column chunks across multiple servers and for specifying a level of redundancy for recovery from failure of storage servers.