Two-Stage Multiple Kernel Learning Method

Abstract

Disclosed are methods and structures of Multiple Kernel learning framed as a standard binary classification problem with additional constraints that ensure the positive definiteness of the learned kernel. Advantageously, the disclosed methods and structures permit the use of binary classification technologies to develop better performing, and more scalable Multiple Kernel Learning methods that are conceptually simpler.

Description

TECHNICAL FIELD

This disclosure relates generally to the field of machine learning and in particular to a two stage method of multiple kernel learning.

BACKGROUND

Kernel functions have become a popular tool of machine learning and methods that automate the task of specifying a suitable kernel as become increasingly important. More particularly, the known Multiple Kernel Learning (MKL) problem of finding a combination of pre-specified base kernels that is suitable for a particular task at hand has received significant interest.

Generally, the prior art has approached this problem along two paths. The first path solves a joint optimization problem over both the weights of the kernel combination and the parameters of the classifier. Such one-stage approach has been described by Lanckriet et al. (See, G. R. G Lanckriet, N. Cristianini, P. Bartlett, L. El Ghaoui and M. I. Jordan, “Learning the Kernel Matrix with Semidefinite Programming”, Journal of Machine Learning Research, 5:27-72, 2004) and has since received significant attention directed at providing faster algorithms (See, A. Rakotomamonjy, F. Bach, S. Canu and Y. Grandvalet, “More Efficiency in Multi Kernel Learning”, in Internation Conference on Machine Learning (ICML-11), pp. 249-256, 2011; S. Sonnenburg, G. Ratsch, C. Schafer, and B. Scholkpf, “Large Scale Multiple Kernel Learning”, Journal of Machine Learning Research, 7, 2006) Likewise, a number of theoretical analysis have been described (See, e.g., C. Cortes, M. Mohri, and A. Ros-tamizadeh, “Two Stage Learning Kernel Algorithms”, in International Conference on Machine Learning, 2010; M. Kloft, U. Brefeld, S. Sonnenburg and A. Zien, “lp-Norm Multiple Kernel Learning”, Journal of Machine Learning Research, 12:953-997, 2011; F. Bach, “Consistency of the Group Lasso and Multiple Kernel Learning”, Journal of Machine Learning Research, 9:1179-1225, 2008). Additionally, extensions to multi-class classification have been explored (See., e.g., A. Zien and C. S. Ong, “Multiclass Multiple Kernel Learning”, in International Conference on Machine Learning, 2007; and finally to non-linear combinations of kernels (See., e.g., C. Cortes, M. Mohri, and A. Rostamizadeh, “Learning non-linear combinations in Kernels”, in Advances in Neural Information Processing Systems, 200.)

The second path in kernel learning follows a two-stage approach: first learn a “good” combination of base kernels using the training data, then use the learned kernel with a standard kernel method such as SVM or kernel ridge regression to obtain a classifier/regressor. This two-stage leaning approaches so far (See., e.g., C. Cortes, M. Mohri, and A. Rostamizadeh, “Two-Stage Learning Kernel Algorithms”, in International Conference on Machine Learning, 2010; and N. Cristianini, J. Shawe-Taylor, A. Elisseef, and J. S. Kandola, “On Kernel-Target Alignment”, in NIPS, 2001) have generally employed the notion of target alignment. Target alignment, intuitively, is a measure of similarity (agreement) between a kernel and the target kernel, which is derived from the training labels, and represents the (an) optimal kernel for the training sample.

Notwithstanding advances, methods, structures or techniques that address such aspects would represent a significant advance in the art.

SUMMARY

An advance is made in the art according to the present disclosure directed to systems and methods for two-stage Multiple Kernel Learning (MKL) wherein the similarity/kernel learning problem is formulated as a standard classification problem in a new instance space (the K-space). In this K-space, any classifier (a K-classifier) directly corresponds to a combination of the pre-specified similarity functions/kernels. Thus the problem of finding a “good” similarity function/kernel reduces to finding a “good” classifier in the K-space, a very familiar problem. Advantageously, the framework is very general and allows the use of any classification algorithm to learn a classifier in the K-space, and hence a combination of the pre-specified similarity functions/kernels. Framing the similarity/kernel learning problem in this way has the distinct advantage that it makes it easy to leverage the any past or future advance in binary classification to obtain a better performing and more scalable system that is conceptually simpler, and easier to use and train.

Advantageously, method and systems according to the present disclosure obtaining higher quality similarity functions and kernels that facilitate obtaining higher performance when used with standard learning algorithms, enable faster operation offer a scalability to large-scale data easily by taking advantage of the significant progress in large-scale classification, allow incorporation of past and future advances in binary classification techniques, including but not limited to stochastic gradient descent, sparse learning, semi-supervised learning, transfer learning and finally are simpler and easier to use than prior art methods and systems.

BRIEF DESCRIPTION OF THE DRAWING

A more complete understanding of the present disclosure may be realized by reference to the accompanying drawings in which:

FIG. 1(a) is a schematic block diagram depicting an exemplary method and system operation according to an aspect of the present disclosure;

FIG. 2(a) is a schematic block diagram depicting the learning of the kernel function of FIG. 1(a), according to an aspect of the present disclosure; and

FIG. 3(a) is a schematic block diagram depicting a contemporary computer system that may advantageously execute methods according to the present disclosure.

DETAILED DESCRIPTION

The following merely illustrates the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its spirit and scope.

Furthermore, all examples and conditional language recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor(s) to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently-known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the art that the diagrams herein represent conceptual views of illustrative structures embodying the principles of the invention.

In addition, it will be appreciated by those skilled in art that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

In the claims hereof any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements which performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The invention as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. Applicant thus regards any means which can provide those functionalities as equivalent as those shown herein. Finally, and unless otherwise explicitly specified herein, the drawings are not drawn to scale.

Thus, for example, it will be appreciated by those skilled in the art that the diagrams herein represent conceptual views of illustrative structures embodying the principles of the disclosure.

FIG. 1(a) is a schematic block diagram that provides an overview of a system operation and method according to an aspect of the present disclosure. As shown in that figure, a training data set (box 100) is input along with a set of pre-specified base similarity or kernel functions (box 102). As those skilled in the art will readily appreciate, a similarity function is a function that takes as arguments two data instances and produces a real number that can be interpreted as a measure of how similar the two instances are. Accordingly, a kernel function is a similarity function that has the additional property of being positive semi-definite.

These inputs are processed and used to compute a new similarity or kernel function that is better suited to the problem at hand (box 104). This new similarity function is constructed by combining the input base similarity functions using the techniques shown in detail in FIG. 2(a). Once a new similarity or kernel function is constructed, it can be used, together with the training set, to produce better classifiers using kernel methods such as Support Vector Machines, K-nearest neighbor, perceptron, etc. (boxes 106,110). Alternatively, the new similarity function can be used in non-linear kernel dimensionality reduction algorithms such as Kernel Principal Component Analysis to obtain a better visualization of the data (boxes 108,112).

Turning now to FIG. 2(a), there is shown a schematic block diagram that details the procedure associated with box 104, the steps involved in producing a new similarity or kernel function by leaning an appropriate combination of pre-specified user supplied similarity or kernel functions. With continued reference to that FIG. 2(a), the system receives as input a set of base similarity or kernel functions (box 102) and a training data set (box 100). The training data can contain different forms of supervised feedback including but not limited to full supervision (as exemplified in box 100), semi-supervision or pairwise supervision.

The construction of the K-Space and a K-training set from the inputs are detailed in box 200. The K-Space is a new space of K-instances. The K-instances are constructed from pairs of training instances as follows: from a pair of training instances (X_i,X_j), a K-instance Z_i,j is constructed as: Z_i,j=(K₁(X_i,X_j), K₂(X_i,X_j), . . . , K_p(X_i,X_j)). If available, the training targets Y_i and Y_j corresponding to the training instances X_i and X_j respectively also define the target for the K-instance T_i,j as T_i,j=1 if Y_i is the same as Y_j and T_i,j=0 otherwise. The K-training set is formed from the set of all K-instances that can be constructed from the input training set. Notably, this specific construction of the K-instances is based on base pre-specified similarity functions which heretofore was unknown in the art.

Continuing with our discussion of FIG. 2(a), box 202 details how a new similarity or kernel function is obtained from the K-training set constructed in box 200. First a binary scoring rule H is learned from the K-training set. A new similarity or kernel function K* is obtained from H as follows: K*(X,X′)=H(Z_x,x′)=H([K₁(X,X′),K₂(X,X′), . . . , K_p(X,X′)]).

Of particular significance to the present disclosure, the special construction of the K-space from box 200 is critical for enabling the use of any binary classification technique to construct the binary scoring rule H. Those skilled in the art will readily appreciate that this is a key advantage of systems and methods according to the present disclosure as it allows easy incorporation of past and future advances in binary classification. As noted above, binary classification techniques for combining pre-specified base similarity or kernel functions to obtain a new similarity/kernel function represents a significant advance in the art.

As those skilled in the art will appreciate, systems and methods according to the present disclosure may be implemented on any of a variety of contemporary programmable computer systems such as that depicted in the schematic block diagram of FIG. 3(a).

The foregoing is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description and the attached Appendix A, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein and in the Appendix A are only illustrative of the principles of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the disclosure. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the disclosure.

Claims

1. A computer implemented method of generating a kernel function comprising:

by a computer:

inputting a training data set and a set of pre-specified base similarity functions;

combining the input base similarity functions by: constructing a new space of K-instances from pairs of training instances such that the K-instances are constructed from the pre-specified similarity functions; and

outputting an indication of the generated kernel function so generated.

2. The computer implemented method of claim 1 wherein said constructing the new space K-instances are constructed according to the following:

constructing a K-training set from a set of K-instances where K-instance Zi,j is represented as Zi,j=(K1(Xi,Xj), K2(Xi,Xj),..., Kp(Xi,Xj)) and determined from a pair of training instances (Xi,Xj).

3. The computer implemented method of claim 2 wherein said kernel function is generated from the K-instance training set by:

determining a binary scoring rule H from the K-training set;

determining a new kernel function K* from H as follows: K*(X,X′)=H(Zx,x′)=H([K1(X,X′),K2(X,X′),..., Kp(X,X′)]).