使用者:Jeff1iu/沙盒

計算機科學中,在線機器學習是一種機器學習的方法。在線機器學習的過程中,數據是有順序地一步步被獲取的,並且每一步中新得到的數據被用來更新模型,使模型的表現更佳。這與批量機器學習不同——批量機器學習的過程中,整個訓練數據集可以立刻得到,並直接被用於模型的生成。在機器學習的過程中,當計算能力受限,無法在整個訓練數據集上進行計算時,在線機器學習是一種常用的方法。除此之外,當有必要讓算法動態地對於數據中的新模式作更新時,或者當數據是由關於時間的函數而產生時(例如股票價格預測問題中),在線機器學習也是適用的方法。

簡介

編輯

監督式學習的設定下,目標為學習一個函數 ,其中 為輸入, 為輸出,從而對於服從 上的聯合概率分布 的實例能有較好的預測結果。

In reality, the learner never knows the true distribution   over instances. Instead, the learner usually has access to a training set of examples  . In this setting, the loss function is given as  , such that   measures the difference between the predicted value   and the true value  . The ideal goal is to select a function  , where   is a space of functions called a hypothesis space, so that some notion of total loss is minimised. Depending on the type of model (statistical or adversarial), one can devise different notions of loss, which lead to different learning algorithms.

在線學習的統計觀點

編輯

In statistical learning models, the training sample   are assumed to have been drawn from the true distribution   and the objective is to minimize the expected "risk"

 

A common paradigm in this situation is to estimate a function   through empirical risk minimization or regularized empirical risk minimization (usually Tikhonov regularization). The choice of loss function here gives rise to several well-known learning algorithms such as regularized least squares and support vector machines. A purely online model in this category would learn based on just the new input  , the current best predictor   and some extra stored information (which is usually expected to have storage requirements independent of training data size). For many formulations, for example nonlinear kernel methods, true online learning is not possible, though a form of hybrid online learning with recursive algorithms can be used where   is permitted to depend on   and all previous data points  . In this case, the space requirements are no longer guaranteed to be constant since it requires storing all previous data points, but the solution may take less time to compute with the addition of a new data point, as compared to batch learning techniques.

An common strategy to overcome the above issues is to learn using mini-batches, which process a small batch of   data points at a time, this can be considered as pseudo-online learning for   much smaller than the total number of training points. Mini-batch techniques are used with repeated passing over the training data to obtain optimized out-of-core versions of machine learning algorithms, for e.g. Stochastic gradient descent. When combined with backpropagation, this is currently the de facto training method for training artificial neural networks.

例子:最小二乘法

編輯

The simple example of linear least squares is used to explain a variety of ideas in online learning. The ideas are general enough to be applied to other settings, for e.g. with other convex loss functions.

批量機器學習

編輯

In the setting of supervised learning with the square loss function, the intent is to minimize the empirical loss,

  where
 .

Let   be the   data matrix and   is the   matrix of target values after the arrival of the first   data points. Assuming that the covariance matrix   is invertible (otherwise it is preferential to proceed in a similar fashion with Tikhonov regularization), the best solution   to the linear least squares problem is given by

 .

Now, calculating the covariance matrix   takes time  , inverting the   matrix takes time  , while the rest of the multiplication takes time  , giving a total time of  . When   total points in the dataset and having to recompute the solution after the arrival of every datapoint  , the naive approach will have a total complexity  . Note that when storing the matrix  , then updating it at each step needs only adding  , which takes   time, reducing the total time to  , but with an additional storage space of   to store  .[1]

在線學習:遞歸最小二乘法

編輯

The recursive least squares algorithm considers an online approach to the least squares problem. It can be shown that by initialising   and  , the solution of the linear least squares problem given in the previous section can be computed by the following iteration:

 
 

The above iteration algorithm can be proved using induction on  .[2] The proof also shows that  . One can look at RLS also in the context of adaptive filters (see RLS).

The complexity for   steps of this algorithm is  , which is an order of magnitude faster than the corresponding batch learning complexity. The storage requirements at every step   here are to store the matrix  , which is constant at  . For the case when   is not invertible, consider the regularised version of the problem loss function  . Then, it's easy to show that the same algorithm works with  , and the iterations proceed to give  .[1]

隨機梯度下降(Stochastic gradient descent, SGD)

編輯

When this is replaced,

 

by

 

or   by  , this becomes the stochastic gradient descent algorithm. In this case, the complexity for   steps of this algorithm reduces to  . The storage requirements at every step   are constant at  .

However, the stepsize   needs to be chosen carefully to solve the expected risk minimization problem, as detailed above. By choosing a decaying step size   one can prove the convergence of the average iterate  . This setting is a special case of stochastic optimization, a well known problem in optimization.[1]

增量隨機梯度下降

編輯

In practice, one can perform multiple stochastic gradient passes (also called cycles or epochs) over the data. The algorithm thus obtained is called incremental gradient method and corresponds to an iteration

 

The main difference with the stochastic gradient method is that here a sequence   is chosen to decide which training point is visited in the  -th step. Such a sequence can be stochastic or deterministic. The number of iterations is then decoupled to the number of points (each point can be considered more than once). The incremental gradient method can be shown to provide a minimizer to the empirical risk.[3] Incremental techniques can be advantageous when considering objective functions made up of a sum of many terms e.g. an empirical error corresponding to a very large dataset.[1]

核方法

編輯

Kernels can be used to extend the above algorithms to non-parametric models (or models where the parameters form an infinite dimensional space). The corresponding procedure will no longer be truly online and instead involve storing all the data points, but is still faster than the brute force method. This discussion is restricted to the case of the square loss, though it can be extended to any convex loss. It can be shown by an easy induction [1] that if   is the data matrix and   is the output after   steps of the SGD algorithm, then,

 

where   and the sequence   satisfies the recursion:

 
  and
 

Notice that here   is just the standard Kernel on  , and the predictor is of the form

 .

Now, if a general kernel   is introduced instead and let the predictor be

 

then the same proof will also show that predictor minimising the least squares loss is obtained by changing the above recursion to

 

The above expression requires storing all the data for updating  . The total time complexity for the recursion when evaluating for the  -th datapoint is  , where   is the cost of evaluating the kernel on a single pair of points.[1] Thus, the use of the kernel has allowed the movement from a finite dimensional parameter space   to a possibly infinite dimensional feature represented by a kernel   by instead performing the recursion on the space of parameters  , whose dimension is the same as the size of the training dataset. In general, this is a consequence of the representer theorem.[1]

進步學習

編輯

Progressive learning is an effective learning model which is demonstrated by the human learning process. It is the process of learning continuously from direct experience. Progressive learning technique (PLT) in machine learning can learn new classes/labels dynamically on the run.[4] Though online learning can learn new samples of data that arrive sequentially, they cannot learn new classes of data being introduced to the model. The learning paradigm of progressive learning, is independent of the number of class constraints and it can learn new classes while still retaining the knowledge of previous classes. Whenever a new class (non-native to the knowledge learnt thus far) is encountered, the classifier gets remodeled automatically and the parameters are calculated in such a way that it retains the knowledge learnt thus far. This technique is suitable for real-world applications where the number of classes is often unknown and online learning from real-time data is required.

在線凸優化

編輯

In online convex optimisation (OCO), the hypothesis set and the loss functions are forced to be convex to obtain stronger learning bounds. The modified sequential game is now as follows:

For  

  • Learner receives input  
  • Learner outputs   from a fixed convex set  
  • Nature sends back a convex loss function  .
  • Learner suffers loss   and updates its model

Thus, when regret is minimised, competition against the best weight vector   occurs. As an example, consider the case of online least squares linear regression. Here, the weight vectors come from the convex set  , and nature sends back the convex loss function  . Note here that   is implicitly sent with  .

Some online prediction problems however cannot fit in the framework of OCO. For example, in online classification, the prediction domain and the loss functions are not convex. In such scenarios, two simple techniques for convexification are used: randomisation and surrogate loss functions.

Some simple online convex optimisation algorithms are:

跟隨領導者(Follow the leader, FTL)

編輯

The simplest learning rule to try is to select (at the current step) the hypothesis that has the least loss over all past rounds. This algorithm is called Follow the leader, and is simply given round   by:

 

This method can thus be looked as a greedy algorithm. For the case of online quadratic optimization (where the loss function is  ), one can show a regret bound that grows as  . However, similar bounds cannot be obtained for the FTL algorithm for other important families of models like online linear optimization. To do so, one modifies FTL by adding regularisation.

跟隨正則領導者(Follow the regularised leader, FTRL)

編輯

This is a natural modification of FTL that is used to stabilise the FTL solutions and obtain better regret bounds. A regularisation function   is chosen and learning performed in round t as follows:

 

As a special example, consider the case of online linear optimisation i.e. where nature sends back loss functions of the form  . Also, let  . Suppose the regularisation function   is chosen for some positive number  . Then, one can show that the regret minimising iteration becomes

 

Note that this can be rewritten as  , which looks exactly like online gradient descent.

If S is instead some convex subspace of  , S would need to be projected onto, leading to the modified update rule

 

This algorithm is known as lazy projection, as the vector   accumulates the gradients. It is also known as Nesterov's dual averaging algorithm. In this scenario of linear loss functions and quadratic regularisation, the regret is bounded by  , and thus the average regret goes to 0 as desired.

在線次梯度下降(Online subgradient descent, OSD)

編輯

The above proved a regret bound for linear loss functions  . To generalise the algorithm to any convex loss function, the subgradient   of   is used as a linear approximation to   near  , leading to the online subgradient descent algorithm:

Initialise parameter  

For  

  • Predict using  , receive   from nature.
  • Choose  
  • If  , update as  
  • If  , project cumulative gradients onto   i.e.  

One can use the OSD algorithm to derive   regret bounds for the online version of SVM's for classification, which use the hinge loss 

其他算法

編輯

Quadratically regularised FTRL algorithms lead to lazily projected gradient algorithms as described above. To use the above for arbitrary convex functions and regularisers, one uses online mirror descent. Another algorithm is called prediction with expert advice. In this case, the hypothesis set consists of   functions. A distribution   over the   experts is maintained, and prediction is performed by sampling an expert from this distribution. For the Euclidean regularisation, one can show a regret bound of  , which can be improved further to a   bound by using a better regulariser.

在線學習的闡釋

編輯

The paradigm of online learning interestingly has different interpretations depending on the choice of the learning model, each of which has distinct implications about the predictive quality of the sequence of functions  . The prototypical stochastic gradient descent algorithm is used for this discussion. As noted above, its recursion is given by

 

The first interpretation consider the stochastic gradient descent method as applied to the problem of minimizing the expected risk   defined above.[5] Indeed, in the case of an infinite stream of data, since the examples   are assumed to be drawn i.i.d. from the distribution  , the sequence of gradients of   in the above iteration are an i.i.d. sample of stochastic estimates of the gradient of the expected risk   and therefore one can apply complexity results for the stochastic gradient descent method to bound the deviation  , where   is the minimizer of  .[6] This interpretation is also valid in the case of a finite training set; although with multiple passes through the data the gradients are no longer independent, still complexity results can be obtained in special cases.

The second interpretation applies to the case of a finite training set and considers the SGD algorithm as an instance of incremental gradient descent method.[3] In this case, one instead looks at the empirical risk:

 

Since the gradients of   in the incremental gradient descent iterations are also stochastic estimates of the gradient of  , this interpretation is also related to the stochastic gradient descent method, but applied to minimize the empirical risk as opposed to the expected risk. Since this interpretation concerns the empirical risk and not the expected risk, multiple passes through the data are readily allowed and actually lead to tighter bounds on the deviations  , where   is the minimizer of  .

實現

編輯

參見

編輯

參考文獻

編輯
  1. ^ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 L. Rosasco, T. Poggio, Machine Learning: a Regularization Approach, MIT-9.520 Lectures Notes, Manuscript, Dec. 2015. Chapter 7 - Online Learning
  2. ^ Yin, Harold J. Kushner, G. George. Stochastic approximation and recursive algorithms and applications Second edition. New York: Springer. 2003: 8–12. ISBN 978-0-387-21769-7. 
  3. ^ 3.0 3.1 Bertsekas, D. P. (2011). Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. Optimization for Machine Learning, 85.
  4. ^ Venkatesan, Rajasekar; Meng Joo, Er. A novel progressive learning technique for multi-class classification. Neurocomputing. 2016, 207: 310–321. doi:10.1016/j.neucom.2016.05.006. 
  5. ^ Bottou, Léon. Online Algorithms and Stochastic Approximations. Online Learning and Neural Networks. Cambridge University Press. 1998. ISBN 978-0-521-65263-6. 
  6. ^ Stochastic Approximation Algorithms and Applications, Harold J. Kushner and G. George Yin, New York: Springer-Verlag, 1997. ISBN 0-387-94916-X; 2nd ed., titled Stochastic Approximation and Recursive Algorithms and Applications, 2003, ISBN 0-387-00894-2.

外部連結

編輯