Locally Weighted Learning in Machine Learning: A Customized Approach

2021 Mini meeting Locally weighted learning Camilla Zacche da Silva Jeff Boisvert

Presentation outline Non-stationarity and machine learning Locally weighted learning and modified SVR approach Customized kernel Results and discussion Conclusions References 2

Non-stationarity and machine learning Unsupervised learning Machine learning: if training/test/development set do not belong to the same distribution, then the algorithm is not capable of learning the true underlying relation. Supervised learning Most often applied 3 categories Reinforcement learning [20] A parallel to geostatistics: 3

Locally weighted learning and the modified SVR approach Considering uneven sampling strategies commonly encountered in geological problems, finding an optimal machine learning model is challenging. Global vs. local cost function: giving equal importance to all training samples may not be appropriate. Locally Weighted Support Vector Regression (LWSVR) Support Vector Regression (SVR) ? 1 2 ? 2+?? 1 2 min ?,?,???? ? ??+ ?? 2+? min ?,?,???? ? ??+ ?? ?=1 ?=1 ??= ??? ??= ? (??)2 C corresponds to the penalty imposed on predictions that lie outside the margin 4

Customized Kernel With SVR non-linearities are considered with the kernel trick, which allows the data to be mapped in a high dimensional space. The most common kernel used in SVR is the Gaussian Radial Basis Function ? ?,? = ? ?( ? ?2) Mahalanobis distance Euclidean distance 5

Results and Discussion LWSVR predicted map (Euclidean) LWSVR predicted map (Mahalanobis) Global predicted map Some search artifacts Implementing additional search strategies 10-fold cross validation LWSVR (Mahalanobis) LWSVR (Euclidean) Global SVR 7.86 8.86 12.0 6.38 8.19 10.8 6.83 8.33 11.0 5.30 6.16 6.49 7.13 7.68 10.4 5.02 6.21 6.37 5.03 5.44 8.00 8.17 8.83 9.45 6.89 7.34 8.09 4.96 5.07 12.0 6.36 7.21 9.46 6

Conclusions Local models divide the fitting process, parameters are adjusted locally. This reduces the challenge of finding optimal parameters for the model; LWSVR reduces the MSE compared to a global approach. LWSVR with Euclidean distance reduces by 24% and LWSVR with Mahalanobis distance 32%. Stationarity is defined differently in a Machine Learning context. It is equally important. Distance based on the covariance between samples increases anisotropy in the model, better representing the phenomenon under study. A kernel with the Mahalanobis distance reduces MSE by 11% in 10-fold cross validation. Local models are more demanding computationally and optimization should be considered. There are search artifacts that must be treated considering different search strategies. Test for local optimization of other SVR parameters, such as gamma and epsilon. 7

References [1] M. Rossi, C. Deutsch, Mineral Resource Estimation, Springer, New York, 2014. [2] A neural network approach to geostatistical simulation, Mathematical Geology 26 (1994). [3] I. K. Kapageridis, Application of NNs systems to grade estimation from exploration data, Ph.D. thesis, University of Nottingham, 1999. [4] P. Tahmasebi, A. Hezarkhani, A fast and independent architecture of artificial neural network for permeability prediction, Journal of Petroleum Science and Engineering 86-87 (2012). [5] Spatial prediction of soil organic matter content integrating artificial neural network and ordinary kriging in Tibetan plateau, Ecological Indicators 45 (2014) 184 194. [6] M. Gangappa, C. Mai, P. Sammulal, Techniques for machine learning based spatial data analysis: Research directions, International Journal of Computer Applications 170 (2017) 9 13. [7] M. S. K. A. A. Hiren Maniar, Srikanth Ryali, Machine-learning methods in geoscience. [8] W. M. H. G. G. B. Hengl T, Nussbaum M, Random forest as a generic framework for predictive modeling of spatial and spatio-temporal variables, PeerJ (2018 [9] M. Samson, C. Deustch, Estimation with ml, 2018. [10] M. Samson, C. Deustch, Elliptical radial basis function vs. radial basis functions, 2019. [11] A. Walch, N. Mohajeri, J.-L. Scartezzini, A spatio-temporal model to estimate hourly solar radiation using Extreme Learning Machines, in: EGU General Assembly Conference Abstracts, EGU General Assembly Conference Abstracts. [12] I. M. Galvan, J. M. Garcia, P. Isasi, A lazy learning approach for building classification models, International Journal of Intelligent systems 26 (2011). [13] B. Mirbagheri, A. Alimohammadi, Integration of local and global support vector machines to improve urban growth modeling, International Journal of Geo-information 7 (2018). [14] M. A. W. Atkenson, C. G., S. Schaal, Locally weighted learning, Artificial Intelligence Review 11 (1997). [15] J. Qu, A practical framework to characterize non-stationary regionalized variables, Ph.D. thesis, University of Alberta, 2018. [16] M. J. Pyrcz, C. V. Deustch, Geostatistical Reservoir Modeling, Oxford university Press, New York, 2014. [17] J. B. Boisvert, Geostatistics with locally varying anisotropy, Ph.D. thesis, University of Alberta, 2010. [18] M. Sugiyama, M. Kawanabe, Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation (Adapative Computation and Machine Learning Series), Massachusetts institute of Technology, Cambridge, 2012. [19] V. Vapnik, The nature of statistical learning theory, Springer, New York,1995. [20] Training and Test sets: Splitting data. https://developers.google.com/ 8