The next presentation of the Rényi Institute's Deep Learning seminar will be held by Csáji Balázs Csanád (SZTAKI) at the 23th of June, 16:00 here.

 

Uncertainty Quantification and Kernels: Distribution-Free Inference for Regression and Classification

 

Constructing efficient models from empirical data is one of the central problems of machine learning, system identification and statistics. In many applications, it is also fundamental to quantify the uncertainty of the obtained models (e.g., for risk management or robust optimization). In this talk, we first present a resampling method, called Sign-Perturbed Sums (SPS), which can construct distribution-free confidence regions for linear regression problems. These regions have exact coverage probabilities for any sample size, they are strongly consistent and they have efficient ellipsoidal outer approximations, which can be computed by semidefinite programming. We then generalize this approach to kernel methods and argue that we can build exact confidence regions for ideal representations of functions we only observe through noise. We call a representation ideal, if its outputs coincide with the corresponding (hidden) noise-free outputs of the true underlying function for all available inputs. We demonstrate these ideas on standard kernel methods such as kernel ridge regression, support vector regression and kernelized LASSO. Finally, we study classification problems and aim at estimating the regression function of binary classification. The regression is a fundamental object of classification as it determines both a Bayes optimal classifier and the misclassification probabilities for given inputs. Based on the previous ideas, we introduce a distribution-free resampling framework for classification that can be used for statistical hypothesis testing and for building confidence regions. We demonstrate this framework by proposing several kernel-based algorithms for binary classification, all of which have exact coverage probabilities and are strongly consistent.

 

Joint work with: Campi, M. C. (University of Brescia, Italy); Weyer, E. (University of Melbourne, Australia); Kis, K. B. (SZTAKI) and Tamás, A. (SZTAKI)

 

Relevant publications:

 

1.       Csáji, B. Cs.; Campi, M. C.; Weyer, E.: Sign-Perturbed Sums: A New System Identification Approach for Constructing Exact Non-Asymptotic Confidence Regions in Linear Regression Models, IEEE Transactions on Signal Processing, IEEE Press, Vol. 69, 2015, pp. 169–181.

2.       Weyer, E.; Campi, M.C.; Csáji, B.Cs.: Asymptotic Properties of SPS Confidence Regions, Automatica, Vol. 82, August 2017, pp. 287-294.

3.       Csáji, B.Cs.; Kis, K.B.: Distribution-Free Uncertainty Quantification for Kernel Methods by Gradient Perturbations, Machine Learning, Springer, Vol. 108, No. 8-9, 2019, pp. 1677–1699.

4.       Csáji, B. Cs.; Tamás, A.: Semi-Parametric Uncertainty Bounds for Binary Classification, 58th IEEE Conference on Decision and Control (CDC), Nice, France, December 11–13, 2019, pp. 4427–4432

5.       Tamás, A.; Csáji, B. Cs.: Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings, IEEE Control Systems Letters, 2021