Bayesian Optimization Let's start with an example for neural networks. If you have a neural network, you hardly know what is happening inside, but what you want to achieve is to optimize, for example, the precision that it finds stuff. So, what you need to do normally is you have different hyperparameters like the learning rate, batch size, and many others that you can tune. Before you start the training to optimize the precision, if you look in more general on this problem we just can call it we have a black box problem where we don't know 100% what is really happening inside. We have a lot of different input variables and what we try to do is to optimize the target value, so what's coming out of the black box problem. And this is exactly where Bayesian optimization is really well-suited to help you to find most efficiently or really efficiently the best target solution. How it works is an iterative process in this case. So, we have five steps that are partially repeated iteratively. I'm going to explain every step afterwards with an example in detail, but yeah, first you have an initial sampling set. So, you need to start with something, so you start with the initial sampling. After having this sampling, you evaluate all the samplings out of the initial samplings with the black box problem. For example, before, you would have different training runs for neural networks to see how they perform based on their hyperparameters. Based on these results, you can start training a Gaussian process regressor. What this means in detail, we get to this later. Based on these results, you do a calculation of an acquisition function and you use this acquisition function, the last step, to identify the next evaluation input. You try to minimize that function and see which evaluation you are going to do next in the black box problem. Then you start the process over and over again until a certain criterion is met. Let's start with a simple example where we have a black box problem, where we only have one input, which is allowed to be in a range between 0 and 10, and we have also one target that we want to optimize. In this case, I take a mathematical function just for you to see clearly the conditions and that we can later see how good the optimization was. So, we take the input and we multiply it with the sine of the input. Let's take a look. I said we start with the initial sampling, which in this case, you have a lot of different possibilities to do initial samplings, like Latin hypercube sampling, guessing, or just, yeah, a random crop. So, you don't have one option that is mandatory. I just took a random crop here and have five samples, which, when I evaluate them one after the other, have different target values, like you can see here. Based on these values, we now have the correct input values, and based on them, we can now train our Gaussian process regressor. It looks like this. So, we have now two different indicators here. I'm going to explain them to you. The difference in Gaussian process regressors is that you don't train just one regression function, but rather train a set of a lot of different tuned regression functions with different kernels and different tails. What you do is the blue line is the mean of all predictions of all functions, while the yellow area indicates the uncertainty of the model and is the standard deviation of all models and their predictions. You can see here, obviously, when we don't have noise at all points, when we have a sample, there's no uncertainty. However, the more the points are away from each other, the uncertainty rises. In the next step, we now have our Gaussian process regressor, and we start to do our acquisition function. What is an acquisition function? Basically, you can have a lot of different approaches, but it's somehow a mathematical function describing a gain or potential optimization volume by a function. In this case, I took a very common one called the lower confidence bound (some know it as the upper confidence bound). What it says is that the acquisition function means we take the normal standard, we take the mean (the blue line), and we take from that the standard deviation times kappa. Kappa, in this case, is a hyperparameter. So, depending on how I choose this kappa, my optimization is going to be more locally focused or more globally focused. At this point, I just want to let you know that we are talking about a minimization problem (I forgot to mention this earlier). Our goal is to get the target as small as possible. I did the same acquisition function on the right, taking a value of kappa equal to 10, just for you to get a first feeling of how it looks like. What you can see is that, depending on how big my kappa is, the more my uncertainty gains in value. At this point, for example, we see that for both cases, we more or less sample at the value between 4 and 6. But still, for the kappa that is 10, the value is more or less between 5 or 6, and for the kappa with 1, it is nearer to 4. Sampling now these two values will lead us to a new point, and we start our iteration.