Advanced Methods for Hyperparameter Tuning So there is more approaches and techniques being offered or proposed basically every year.So, , there are, there is a lot of experimentation. There is a lot of experimental libraries out there also, but I want to talk to you about most promising ones or the ones that people use the most in the industry or out there. Evolutionary computing ones are the ones that I find the most interesting. I've actually taking a class in evolutionary computing when I was doing my masters. And it's a very interesting, , program. Logic. So what you do is you basically approach your whole program while you're trying to optimize something. So basically evolutionary computing school is to do global optimization. You create your initialize, a population, basically like you think of them as populations. So I put in right now, our population is going to be one individual is going to be one setting for our hypertension. So you create a lot of these, , individuals there's in this population and then you evaluate. Based on how much air that they're giving you. And then you are basically going to eliminate some of them randomly, but some of them also based on natural selection. So the worst error ones are going to be eliminated. And then you're going to have some randomization included in there with the mutation. And you're going to do this whole, , approach again, except for the initialization. And you're going to have some selection and then some mutation that is randomization and then themselves selection. Again, until you find some values that are really good for your model, but you don't have to implement it yourself. Of course, there is already a library for that.SK learn deep is one lifted libraries that you can use to, , adapt evolutionary computing techniques in your hyper parameters. Then we have basion search basion search is basically the closest thing to doing zooming in. , what it does is basically tries to understand a probabilistic model of the relationship between the hyper-parameters and the cost function. So let's say. Like the example that I've given you, , this is the cost function. As it gets bluer, this is a lower cost and the red ones are really high costs. And these are the hyper-parameters based on search. Basically tries to understand what the. Relationship looks like what this graphic or the space would look like and then focus on the ones that are, , that it thinks probabilistically thinks that is going to give it the lowest cost. So that's how that one works. , for the libraries that you can use, you can ha you can use spare mint, , or psychic optimize. It also has a. Bayesian search, , implementation. Then we have gradient base search, gradient base search basically takes us trying to find the optimal gray, , hyper-parameters settings problem. Like the learning problem that we have. So, you know, what we did was to initialize. And biases randomly on our network. And then we'll look at the cost and then we see how these parameters are affecting the cost. And then we update this parameters. So basically it does more or less the same thing for the hyper parameters. It looks at how the effect, the costs and tries to update them accordingly to make the cost lover, , There are some libraries that you can use to do try grading best based search, even though the sounds very common sense or really promising, there is not actually that much attention on this. , but of course, if you're interested, you can go and try it out. The library that you can use for that is eight attuned. That is actually from Amazon, AWS. And then we have early stepping based search. , so there are different types of algorithms and early stopping based search, but basically their goal is to stop the on promising runs earlier so that you can spend more resources on the settings that are more promising. Some of the, , examples of algorithms that are in. Class our successive helping, or there is a synchronous success of holidaying or hyper band and hyper band actually is a library also that you can go and try out for yourself today. , there are some other libraries that I haven't mentioned as part of any specific, , algebra. There is HyperAdapt , and then you have coped and hyperized basically built on hyperoptic to you to be used with Keras. You have Talos, , Carol's tuner obviously works with Keras, and then you have SharePoint mostly. , libraries are implementing a bunch of different, , algorithms.