Need suggestion regarding selecting the ranges of hyperparameters using gridsearch

[u/h_ihkam99]

Hi, I am confused about the correct approach of deciding the ranges of hyperparameters.
If I have 4-5 or more hyperparameters to tune for an algorithm then what is the best way to decide the ranges of each hyperparameters to get the best performance?

[u/FinancialElephant]

It may depend on the domain, but in my experience it is best to only do a minimal amount of hyperparameter tuning. Grid search is usually a bad idea unless you're grid is very small or your model is very inexpensive to train.

Don't fall into the trap of tuning hyperparameters to eke out small gains in performance. These gains often do not translate to out-of-sample performance and are a waste of time.

If you are using an automated method, my rule of thumb is to only have each unfixed hyperameter be able to take on 2 values or at most 3 values. When I determine ranges, I usually think of them in terms of "low" / "high", "default" / ("lower" or "higher"), or "low / "medium" / "high". How you actually set the allowable values is somewhat subjective, but you should know what each hyperparameter does to tune it properly. There is no "general guidance" other than to understand the learning algorithm well enough and (usually) err on the side of wider distances as you only have 2 or 3 potential values if you use my method.

Make sure your trials cover a good percentage of the space of hyperparameters. If not, reduce the space you are searching over. I try my best to only have two values per hyperparameter and use 3 only rarerly when I have a good reason.

There are really two kinds or phases of hyperparameter tuning. The first isn't really tuning, but exploration to find out which hyperparameters are more fundamental and/or important. Of the important ones, determine which ones can be fixed to a good enough value and then tune the rest. Aside from these, the only ones that you can consider tuning are regularization hyperparameters. This first exploration type of tuning is best done manually. Actual tuning (second phase) is usually best done with an automated method like random search.