I want to hyper-parameter optimize multiple time series forecasting models on the same data. I’m using the Optuna Sweeper plugin for Hydra. The different models have different hyper-parameters and therefore different search spaces. At the moment my config file looks like this:
defaults: - datasets: data - models: Ets - override hydra/sweeper: optuna - override hydra/sweeper/sampler: tpe hydra: # launcher: # n_jobs: 10 run: dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S} sweeper: sampler: seed: 123 direction: minimize study_name: main_val storage: null n_trials: 2 n_jobs: 4 search_space: # Ets models.damped_trend: type: categorical choices: - 'True' - 'False' # Theta # models.method: # type: categorical # choices: # - 'additive' # - 'multiplicative'
Now, when I run the main_val.py file with –multirun, I get the optimal hyper-parameters for Ets. Great. But when I want to run the optimization for another model, in this example Theta, I have to manually comment out the search space for Ets and uncomment the search space for Theta. In reality, each model has much more parameters to optimize and I’m working with 10 different models. This makes my config file quite long and confusing and this commenting/uncommenting stuff is both annoying and error-prone.
I would like to import the search space for each model from another yaml file. Is that possible?
I tried the following:
defaults: - datasets: data - models: Ets - search_spaces: Ets - override hydra/sweeper: optuna - override hydra/sweeper/sampler: tpe hydra: # launcher: # n_jobs: 10 run: dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S} sweeper: sampler: seed: 123 direction: minimize study_name: main_val storage: null n_trials: 2 n_jobs: 4 search_space: search_spaces
with the file search_spaces/Ets.yaml looking like this:
models.damped_trend: type: categorical choices: - 'True' - 'False'
But I got the error:
Validation error while composing config: Cannot assign str to Dict[str, Any] full_key: hydra.sweeper.search_space object_type=OptunaSweeperConf
Advertisement
Answer
Here are two options:
- Use a
@package
directive - Use a variable interpolation
In detail:
Using an @package
directive
An @package
directive can be used to place Ets.yaml
in the hydra.sweeper.search_space
package:
defaults: - datasets: data - models: Ets - search_spaces@hydra.sweeper.search_space: Ets - override hydra/sweeper: optuna - override hydra/sweeper/sampler: tpe hydra: run: dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S} sweeper: sampler: seed: 123 direction: minimize study_name: main_val storage: null n_trials: 2 n_jobs: 4
Using a variable interpolation:
A string interpolation can be used to create a reference from hydra.sweeper.search_spaces
to the top-level search_spaces
config.
defaults: - datasets: data - models: Ets - search_spaces: Ets - override hydra/sweeper: optuna - override hydra/sweeper/sampler: tpe hydra: run: dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S} sweeper: sampler: seed: 123 direction: minimize study_name: main_val storage: null n_trials: 2 n_jobs: 4 search_space: ${search_spaces}