I want to build a brute-force approach that tests all actions in a Gym action space before selecting the best one. Is there any simple, straight-forward way to get all possible actions?
Specifically, my action space is
import gym action_space = gym.spaces.MultiDiscrete([5 for _ in range(4)])
I know I can sample a random action with action_space.sample() and also check if an action is contained in the action space, but I want to generate a list of all possible action within that space.
Is there anything more elegant (and performant) than just a bunch of for loops? The problem with for loops is that I want it to work with any size of action space, so I cannot hard-code 4 for loops to walk through the different actions.
Advertisement
Answer
The actions in a gym environment are usually represented by integers only, this mean if you get the total number of possible actions, then an array of all possible actions can be created.
The way to get the total number of possible actions in a gym environment depends on the type of action space it has, for your case it’s a MultiDiscrete action space and thus the attribute nvec can be used as mentioned here by @Valentin Macé like so -:
>> print(env.action_space.nvec) array([5, 5, 5, 5], dtype=int64)
Note that the attribute nvec stands for n vector, since its output is a multidimensional vector. Also note that the attribute is a numpy array.
Now that we have the array to convert it into a list of lists of actions assuming that since the action_space.sample function returns a numpy array of a random function from each of the dimensions of the MultiDiscrete action_space i.e. -:
>> env.action_space.sample() # This does not return a single action but 4 actions for your case since you have a multi discrete action space of length 4. array([2, 2, 0, 1], dtype=int64)
So thus to convert the array to a list of lists of possible actions in each dimensions we can use list comprehensions like so -:
>> [list(range(1, (k + 1))) for k in action_space.nvec] [[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]
Note that this is scalable to any number of dimensions and is also quite efficient performance wise.
Now you can loop over the possible actions in each dimension using only two loops like so -:
possible_actions = [list(range(1, (k + 1))) for k in action_space.nvec]
for action_dim in possible_actions :
for action in action_dim :
# Find best action.....
pass
For more info about the same I would like you to also visit this thread on github, with a somewhat similar issue being discussed incase you find the same useful.
EDIT: So as per the comment of yours @CGFoX I assume that you want it such that all the possible combination vectors of the actions can be generated as a list for any number of dimensions, somewhat like so -:
>> get_actions() [[1, 1, 1, 1], [1, 1, 1, 2] ....] # For all possible combinations.
The same can be achieved like so using recursion and with only two loops, this is also expandable to as many dimensions as provided.
def flatten(actions) :
# This function flattens any actions passed somewhat like so -:
# INPUT -: [[1, 2, 3], 4, 5]
# OUTPUT -: [1, 2, 3, 4, 5]
new_actions = [] # Initializing the new flattened list of actions.
for action in actions :
# Loop through the actions
if type(action) == list :
# If any actions is a pair of actions i.e. a list e.g. [1, 1] then
# add it's elements to the new_actions list.
new_actions += action
elif type(action) == int :
# If the action is an integer then append it directly to the new_actions
# list.
new_actions.append(action)
# Returns the new_actions list generated.
return new_actions
def get_actions(possible_actions) :
# This functions recieves as input the possibilities of actions for every dimension
# and returns all possible dimensional combinations for the same.
# Like so -:
# INPUT-: [[1, 2, 3, 4], [1, 2, 3, 4]] # Example for 2 dimensions but can be scaled for any.
# OUTPUT-: [[1, 1], [1, 2], [1, 3] ... [4, 1] ... [4, 4]]
if len(possible_actions) == 1 :
# If there is only one possible list of actions then it itself is the
# list containing all possible combinations and thus is returned.
return possible_actions
pairs = [] # Initializing a list to contain all pairs of actions generated.
for action in possible_actions[0] :
# Now we loop over the first set of possibilities of actions i.e. index 0
# and we make pairs of it with the second set i.e. index 1, appending each pair
# to the pairs list.
# NOTE: Incase the function is recursively called the first set of possibilities
# of actions may contain vectors and thus the newly formed pair has to be flattened.
# i.e. If a pair has already been made in previous generation like so -:
# [[[1, 1], [2, 2], [3, 3] ... ], [1, 2, 3, 4]]
# Then the pair formed will be this -: [[[1, 1], 1], [[1, 1], 2] ... ]
# But we want them to be flattened like so -: [[1, 1, 1], [1, 1, 2] ... ]
for action2 in possible_actions[1] :
pairs.append(flatten([action, action2]))
# Now we create a new list of all possible set of actions by combining the
# newly generated pairs and the sets of possibilities of actions that have not
# been paired i.e. sets other than the first and the second.
# NOTE: When we made pairs we did so only for the first two indexes and not for
# all thus to do so we make a new list with the sets that remained unpaired
# and the paired set. i.e.
# BEFORE PAIRING -: [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]
# AFTER PAIRING -: [[[1, 1], [1, 2] ... ], [1, 2, 3, 4]] # Notice how the third set
# i.e. the index 2 is still unpaired and first two sets have been paired.
new_possible_actions = [pairs] + possible_actions[2 : ]
# Now we recurse the function and call it within itself to make pairs for the
# left out sets, Note that since the first two sets were combined to form a paired
# first set now this set will be paired with the third set.
# This recursion will keep happening until all the sets have been paired to form
# a single set with all possible combinations.
possible_action_vectors = get_actions(new_possible_actions)
# Finally the result of the recursion is returned.
# NOTE: Only the first index is returned since now the first index contains the
# paired set of actions.
return possible_action_vectors[0]
Once we have this function defined it can be used with our previously generated set of possibilities of actions to get all possible combinations like so -:
possible_actions = [list(range(1, (k + 1))) for k in action_space.nvec] print(get_actions(possible_actions)) >> [[1, 1, 1, 1], [1, 1, 1, 2], [1, 1, 1, 3], [1, 1, 1, 4], [1, 1, 1, 5], `[1, 1, 2, 1], [1, 1, 2, 2], [1, 1, 2, 3], [1, 1, 2, 4], [1, 1, 2, 5], [1, 1, 3, 1], [1, 1, 3, 2], [1, 1, 3, 3], [1, 1, 3, 4], [1, 1, 3, 5], [1, 1, 4, 1], [1, 1, 4, 2], [1, 1, 4, 3], [1, 1, 4, 4], [1, 1, 4, 5], [1, 1, 5, 1], [1, 1, 5, 2], [1, 1, 5, 3], [1, 1, 5, 4], [1, 1, 5, 5], [1, 2, 1, 1], [1, 2, 1, 2], [1, 2, 1, 3], [1, 2, 1, 4], [1, 2, 1, 5], [1, 2, 2, 1], [1, 2, 2, 2], [1, 2, 2, 3], [1, 2, 2, 4], [1, 2, 2, 5], [1, 2, 3, 1], [1, 2, 3, 2], [1, 2, 3, 3], [1, 2, 3, 4], [1, 2, 3, 5], [1, 2, 4, 1], [1, 2, 4, 2], [1, 2, 4, 3], [1, 2, 4, 4], [1, 2, 4, 5], [1, 2, 5, 1], [1, 2, 5, 2], [1, 2, 5, 3], [1, 2, 5, 4], [1, 2, 5, 5], [1, 3, 1, 1], [1, 3, 1, 2], [1, 3, 1, 3], [1, 3, 1, 4], [1, 3, 1, 5], [1, 3, 2, 1], [1, 3, 2, 2], [1, 3, 2, 3], [1, 3, 2, 4], [1, 3, 2, 5], [1, 3, 3, 1], [1, 3, 3, 2], [1, 3, 3, 3], [1, 3, 3, 4], [1, 3, 3, 5], [1, 3, 4, 1], [1, 3, 4, 2], [1, 3, 4, 3], [1, 3, 4, 4], [1, 3, 4, 5], [1, 3, 5, 1], [1, 3, 5, 2], [1, 3, 5, 3], [1, 3, 5, 4], [1, 3, 5, 5], [1, 4, 1, 1], [1, 4, 1, 2], [1, 4, 1, 3], [1, 4, 1, 4], [1, 4, 1, 5], [1, 4, 2, 1], [1, 4, 2, 2], [1, 4, 2, 3], [1, 4, 2, 4], [1, 4, 2, 5], [1, 4, 3, 1], [1, 4, 3, 2], [1, 4, 3, 3], [1, 4, 3, 4], [1, 4, 3, 5], [1, 4, 4, 1], [1, 4, 4, 2], [1, 4, 4, 3], [1, 4, 4, 4], [1, 4, 4, 5], [1, 4, 5, 1], [1, 4, 5, 2], [1, 4, 5, 3], [1, 4, 5, 4], [1, 4, 5, 5], [1, 5, 1, 1], [1, 5, 1, 2], [1, 5, 1, 3], [1, 5, 1, 4], [1, 5, 1, 5], [1, 5, 2, 1], [1, 5, 2, 2], [1, 5, 2, 3], [1, 5, 2, 4], [1, 5, 2, 5], [1, 5, 3, 1], [1, 5, 3, 2], [1, 5, 3, 3], [1, 5, 3, 4], [1, 5, 3, 5], [1, 5, 4, 1], [1, 5, 4, 2], [1, 5, 4, 3], [1, 5, 4, 4], [1, 5, 4, 5], [1, 5, 5, 1], [1, 5, 5, 2], [1, 5, 5, 3], [1, 5, 5, 4], [1, 5, 5, 5], [2, 1, 1, 1], [2, 1, 1, 2], [2, 1, 1, 3], [2, 1, 1, 4], [2, 1, 1, 5], [2, 1, 2, 1], [2, 1, 2, 2], [2, 1, 2, 3], [2, 1, 2, 4], [2, 1, 2, 5], [2, 1, 3, 1], [2, 1, 3, 2], [2, 1, 3, 3], [2, 1, 3, 4], [2, 1, 3, 5], [2, 1, 4, 1], [2, 1, 4, 2], [2, 1, 4, 3], [2, 1, 4, 4], [2, 1, 4, 5], [2, 1, 5, 1], [2, 1, 5, 2], [2, 1, 5, 3], [2, 1, 5, 4], [2, 1, 5, 5], [2, 2, 1, 1], [2, 2, 1, 2], [2, 2, 1, 3], [2, 2, 1, 4], [2, 2, 1, 5], [2, 2, 2, 1], [2, 2, 2, 2], [2, 2, 2, 3], [2, 2, 2, 4], [2, 2, 2, 5], [2, 2, 3, 1], [2, 2, 3, 2], [2, 2, 3, 3], [2, 2, 3, 4], [2, 2, 3, 5], [2, 2, 4, 1], [2, 2, 4, 2], [2, 2, 4, 3], [2, 2, 4, 4], [2, 2, 4, 5], [2, 2, 5, 1], [2, 2, 5, 2], [2, 2, 5, 3], [2, 2, 5, 4], [2, 2, 5, 5], [2, 3, 1, 1], [2, 3, 1, 2], [2, 3, 1, 3], [2, 3, 1, 4], [2, 3, 1, 5], [2, 3, 2, 1], [2, 3, 2, 2], [2, 3, 2, 3], [2, 3, 2, 4], [2, 3, 2, 5], [2, 3, 3, 1], [2, 3, 3, 2], [2, 3, 3, 3], [2, 3, 3, 4], [2, 3, 3, 5], [2, 3, 4, 1], [2, 3, 4, 2], [2, 3, 4, 3], [2, 3, 4, 4], [2, 3, 4, 5], [2, 3, 5, 1], [2, 3, 5, 2], [2, 3, 5, 3], [2, 3, 5, 4], [2, 3, 5, 5], [2, 4, 1, 1], [2, 4, 1, 2], [2, 4, 1, 3], [2, 4, 1, 4], [2, 4, 1, 5], [2, 4, 2, 1], [2, 4, 2, 2], [2, 4, 2, 3], [2, 4, 2, 4], [2, 4, 2, 5], [2, 4, 3, 1], [2, 4, 3, 2], [2, 4, 3, 3], [2, 4, 3, 4], [2, 4, 3, 5], [2, 4, 4, 1], [2, 4, 4, 2], [2, 4, 4, 3], [2, 4, 4, 4], [2, 4, 4, 5], [2, 4, 5, 1], [2, 4, 5, 2], [2, 4, 5, 3], [2, 4, 5, 4], [2, 4, 5, 5], [2, 5, 1, 1], [2, 5, 1, 2], [2, 5, 1, 3], [2, 5, 1, 4], [2, 5, 1, 5], [2, 5, 2, 1], [2, 5, 2, 2], [2, 5, 2, 3], [2, 5, 2, 4], [2, 5, 2, 5], [2, 5, 3, 1], [2, 5, 3, 2], [2, 5, 3, 3], [2, 5, 3, 4], [2, 5, 3, 5], [2, 5, 4, 1], [2, 5, 4, 2], [2, 5, 4, 3], [2, 5, 4, 4], [2, 5, 4, 5], [2, 5, 5, 1], [2, 5, 5, 2], [2, 5, 5, 3], [2, 5, 5, 4], [2, 5, 5, 5], [3, 1, 1, 1], [3, 1, 1, 2], [3, 1, 1, 3], [3, 1, 1, 4], [3, 1, 1, 5], [3, 1, 2, 1], [3, 1, 2, 2], [3, 1, 2, 3], [3, 1, 2, 4], [3, 1, 2, 5], [3, 1, 3, 1], [3, 1, 3, 2], [3, 1, 3, 3], [3, 1, 3, 4], [3, 1, 3, 5], [3, 1, 4, 1], [3, 1, 4, 2], [3, 1, 4, 3], [3, 1, 4, 4], [3, 1, 4, 5], [3, 1, 5, 1], [3, 1, 5, 2], [3, 1, 5, 3], [3, 1, 5, 4], [3, 1, 5, 5], [3, 2, 1, 1], [3, 2, 1, 2], [3, 2, 1, 3], [3, 2, 1, 4], [3, 2, 1, 5], [3, 2, 2, 1], [3, 2, 2, 2], [3, 2, 2, 3], [3, 2, 2, 4], [3, 2, 2, 5], [3, 2, 3, 1], [3, 2, 3, 2], [3, 2, 3, 3], [3, 2, 3, 4], [3, 2, 3, 5], [3, 2, 4, 1], [3, 2, 4, 2], [3, 2, 4, 3], [3, 2, 4, 4], [3, 2, 4, 5], [3, 2, 5, 1], [3, 2, 5, 2], [3, 2, 5, 3], [3, 2, 5, 4], [3, 2, 5, 5], [3, 3, 1, 1], [3, 3, 1, 2], [3, 3, 1, 3], [3, 3, 1, 4], [3, 3, 1, 5], [3, 3, 2, 1], [3, 3, 2, 2], [3, 3, 2, 3], [3, 3, 2, 4], [3, 3, 2, 5], [3, 3, 3, 1], [3, 3, 3, 2], [3, 3, 3, 3], [3, 3, 3, 4], [3, 3, 3, 5], [3, 3, 4, 1], [3, 3, 4, 2], [3, 3, 4, 3], [3, 3, 4, 4], [3, 3, 4, 5], [3, 3, 5, 1], [3, 3, 5, 2], [3, 3, 5, 3], [3, 3, 5, 4], [3, 3, 5, 5], [3, 4, 1, 1], [3, 4, 1, 2], [3, 4, 1, 3], [3, 4, 1, 4], [3, 4, 1, 5], [3, 4, 2, 1], [3, 4, 2, 2], [3, 4, 2, 3], [3, 4, 2, 4], [3, 4, 2, 5], [3, 4, 3, 1], [3, 4, 3, 2], [3, 4, 3, 3], [3, 4, 3, 4], [3, 4, 3, 5], [3, 4, 4, 1], [3, 4, 4, 2], [3, 4, 4, 3], [3, 4, 4, 4], [3, 4, 4, 5], [3, 4, 5, 1], [3, 4, 5, 2], [3, 4, 5, 3], [3, 4, 5, 4], [3, 4, 5, 5], [3, 5, 1, 1], [3, 5, 1, 2], [3, 5, 1, 3], [3, 5, 1, 4], [3, 5, 1, 5], [3, 5, 2, 1], [3, 5, 2, 2], [3, 5, 2, 3], [3, 5, 2, 4], [3, 5, 2, 5], [3, 5, 3, 1], [3, 5, 3, 2], [3, 5, 3, 3], [3, 5, 3, 4], [3, 5, 3, 5], [3, 5, 4, 1], [3, 5, 4, 2], [3, 5, 4, 3], [3, 5, 4, 4], [3, 5, 4, 5], [3, 5, 5, 1], [3, 5, 5, 2], [3, 5, 5, 3], [3, 5, 5, 4], [3, 5, 5, 5], [4, 1, 1, 1], [4, 1, 1, 2], [4, 1, 1, 3], [4, 1, 1, 4], [4, 1, 1, 5], [4, 1, 2, 1], [4, 1, 2, 2], [4, 1, 2, 3], [4, 1, 2, 4], [4, 1, 2, 5], [4, 1, 3, 1], [4, 1, 3, 2], [4, 1, 3, 3], [4, 1, 3, 4], [4, 1, 3, 5], [4, 1, 4, 1], [4, 1, 4, 2], [4, 1, 4, 3], [4, 1, 4, 4], [4, 1, 4, 5], [4, 1, 5, 1], [4, 1, 5, 2], [4, 1, 5, 3], [4, 1, 5, 4], [4, 1, 5, 5], [4, 2, 1, 1], [4, 2, 1, 2], [4, 2, 1, 3], [4, 2, 1, 4], [4, 2, 1, 5], [4, 2, 2, 1], [4, 2, 2, 2], [4, 2, 2, 3], [4, 2, 2, 4], [4, 2, 2, 5], [4, 2, 3, 1], [4, 2, 3, 2], [4, 2, 3, 3], [4, 2, 3, 4], [4, 2, 3, 5], [4, 2, 4, 1], [4, 2, 4, 2], [4, 2, 4, 3], [4, 2, 4, 4], [4, 2, 4, 5], [4, 2, 5, 1], [4, 2, 5, 2], [4, 2, 5, 3], [4, 2, 5, 4], [4, 2, 5, 5], [4, 3, 1, 1], [4, 3, 1, 2], [4, 3, 1, 3], [4, 3, 1, 4], [4, 3, 1, 5], [4, 3, 2, 1], [4, 3, 2, 2], [4, 3, 2, 3], [4, 3, 2, 4], [4, 3, 2, 5], [4, 3, 3, 1], [4, 3, 3, 2], [4, 3, 3, 3], [4, 3, 3, 4], [4, 3, 3, 5], [4, 3, 4, 1], [4, 3, 4, 2], [4, 3, 4, 3], [4, 3, 4, 4], [4, 3, 4, 5], [4, 3, 5, 1], [4, 3, 5, 2], [4, 3, 5, 3], [4, 3, 5, 4], [4, 3, 5, 5], [4, 4, 1, 1], [4, 4, 1, 2], [4, 4, 1, 3], [4, 4, 1, 4], [4, 4, 1, 5], [4, 4, 2, 1], [4, 4, 2, 2], [4, 4, 2, 3], [4, 4, 2, 4], [4, 4, 2, 5], [4, 4, 3, 1], [4, 4, 3, 2], [4, 4, 3, 3], [4, 4, 3, 4], [4, 4, 3, 5], [4, 4, 4, 1], [4, 4, 4, 2], [4, 4, 4, 3], [4, 4, 4, 4], [4, 4, 4, 5], [4, 4, 5, 1], [4, 4, 5, 2], [4, 4, 5, 3], [4, 4, 5, 4], [4, 4, 5, 5], [4, 5, 1, 1], [4, 5, 1, 2], [4, 5, 1, 3], [4, 5, 1, 4], [4, 5, 1, 5], [4, 5, 2, 1], [4, 5, 2, 2], [4, 5, 2, 3], [4, 5, 2, 4], [4, 5, 2, 5], [4, 5, 3, 1], [4, 5, 3, 2], [4, 5, 3, 3], [4, 5, 3, 4], [4, 5, 3, 5], [4, 5, 4, 1], [4, 5, 4, 2], [4, 5, 4, 3], [4, 5, 4, 4], [4, 5, 4, 5], [4, 5, 5, 1], [4, 5, 5, 2], [4, 5, 5, 3], [4, 5, 5, 4], [4, 5, 5, 5], [5, 1, 1, 1], [5, 1, 1, 2], [5, 1, 1, 3], [5, 1, 1, 4], [5, 1, 1, 5], [5, 1, 2, 1], [5, 1, 2, 2], [5, 1, 2, 3], [5, 1, 2, 4], [5, 1, 2, 5], [5, 1, 3, 1], [5, 1, 3, 2], [5, 1, 3, 3], [5, 1, 3, 4], [5, 1, 3, 5], [5, 1, 4, 1], [5, 1, 4, 2], [5, 1, 4, 3], [5, 1, 4, 4], [5, 1, 4, 5], [5, 1, 5, 1], [5, 1, 5, 2], [5, 1, 5, 3], [5, 1, 5, 4], [5, 1, 5, 5], [5, 2, 1, 1], [5, 2, 1, 2], [5, 2, 1, 3], [5, 2, 1, 4], [5, 2, 1, 5], [5, 2, 2, 1], [5, 2, 2, 2], [5, 2, 2, 3], [5, 2, 2, 4], [5, 2, 2, 5], [5, 2, 3, 1], [5, 2, 3, 2], [5, 2, 3, 3], [5, 2, 3, 4], [5, 2, 3, 5], [5, 2, 4, 1], [5, 2, 4, 2], [5, 2, 4, 3], [5, 2, 4, 4], [5, 2, 4, 5], [5, 2, 5, 1], [5, 2, 5, 2], [5, 2, 5, 3], [5, 2, 5, 4], [5, 2, 5, 5], [5, 3, 1, 1], [5, 3, 1, 2], [5, 3, 1, 3], [5, 3, 1, 4], [5, 3, 1, 5], [5, 3, 2, 1], [5, 3, 2, 2], [5, 3, 2, 3], [5, 3, 2, 4], [5, 3, 2, 5], [5, 3, 3, 1], [5, 3, 3, 2], [5, 3, 3, 3], [5, 3, 3, 4], [5, 3, 3, 5], [5, 3, 4, 1], [5, 3, 4, 2], [5, 3, 4, 3], [5, 3, 4, 4], [5, 3, 4, 5], [5, 3, 5, 1], [5, 3, 5, 2], [5, 3, 5, 3], [5, 3, 5, 4], [5, 3, 5, 5], [5, 4, 1, 1], [5, 4, 1, 2], [5, 4, 1, 3], [5, 4, 1, 4], [5, 4, 1, 5], [5, 4, 2, 1], [5, 4, 2, 2], [5, 4, 2, 3], [5, 4, 2, 4], [5, 4, 2, 5], [5, 4, 3, 1], [5, 4, 3, 2], [5, 4, 3, 3], [5, 4, 3, 4], [5, 4, 3, 5], [5, 4, 4, 1], [5, 4, 4, 2], [5, 4, 4, 3], [5, 4, 4, 4], [5, 4, 4, 5], [5, 4, 5, 1], [5, 4, 5, 2], [5, 4, 5, 3], [5, 4, 5, 4], [5, 4, 5, 5], [5, 5, 1, 1], [5, 5, 1, 2], [5, 5, 1, 3], [5, 5, 1, 4], [5, 5, 1, 5], [5, 5, 2, 1], [5, 5, 2, 2], [5, 5, 2, 3], [5, 5, 2, 4], [5, 5, 2, 5], [5, 5, 3, 1], [5, 5, 3, 2], [5, 5, 3, 3], [5, 5, 3, 4], [5, 5, 3, 5], [5, 5, 4, 1], [5, 5, 4, 2], [5, 5, 4, 3], [5, 5, 4, 4], [5, 5, 4, 5], [5, 5, 5, 1], [5, 5, 5, 2], [5, 5, 5, 3], [5, 5, 5, 4], [5, 5, 5, 5]]
EDIT-2 : I have fixed some code which previously was returning a nested list, now the list returned is the one with the pairs and is not nested within another list.
EDIT-3-: Fixed my spelling mistakes.