带有生成器的笛卡尔乘积

Question

这链接到列表的嵌套字典的笛卡尔乘积

假设我有一个嵌套的dict，其中有表示多个配置的列表，比如：

{'algorithm': ['PPO', 'A2C', 'DQN'],
'env_config': {'env': 'GymEnvWrapper-Atari',
'env_config': {'AtariEnv': {'game': ['breakout', 'pong']}}}

目标是计算嵌套dict内列表的笛卡尔乘积，以获得所有可能的配置。

这就是我目前所得到的：

def product(*args, repeat=1, root=False):
pools = [tuple(pool) for pool in args] * repeat
result = [[]]
for pool in pools:
result = [x+[y] for x in result for y in pool]
print("************************")
print(root)
for r in result:
print(tuple(r))
print("************************")
for prod in result:
yield tuple(prod)

def recursive_cartesian_product(dic, root=True):
# based on https://stackoverflow.com/a/50606871/11051330
# added differentiation between list and entry to protect strings in dicts
# with uneven depth
keys, values = dic.keys(), dic.values()
vals = (recursive_cartesian_product(v, False) if isinstance(v, dict)
else v if isinstance(v, list) else (v,) for v in
values)
print("!", root)
for conf in product(*vals, root=root):
print(conf)
yield dict(zip(keys, conf))

这是相关的输出：

************************
True
('PPO', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}})
('PPO', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('A2C', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}})
('A2C', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('DQN', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}})
('DQN', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
************************
('PPO', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}})
('PPO', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('A2C', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('A2C', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('DQN', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})
('DQN', {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}})

请注意product中的print语句是如何正确工作的，而yield中的print则会失败，并且不会在以后的配置中更改env的值。

Answer 1

事实证明，问题不在上述函数内部，而是在函数外部。生成的conf被传递给一个名为**kwargs的函数，这使生成器出错。

这里有一个快速解决方案：

def recursive_cartesian_product(dic):
# based on https://stackoverflow.com/a/50606871/11051330
# added differentiation between list and entry to protect strings
# yield contains deepcopy. important as use otherwise messes up generator
keys, values = dic.keys(), dic.values()
vals = (recursive_cartesian_product(v) if isinstance(v, dict)
else v if isinstance(v, list) else (v,) for v in
values)
for conf in itertools.product(*vals):
yield deepcopy(dict(zip(keys, conf)))

Answer 2

itertools已经有了product类型：

from itertools import product

d = {'algorithm': ['PPO', 'A2C', 'DQN'],
'env_config': {'env': 'GymEnvWrapper-Atari',
'env_config': {'AtariEnv': {'game': ['breakout', 'pong']}}}
for algo, game in product(d['algorithm'],
d['env_config']['env_config']['AtariEnv']['game']):
print((algo, {'env': 'GymEnvWrapper-Atari', 
'env_config': {'AtariEnv': {'game': game}}})) 

Answer 3

使用itertools.product确实比自己滚动更简单。

如果您不希望env_config发生更改(游戏名称除外(，则无需实现通用递归dict访问者
因此，您只想要具有game名称的algorithms的产品，始终使用AtariEnv，然后：

from itertools import product
possible_configurations = {'algorithm': ['PPO', 'A2C', 'DQN'],
'env_config': {'env': 'GymEnvWrapper-Atari',
'env_config': {'AtariEnv': {'game': ['breakout', 'pong']}}}}
algorithms = tuple(possible_configurations["algorithm"])
games = tuple(
{"env": "GymEnvWrapper-Atari", "env_config": {"AtariEnv": {"game": game_name}}}
for game_name in possible_configurations["env_config"]["env_config"]["AtariEnv"]["game"]
)
factors = (algorithms, games)
for config in product(*factors):
print(config)

如果你喜欢一个通用的解决方案，这里是我的：

from itertools import product
possible_configurations = {'algorithm': ['PPO', 'A2C', 'DQN'],
'env_config': {'env': 'GymEnvWrapper-Atari',
'env_config': {'AtariEnv': {'game': ['breakout', 'pong']}}}}

def product_visitor(obj):
if isinstance(obj, dict):
yield from (
dict(possible_product)
for possible_product in product(
*(
[(key, possible_value) for possible_value in product_visitor(value)]
for key, value in obj.items())))
elif isinstance(obj, list):
for value in obj:
yield from product_visitor(value)
else:  # either a string, a number, a boolean or null (all scalars)
yield obj

configs = tuple(product_visitor(possible_configurations))
print("n".join(map(str, configs)))
assert configs == (
{'algorithm': 'PPO', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}}},
{'algorithm': 'PPO', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}}},
{'algorithm': 'A2C', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}}},
{'algorithm': 'A2C', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}}},
{'algorithm': 'DQN', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'breakout'}}}},
{'algorithm': 'DQN', 'env_config': {'env': 'GymEnvWrapper-Atari', 'env_config': {'AtariEnv': {'game': 'pong'}}}},
)