错误:DQN要求每个操作都有一个维度的模型

Question

我正在构建一个定义了模型的RL代理：

def build_model(height,width,channels,actions):
model = Sequential()
model.add(Conv2D(32,(8,8),strides=(4,4),activation='relu',input_shape=(3,height,width,channels)))
model.add(Conv2D(64,(4,4),strides=(2,2),activation='relu'))
model.add(Dense(512,activation='relu'))
model.add(Dense(256,activation='relu'))
model.add(Dense(actions,activation='linear'))
return model

然后创建了一个模型：

env = gym.make('SpaceInvaders-v0')
height,width,channels = env.observation_space.shape
actions = env.action_space.n
model = build_model(height,width,channels,actions)

然后定义Agent：

def build_agent(model,actions):
policy = LinearAnnealedPolicy(EpsGreedyQPolicy(),attr='eps',value_max=1,value_min=.1,value_test=-.2, nb_steps=10000)
memory = SequentialMemory(limit=2000,window_length=3)
dqn = DQNAgent(model=model,memory=memory,policy=policy,dueling_type='avg',nb_actions=actions,nb_steps_warmup=1000)

当我尝试建立代理时：

dqn = build_agent(model,actions)

我得到这个错误：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-100-320eef5d391d> in <module>
----> 1 dqn = build_agent(model,actions)
<ipython-input-95-97801a497208> in build_agent(model, actions)
2     policy = LinearAnnealedPolicy(EpsGreedyQPolicy(),attr='eps',value_max=1,value_min=.1,value_test=-.2, nb_steps=10000)
3     memory = SequentialMemory(limit=2000,window_length=3)
----> 4     dqn = DQNAgent(model=model,memory=memory,policy=policy,dueling_type='avg',nb_actions=actions,nb_steps_warmup=1000)
~/Documents/RL/pRL/lib/python3.8/site-packages/rl/agents/dqn.py in __init__(self, model, policy, test_policy, enable_double_dqn, enable_dueling_network, dueling_type, *args, **kwargs)
105         # Validate (important) input.
106         if list(model.output.shape) != list((None, self.nb_actions)):
--> 107             raise ValueError(f'Model output "{model.output}" has invalid shape. DQN expects a model that has one dimension for each action, in this case {self.nb_actions}.')
108 
109         # Parameters.
ValueError: Model output "Tensor("dense_30/BiasAdd:0", shape=(None, 3, 24, 18, 6), dtype=float32)" has invalid shape. DQN expects a model that has one dimension for each action, in this case 6.

我检查了model.output.shape和self.nb_actions:的值

print(model.output_shape)
# (None, 3, 24, 18, 6)
print(actions) # This is self.nb_action
# 6

我不知道"model.output"的形状应该是什么。在这个错误中，它看起来应该是"(None，6("，但我不知道我必须做什么更改才能得到这个形状。

Answer 1

将model.output.shape转换为元组，然后比较if元组(model.output.shape(！=(无，4(

Answer 2

刚刚尝试了所有的方法，并到处寻找答案，请注意，在模型摘要中，输入已经是三维的。因此添加一个平坦层解决了我的问题

def build_model(height,width,channels,actions):
model = Sequential()
model.add(Conv2D(32,(8,8),strides=(4,4),activation='relu',input_shape=(3,height,width,channels)))
model.add(Conv2D(64,(4,4),strides=(2,2),activation='relu'))
model.add(Dense(512,activation='relu'))
model.add(Dense(256,activation='relu'))
model.add(Flatten())    #add it anywhere you want tbh <<<<<<<<<<<<<<<<<<<<
model.add(Dense(actions,activation='linear'))
return model