如何实现随机深度，并随机放弃整个卷积层?

Question

我想实现这个想法:https://arxiv.org/abs/1603.09382。其基本思想是在训练过程中基于"keep probs"(保持概率)(如Dropout)放弃Conv2D层。我想我可以用这样一个自定义图层:

class StochasticConv2D(layers.Layer):
def __init__(self, **kwargs):
super(StochasticConv2D, self).__init__()
self.conv2D = layers.Conv2D(**kwargs)
def call(self, inputs, training, keep_prob):
if training and (np.random.uniform() > keep_prob):
return inputs
return self.conv2D(inputs)

当我尝试训练= True时，我得到这个错误:

ValueError: tf.function-decorated function tried to create variables on non-first call.

如果我让它工作，我不太确定如何实现非训练模式。我定义的模型第二次训练=假，并加载在训练中保存的权重?如果我将validation_data传递给model.fit()，那么如何才能"training"呢?在运行验证时设置为false ?

Answer 1

为了随机冻结滤波器，你可以用卷积滤波器的通道数的形状制作一个tf.keras.layers.Dropout层。这里，我们有10:

import tensorflow as tf
import numpy as np
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(10, 3, input_shape=(28, 28, 1)),
tf.keras.layers.Dropout(.5, noise_shape=(1, 1, 1, 10))])
x = np.random.rand(1, 28, 28, 1)
np.max(model(x, training=True), axis=(1, 2))

array([[-0.        , -0.        ,  0.        ,  0.53856176, -0.        ,
-0.        ,  0.16301194, -0.        ,  0.76797724,  0.54769045]],
dtype=float32)

这些是10个卷积滤波器的所有最大值。你会发现其中一半都是0。

要退出一个图层，你可以做类似的事情:

import tensorflow as tf
import numpy as np
conv_dropout_layer = tf.keras.Sequential([
tf.keras.layers.Conv2D(4, 3),
tf.keras.layers.Dropout(.5, noise_shape=(1, 1, 1, 1))])
x = np.random.rand(1, 28, 28, 1)
model(x, training=True)

一半的时间，所有这些重量将被冻结。

要返回卷积结果的恒等式，可以这样做:

import tensorflow as tf
import numpy as np
class StochasticConv2D(tf.keras.layers.Layer):
def __init__(self, filters, kernel_size, **kwargs):
super(StochasticConv2D, self).__init__()
self.filters = filters
self.kernel_size = kernel_size
self.conv2D = tf.keras.layers.Conv2D(filters, kernel_size, padding='SAME', **kwargs)
def call(self, inputs, **kwargs):
coin_toss = tf.random.uniform(())
return tf.cond(tf.greater(.5, coin_toss), lambda: inputs, lambda: self.conv2D(inputs))

x = np.random.rand(1, 7, 7, 10)
s = StochasticConv2D(10, 3)
s(x, training=True).shape

Answer 2

似乎这样做了(先前解决方案的修改版本):

class StochasticConv2D(layers.Layer):
def __init__(self, keep_prob, **kwargs):
super(StochasticConv2D, self).__init__()
self.keep_prob = keep_prob
self.conv2D = layers.Conv2D(**kwargs)
def call(self, inputs):
if keras.backend.learning_phase():
coin_toss = tf.random.uniform(())
return tf.cond(tf.greater(coin_toss, self.keep_prob), lambda: inputs, lambda: self.conv2D(inputs))

return self.conv2D(inputs)

Answer 3

有一个来自tensorflow_addons的StochasticDepth层

import tensorflow_addons as tfa
import numpy as np
import tensorflow as tf
inputs = tf.keras.Input(shape=(28, 28, 1))
residual = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), 
activation="relu",
padding='SAME')(inputs)
x = tfa.layers.StochasticDepth()([inputs, residual])
x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
residual = tf.keras.layers.Conv2D(32, kernel_size=(3, 3), 
activation="relu",
padding='SAME')(x)
x = tfa.layers.StochasticDepth()([x, residual])
x = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(x)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Dropout(0.5)(x)
outputs = tf.keras.layers.Dense(10, 
activation="softmax")(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
model.summary()
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 28, 28, 1).astype("float32") / 255
x_test = x_test.reshape(10000, 28, 28, 1).astype("float32") / 255
model.compile(
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
optimizer=tf.keras.optimizers.RMSprop(),
metrics=["accuracy"],
)
history = model.fit(x_train, y_train, 
batch_size=64, epochs=2, 
validation_split=0.2)