如何使用 Keras 实现基于 sobel 滤波器的自定义损失函数

Question

我是DL和Keras的新手，目前我正在尝试在Keras中实现基于sobel过滤器的自定义损失函数。

这个想法是计算索贝尔滤波预测和索贝尔滤波地面实况图像的均方损失。

到目前为止，我的自定义损失函数如下所示：

from scipy import ndimage
def mse_sobel(y_true, y_pred):
    for i in range (0, y_true.shape[0]):
        dx_true = ndimage.sobel(y_true[i,:,:,:], 1)
        dy_true = ndimage.sobel(y_true[i,:,:,:], 2)
        mag_true[i,:,:,:] = np.hypot(dx_true, dy_true)
        mag_true[i,:,:,:] *= 1.0 / np.max(mag_true[i,:,:,:])
        dx_pred = ndimage.sobel(y_pred[i,:,:,:], 1)
        dy_pred = ndimage.sobel(y_pred[i,:,:,:], 2)
        mag_pred[i,:,:,:] = np.hypot(dx_pred, dy_pred)
        mag_pred[i,:,:,:] *= 1.0 / np.max(mag_pred[i,:,:,:])
    return(K.mean(K.square(mag_pred - mag_true), axis=-1))

使用此损失函数会导致此错误：

in mse_sobel
for i in range (0, y_true.shape[0]):
TypeError: __index__ returned non-int (type NoneType)

使用调试器我发现，y_true.shape只返回None - 很好。但是当我将y_true.shape替换为例如1，使其看起来像这样for i in range (0,1):时，会发生另一个错误：

in sobel
axis = _ni_support._check_axis(axis, input.ndim)
in _check_axis
raise ValueError('invalid axis')
ValueError: invalid axis

在这里，我不确定为什么轴似乎无效？

谁能帮我弄清楚如何实现该损失函数？非常感谢您的帮助！

Answer 1

损失必须通过张量操作、使用 keras 后端或 tensorflow/theano/cntk 函数来实现。这是保持反向传播的唯一方法。使用numpy，scipy等会破坏图形。

让我们导入 keras 后端：

import keras.backend as K

定义过滤器：

#this contains both X and Y sobel filters in the format (3,3,1,2)
#size is 3 x 3, it considers 1 input channel and has two output channels: X and Y
sobelFilter = K.variable([[[[1.,  1.]], [[0.,  2.]],[[-1.,  1.]]],
                      [[[2.,  0.]], [[0.,  0.]],[[-2.,  0.]]],
                      [[[1., -1.]], [[0., -2.]],[[-1., -1.]]]])

在这里，一个为每个输入通道重复过滤器的函数，以防您的图像是 RGB 或具有超过 1 个通道。这将只复制每个输入通道的 sobel 滤波器：(3,3,inputChannels, 2) ：

def expandedSobel(inputTensor):
    #this considers data_format = 'channels_last'
    inputChannels = K.reshape(K.ones_like(inputTensor[0,0,0,:]),(1,1,-1,1))
    #if you're using 'channels_first', use inputTensor[0,:,0,0] above
    return sobelFilter * inputChannels

这是损失函数：

def sobelLoss(yTrue,yPred):
    #get the sobel filter repeated for each input channel
    filt = expandedSobel(yTrue)
    #calculate the sobel filters for yTrue and yPred
    #this generates twice the number of input channels 
    #a X and Y channel for each input channel
    sobelTrue = K.depthwise_conv2d(yTrue,filt)
    sobelPred = K.depthwise_conv2d(yPred,filt)
    #now you just apply the mse:
    return K.mean(K.square(sobelTrue - sobelPred))

在模型中应用此损失：

model.compile(loss=sobelLoss, optimizer = ....)

---

我的经验表明，计算统一的sobel滤波器sqrt(X² + Y²)会带来可怕的结果，并且生成的图像听起来像棋盘。但是，如果您确实想要它：

def squareSobelLoss(yTrue,yPred):
    #same beginning as the other loss
    filt = expandedSobel(yTrue)
    squareSobelTrue = K.square(K.depthwise_conv2d(yTrue,filt))
    squareSobelPred = K.square(K.depthwise_conv2d(yPred,filt))
    #here, since we've got 6 output channels (for an RGB image)
    #let's reorganize in order to easily sum X² and Y²: change (h,w,6) to (h,w,3,2)
    #caution: this method of reshaping only works in tensorflow
    #if you do need this in other backends, let me know
    newShape = K.shape(squareSobelTrue)
    newShape = K.concatenate([newShape[:-1],
                              newShape[-1:]//2,
                              K.variable([2],dtype='int32')])
    #sum the last axis (the one that is 2 above, representing X² and Y²)                      
    squareSobelTrue = K.sum(K.reshape(squareSobelTrue,newShape),axis=-1)
    squareSobelPred = K.sum(K.reshape(squareSobelPred,newShape),axis=-1)
    #since both previous values are already squared, maybe we shouldn't square them again? 
    #but you can apply the K.sqrt() in both, and then make the difference, 
    #and then another square, it's up to you...    
    return K.mean(K.abs(squareSobelTrue - squareSobelPred))