为什么我的模型使用Tenserflow和Keras GPU OOM错误

Question

我正在尝试运行我的模型，但我正在运行一个错误

2021-06-03 01:20:42.015864: W tensorflow/core/common_runtime/bfc_allocator.cc:467] **************************************************************************__________________________
2021-06-03 01:20:42.015984: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at concat_op.cc:158 : Resource exhausted: OOM when allocating tensor with shape[8938,46080] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8938,46080] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:ConcatV2] name: concat

我的代码：

import numpy as np
import tensorflow as tf
from cv2 import cv2
from keras.applications.densenet import preprocess_input
from tensorflow import keras
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras.optimizers import Adam, SGD, RMSprop
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
from tensorflow.keras.preprocessing import image
from tensorflow.keras.models import Model
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import MaxPool2D, MaxPool3D, GlobalAveragePooling2D, Reshape, GlobalMaxPooling2D, MaxPooling2D, Flatten, AveragePooling2D
# physical_devices = tf.config.experimental.list_physical_devices('GPU')
# print("Num GPU Available", len(physical_devices))
# tf.config.experimental.set_memory_growth(physical_devices[0], True)
train_path = 'data/train'
test_path = 'data/test'
batch_size = 16
image_size = (360, 360)
train_batches = ImageDataGenerator(
preprocessing_function=preprocess_input,
# rescale=1./255,
horizontal_flip=True,
rotation_range=.3,
width_shift_range=.2,
height_shift_range=.2,
zoom_range=.2
).flow_from_directory(directory=train_path,
target_size=image_size,
color_mode='rgb',
batch_size=batch_size,
shuffle=True)
test_batches = ImageDataGenerator(
preprocessing_function=preprocess_input
# rescale=1./255
).flow_from_directory(directory=test_path,
target_size=image_size,
color_mode='rgb',
batch_size=batch_size,
shuffle=True)
# mobile = tf.keras.applications.mobilenet.MobileNet()
mobile = tf.keras.applications.mobilenet_v2.MobileNetV2(include_top=False, weights='imagenet', input_shape=(360, 360, 3))
x = MaxPool2D()(mobile.layers[-1].output)
x = Flatten()(x)
model = Model(inputs=mobile.input, outputs=x)
train_features = model.predict(train_batches, train_batches.labels)
test_features = model.predict(test_batches, test_batches.labels)
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
train_scaled = scaler.fit_transform(train_features)
test_scaled = scaler.fit_transform(test_features)
from sklearn.svm import SVC
svm = SVC()
svm.fit(train_scaled, train_batches.labels)
print('train accuracy:')
print(svm.score(train_scaled, train_batches.labels))
print('test accuracy:')
print(svm.score(test_scaled, test_batches.labels))

Answer 1

如果减少batch_size值并不能解决问题。您可以启用cuda统一内存。此处

Answer 2

此错误为内存不足。尝试降低batch_size的值。