我正在使用Keras，Tensorflow GPU后端和CUDA 9.1构建一个图像分类系统，运行在Ubuntu 18.04上。
我正在使用一个非常大的图像数据集，其中包含 120 万张图像、15k 类，大小为 335 GB。
我可以毫无问题地在 90,000 张图像上训练我的网络。 但是，当我放大并使用包含 120 万张图像的整个数据集时，我得到如下所示的错误，我认为这与内存不足有关。
我使用的是具有11GB内存的GeForce GTX 1080，我有128GB的RAM，300GB的交换文件和具有16个内核的AMD Threadripper 1950X。
我遵循给出的建议来解决类似的问题。 我现在使用更小的批量大小 10 甚至更小，以及更小的致密内层 256，在第一个训练时期开始之前，我仍然收到如下所示的相同错误。
[更新]：我发现内存错误发生在 VGG16predict_generator呼叫期间，甚至在我的网络构建或训练之前。 请参阅下面的代码。
首先，警告和错误：
2018-05-19 20:24:01.255788: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 5635855360 bytes on host: CUresult(304)
2018-05-19 20:24:01.255850: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 5635855360
然后例外：
2018-05-19 13:56:40.472404: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 68719476736
InUse: 15548829696
MaxInUse: 15548829696
NumAllocs: 15542
MaxAllocSize: 16777216
2018-05-19 13:56:40.472563: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
Traceback (most recent call last):
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "bottleneck.py", line 37, in <module>
bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 2510, in predict_generator
outs = self.predict_on_batch(x)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 1945, in predict_on_batch
outputs = self.predict_function(ins)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2478, in __call__
**self.session_kwargs)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
[[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
这是我的代码：
import numpy as np
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.layers import Dropout, Flatten, Dense
from keras.models import Sequential
from keras.preprocessing.image import ImageDataGenerator
from keras import applications
from keras.utils.np_utils import to_categorical
import matplotlib.pyplot as plt
# Dimensions of our images.
img_width, img_height = 224, 224
train_data_dir = './train_sample'
epochs = 100
batch_size = 10
# Data preprocessing
# Pixel values rescaling from [0, 255] to [0, 1] interval
datagen = ImageDataGenerator(rescale=1. / 255)
# Retrieve images and their classes for training set.
train_generator_bottleneck = datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode=None,
shuffle=False)
num_classes = len(train_generator_bottleneck.class_indices)
model_vgg = applications.VGG16(include_top=False, weights='imagenet')
bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
np.save('../models/bottleneck_features_train.npy', bottleneck_features_train)
train_data = np.load('../models/bottleneck_features_train.npy')
train_labels = to_categorical(train_generator_bottleneck.classes, num_classes=num_classes)
model_top = Sequential()
model_top.add(Flatten(input_shape=train_data.shape[1:]))
model_top.add(Dense(256, activation='relu'))
model_top.add(Dropout(0.5))
model_top.add(Dense(num_classes, activation='softmax'))
model_top.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# Model saving callback
checkpointer = ModelCheckpoint(filepath='../models/bottleneck_features.h5', monitor='val_acc', verbose=1,
save_best_only=True)
# Early stopping
early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=5)
history = model_top.fit(
train_data,
train_labels,
verbose=2,
epochs=epochs,
batch_size=batch_size,
callbacks=[checkpointer, early_stopping],
validation_split=0.3)
我不相信这里的问题batch_size，正如你提到的，它已经很低了。此外，因为您说它适用于 90k 图像，所以问题可能是train_data无法容纳内存中的 GPU(在每个拟合时期开始时都需要(。为了缓解这个问题，您需要将model_top与generator相匹配，就像您从predict_generator获得功能一样。一种方法是将生成器类包装在train_data周围，但我只会连接两个模型(注意我无法测试这一点，但我认为这是正确的(：
model_vgg = applications.VGG16(include_top=False, weights='imagenet')
model_top = Flatten()(model_vgg)
model_top = Dense(256, activation='relu')(model_top)
model_top = Dropout(0.3)(model_top)
model_top = Dense(num_classes, activation='softmax')(model_top)
model = Model(inputs=model_vgg.inputs, outputs=model_top)
model.compile(loss='sparse_categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# Model saving callback
checkpointer = ModelCheckpoint(filepath='../models/bottleneck_features.h5', monitor='val_acc', verbose=1,
save_best_only=True)
# Early stopping
early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=5)
history = model.fit_generator(
train_data,
train_labels,
verbose=2,
steps_per_epoch=steps_per_epoch,
batch_size=batch_size,
callbacks=[checkpointer, early_stopping],
...)
我将categorical_crossentropy更改为sparse_categorical_crossentropy，以便只能发送索引作为标签，否则相同。您需要提供steps_per_epoch作为训练数据的长度/批量大小。或者只是测试任何数字。我还使用了 keras 函数式 API 来更清楚地说明这一点。
这也将允许 VGG 顶部的重量发生变化，以帮助您更好地分类。如果由于某种原因这不是您想要的，您可以通过遍历所有 vgg 层并将trainable设置为 false 来冻结它。
LMK 如果它有效。